* [RFC v1 0/5] Introduce SCI-mediator feature
@ 2021-12-14 9:34 Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
` (5 more replies)
0 siblings, 6 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel
Cc: Oleksii Moisieiev, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Bertrand Marquis, Andrew Cooper,
George Dunlap, Jan Beulich, Wei Liu, Anthony PERARD,
Juergen Gross
Introducing the feature, called SCI mediator.
It's purpose is to redirect SCMI requests from the domains to firmware
(SCP, ATF etc), which controls the power/clock/resets etc.
The idea is to make SCP firmware (or similar, such as AT-F) responsible for
control power/clock/resets and provide SCMI interface so controls can be shared
between the Domains.
Originally, we've met a problem, that the devices, shared between different
Domains, can't have an access to HW registers to work with clocks/resets/power
etc. You have to pass cpg to the Domain, so the devices can access HW directly.
The solution for this is to move HW controls over power/clock/resets to
SCP firmware and use Linux-kernel SCMI drivers to pass requests to SCP.
Xen is responsible for permissions setting, so Domain can access only to
power/clock/resets which are related to this Domain. Also XEN is the mediator
which redirects SCMI requests, adding agentID so firmware should know the
sender.
SMC is currently used as transport, but this should be configurable.
Here is the high level design:
SCI (System Control Interface) feature can be enabled in xen_config:
> CONFIG_SCI=y
Mediator can be configured:
> CONFIG_SCMI_SMC=y
Currently, only SCMI_SMC mediator is implemented, which using shared memory
region to communicate with firmware and SMC as transport.
Xen scmi should be configured in the device-tree.
Format is the following:
cpu_scp_shm: scp-shmem@0x53FF0000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF0000 0x0 0x1000>;
};
firmware {
scmi {
compatible = "arm,scmi-smc";
arm,smc-id = <0x82000002>;
shmem = <&cpu_scp_shm>;
#address-cells = <1>;
#size-cells = <0>;
scmi_power: protocol@11 {
reg = <0x11>;
#power-domain-cells = <1>;
};
scmi_clock: protocol@14 {
reg = <0x14>;
#clock-cells = <1>;
};
scmi_reset: protocol@16 {
reg = <0x16>;
#reset-cells = <1>;
};
};
};
Where:
&cpu_scp_shm is the shared memory for scmi buffers;
0x53FF0000, size 0x1000 is the platform specific free address, which provide
space for the communication.
&scmi node, which should be copied to Dom0 device-tree.
Device configured to use scmi:
&avb {
scmi_devid = <0>;
clocks = <&scmi_clock 0>;
power-domains = <&scmi_power 0>;
resets = <&scmi_reset 0>;
};
Where:
scmi_devid - id from the firmware, which is assigned for AVB.
During initialization, XEN scans probes the first SCI-mediator driver which has
matching node in the device-tree. If no device-tree was provided, then the
first registered mediator driver should be probed.
DomX should be configured:
Device-tree should include the same nodes, described above.
&cpu_scp_shm should be altered during domain creation. Xen allocates free page
from the memory region, provided in &cpu_scp_shm in XEN device-tree, so each
domain should have unique page. Nodes &cpu_scp_shm and /firmware/scmi should be
copied from partial device-tree to domain device-tree, so kernel can initialize
scmi driver.
SCI mediator can be enabled in dom.cfg the following way:
>sci = "scmi_smc"
which sets scmi_smc to be used for the domain.
Oleksii Moisieiev (5):
xen/arm: add support for Renesas R-Car Gen3 platform
xen/arm: add generic SCI mediator framework
xen/arm: introduce SCMI-SMC mediator driver
tools/arm: add "scmi_smc" option to xl.cfg
xen/arm: add SCI mediator support for DomUs
MAINTAINERS | 6 +
docs/man/xl.cfg.5.pod.in | 22 +
tools/include/libxl.h | 5 +
tools/include/xenctrl.h | 3 +
tools/include/xenguest.h | 2 +
tools/libs/ctrl/xc_domain.c | 23 +
tools/libs/guest/xg_dom_arm.c | 5 +-
tools/libs/light/libxl_arm.c | 122 ++++-
tools/libs/light/libxl_create.c | 54 +-
tools/libs/light/libxl_dom.c | 1 +
tools/libs/light/libxl_internal.h | 4 +
tools/libs/light/libxl_types.idl | 6 +
tools/xl/xl_parse.c | 9 +
xen/arch/arm/Kconfig | 10 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 24 +
xen/arch/arm/domain_build.c | 11 +
xen/arch/arm/domctl.c | 15 +
xen/arch/arm/platforms/Makefile | 1 +
xen/arch/arm/platforms/rcar3.c | 47 ++
xen/arch/arm/sci/Kconfig | 10 +
xen/arch/arm/sci/Makefile | 2 +
xen/arch/arm/sci/sci.c | 128 +++++
xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++
xen/arch/arm/setup.c | 1 +
xen/arch/arm/xen.lds.S | 7 +
xen/include/asm-arm/domain.h | 4 +
xen/include/asm-arm/sci/sci.h | 162 ++++++
xen/include/public/arch-arm.h | 11 +
xen/include/public/domctl.h | 9 +
30 files changed, 1485 insertions(+), 15 deletions(-)
create mode 100644 xen/arch/arm/platforms/rcar3.c
create mode 100644 xen/arch/arm/sci/Kconfig
create mode 100644 xen/arch/arm/sci/Makefile
create mode 100644 xen/arch/arm/sci/sci.c
create mode 100644 xen/arch/arm/sci/scmi_smc.c
create mode 100644 xen/include/asm-arm/sci/sci.h
--
2.27.0
^ permalink raw reply [flat|nested] 95+ messages in thread
* [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
@ 2021-12-14 9:34 ` Oleksii Moisieiev
2021-12-15 6:38 ` Oleksandr Andrushchenko
` (2 more replies)
2021-12-14 9:34 ` [RFC v1 2/5] xen/arm: add generic SCI mediator framework Oleksii Moisieiev
` (4 subsequent siblings)
5 siblings, 3 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel
Cc: Oleksii Moisieiev, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Bertrand Marquis
Implementation includes platform-specific smc handler for rcar3 platform.
Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
---
xen/arch/arm/platforms/Makefile | 1 +
xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+)
create mode 100644 xen/arch/arm/platforms/rcar3.c
diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
index 8632f4115f..b64c25de6c 100644
--- a/xen/arch/arm/platforms/Makefile
+++ b/xen/arch/arm/platforms/Makefile
@@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
obj-$(CONFIG_ALL32_PLAT) += midway.o
obj-$(CONFIG_ALL32_PLAT) += omap5.o
obj-$(CONFIG_ALL32_PLAT) += rcar2.o
+obj-$(CONFIG_RCAR3) += rcar3.o
obj-$(CONFIG_ALL64_PLAT) += seattle.o
obj-$(CONFIG_ALL_PLAT) += sunxi.o
obj-$(CONFIG_ALL64_PLAT) += thunderx.o
diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
new file mode 100644
index 0000000000..d740145c71
--- /dev/null
+++ b/xen/arch/arm/platforms/rcar3.c
@@ -0,0 +1,46 @@
+/*
+ * xen/arch/arm/platforms/rcar3.c
+ *
+ * Renesas R-Car Gen3 specific settings
+ *
+ * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
+ * Copyright (C) 2021 EPAM Systems
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <asm/platform.h>
+
+static bool rcar3_smc(struct cpu_user_regs *regs)
+{
+ return false;
+}
+
+static const char *const rcar3_dt_compat[] __initconst =
+{
+ "renesas,r8a7795",
+ "renesas,r8a7796",
+ NULL
+};
+
+PLATFORM_START(rcar3, "Renesas R-Car Gen3")
+ .compatible = rcar3_dt_compat,
+ .smc = rcar3_smc
+PLATFORM_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
2.27.0
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [RFC v1 2/5] xen/arm: add generic SCI mediator framework
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
@ 2021-12-14 9:34 ` Oleksii Moisieiev
2021-12-17 2:45 ` Stefano Stabellini
2021-12-14 9:34 ` [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver Oleksii Moisieiev
` (3 subsequent siblings)
5 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel
Cc: Oleksii Moisieiev, Andrew Cooper, George Dunlap, Jan Beulich,
Julien Grall, Stefano Stabellini, Wei Liu, Volodymyr Babchuk,
Bertrand Marquis
This patch adds the basic framework for SCI mediator.
SCI is System Control Interface, which is designed to redirect
requests for the Hardware (such as power-domain/clock/resets etc)
from the Domains to the firmware. Originally, cpg should be passed
to the domain so it can work with power-domains/clocks/resets etc.
Considering that cpg can't be split between the Domains, we get the
limitation that the devices, which are using power-domains/clocks/resets
etc, couldn't be split between the domains.
The solution is to move the power-domain/clock/resets etc to the
Firmware (such as SCP firmware or ATF) and provide interface for the Domains.
XEN shoud have an entity, caled SCI-Mediator, which is responsible for messages
redirection between Domains and Firmware and for permission handling.
This is how it works: user can build XEN with multiple SCI mediators.
See the next patches, where SCMI-SMC mediator is introduced.
SCI mediator register itself with REGISTER_SCI_MEDIATOR() macro.
At run-time, during initialization, framework calls probe for the first
matching device in the device-tree. When no device-tree is present - the
first registered mediator should be probed.
Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
---
MAINTAINERS | 6 ++
xen/arch/arm/Kconfig | 8 ++
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 24 +++++
xen/arch/arm/domain_build.c | 11 +++
xen/arch/arm/platforms/rcar3.c | 3 +-
xen/arch/arm/sci/Makefile | 1 +
xen/arch/arm/sci/sci.c | 128 ++++++++++++++++++++++++++
xen/arch/arm/setup.c | 1 +
xen/arch/arm/xen.lds.S | 7 ++
xen/include/asm-arm/domain.h | 4 +
xen/include/asm-arm/sci/sci.h | 162 +++++++++++++++++++++++++++++++++
xen/include/public/arch-arm.h | 10 ++
13 files changed, 365 insertions(+), 1 deletion(-)
create mode 100644 xen/arch/arm/sci/Makefile
create mode 100644 xen/arch/arm/sci/sci.c
create mode 100644 xen/include/asm-arm/sci/sci.h
diff --git a/MAINTAINERS b/MAINTAINERS
index e43dc0edce..5f96ea35ba 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -512,6 +512,12 @@ S: Supported
F: xen/arch/arm/tee/
F: xen/include/asm-arm/tee
+SCI MEDIATORS
+M: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
+S: Supported
+F: xen/arch/arm/sci
+F: xen/include/asm-arm/sci
+
TOOLSTACK
M: Wei Liu <wl@xen.org>
S: Supported
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index ecfa6822e4..186e1db389 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -106,6 +106,14 @@ config TEE
source "arch/arm/tee/Kconfig"
+config SCI
+ bool "Enable SCI mediators support"
+ default n
+ help
+ This option enables generic SCI (System Control Interface) mediators
+ support. It allows guests to control system resourcess via one of
+ SCI mediators implemented in XEN.
+
endmenu
menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 07f634508e..6366ff55e5 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -8,6 +8,7 @@ obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
obj-$(CONFIG_HAS_VPCI) += vpci.o
+obj-$(CONFIG_SCI) += sci/
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 96e1b23550..80d0a23767 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -34,6 +34,7 @@
#include <asm/platform.h>
#include <asm/procinfo.h>
#include <asm/regs.h>
+#include <asm/sci/sci.h>
#include <asm/tee/tee.h>
#include <asm/vfp.h>
#include <asm/vgic.h>
@@ -688,6 +689,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
return -EINVAL;
}
+ if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE &&
+ config->arch.sci_type != sci_get_type() )
+ {
+ dprintk(XENLOG_INFO, "Unsupported SCI type\n");
+ return -EINVAL;
+ }
+
return 0;
}
@@ -764,6 +772,15 @@ int arch_domain_create(struct domain *d,
/* At this stage vgic_reserve_virq should never fail */
if ( !vgic_reserve_virq(d, GUEST_EVTCHN_PPI) )
BUG();
+
+ if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE )
+ {
+ if ( (rc = sci_domain_init(d, config->arch.sci_type)) != 0)
+ goto fail;
+
+ if ( (rc = sci_get_channel_info(d, &config->arch)) != 0)
+ goto fail;
+ }
}
/*
@@ -796,6 +813,7 @@ void arch_domain_destroy(struct domain *d)
domain_vgic_free(d);
domain_vuart_free(d);
free_xenheap_page(d->shared_info);
+ sci_domain_destroy(d);
#ifdef CONFIG_ACPI
free_xenheap_pages(d->arch.efi_acpi_table,
get_order_from_bytes(d->arch.efi_acpi_len));
@@ -996,6 +1014,7 @@ enum {
PROG_xen,
PROG_page,
PROG_mapping,
+ PROG_sci,
PROG_done,
};
@@ -1056,6 +1075,11 @@ int domain_relinquish_resources(struct domain *d)
if ( ret )
return ret;
+ PROGRESS(sci):
+ ret = sci_relinquish_resources(d);
+ if ( ret )
+ return ret;
+
PROGRESS(done):
break;
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index d02bacbcd1..38874615dd 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -22,6 +22,7 @@
#include <asm/kernel.h>
#include <asm/setup.h>
#include <asm/tee/tee.h>
+#include <asm/sci/sci.h>
#include <asm/platform.h>
#include <asm/psci.h>
#include <asm/setup.h>
@@ -1894,6 +1895,10 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
return res;
}
}
+
+ res = sci_add_dt_device(d, dev);
+ if ( res < 0 )
+ return res;
}
res = handle_device_interrupts(d, dev, need_mapping);
@@ -3075,6 +3080,10 @@ static int __init construct_dom0(struct domain *d)
if ( rc < 0 )
return rc;
+ rc = sci_domain_init(d, sci_get_type());
+ if ( rc < 0 )
+ return rc;
+
if ( acpi_disabled )
rc = prepare_dtb_hwdom(d, &kinfo);
else
@@ -3109,6 +3118,8 @@ void __init create_dom0(void)
dom0_cfg.arch.tee_type = tee_get_type();
dom0_cfg.max_vcpus = dom0_max_vcpus();
+ dom0_cfg.arch.sci_type = sci_get_type();
+
if ( iommu_enabled )
dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
index d740145c71..a268b09454 100644
--- a/xen/arch/arm/platforms/rcar3.c
+++ b/xen/arch/arm/platforms/rcar3.c
@@ -18,10 +18,11 @@
*/
#include <asm/platform.h>
+#include <asm/sci/sci.h>
static bool rcar3_smc(struct cpu_user_regs *regs)
{
- return false;
+ return sci_handle_call(current->domain, regs);
}
static const char *const rcar3_dt_compat[] __initconst =
diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
new file mode 100644
index 0000000000..837dc7492b
--- /dev/null
+++ b/xen/arch/arm/sci/Makefile
@@ -0,0 +1 @@
+obj-y += sci.o
diff --git a/xen/arch/arm/sci/sci.c b/xen/arch/arm/sci/sci.c
new file mode 100644
index 0000000000..5961b4cd5d
--- /dev/null
+++ b/xen/arch/arm/sci/sci.c
@@ -0,0 +1,128 @@
+/*
+ * xen/arch/arm/sci/sci.c
+ *
+ * Generic part of SCI mediator driver
+ *
+ * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
+ * Copyright (C) 2021, EPAM Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/sched.h>
+#include <xen/types.h>
+
+#include <asm/sci/sci.h>
+
+extern const struct sci_mediator_desc _sscimediator[], _escimediator[];
+static const struct sci_mediator_desc __read_mostly *cur_mediator;
+
+bool sci_handle_call(struct domain *d, void *args)
+{
+ if ( unlikely(!cur_mediator) )
+ return false;
+
+ return cur_mediator->ops->handle_call(d, args);
+}
+
+int sci_domain_init(struct domain *d, uint16_t sci_type)
+{
+ if ( sci_type == XEN_DOMCTL_CONFIG_SCI_NONE )
+ return 0;
+
+ if ( unlikely(!cur_mediator) )
+ return -ENODEV;
+
+ if ( cur_mediator->sci_type != sci_type )
+ return -EINVAL;
+
+ return cur_mediator->ops->domain_init(d);
+}
+
+void sci_domain_destroy(struct domain *d)
+{
+ if ( unlikely(!cur_mediator) )
+ return;
+
+ cur_mediator->ops->domain_destroy(d);
+}
+
+int sci_relinquish_resources(struct domain *d)
+{
+ if ( unlikely(!cur_mediator) )
+ return 0;
+
+ return cur_mediator->ops->relinquish_resources(d);
+}
+
+
+int sci_add_dt_device(struct domain *d, struct dt_device_node *dev)
+{
+ if ( unlikely(!cur_mediator) )
+ return 0;
+
+ return cur_mediator->ops->add_dt_device(d, dev);
+}
+
+int sci_get_channel_info(struct domain *d,
+ struct xen_arch_domainconfig *config)
+{
+ if ( unlikely(!cur_mediator) )
+ return 0;
+
+ return cur_mediator->ops->get_channel_info(d->arch.sci, config);
+}
+
+uint16_t sci_get_type(void)
+{
+ if ( unlikely(!cur_mediator) )
+ return XEN_DOMCTL_CONFIG_SCI_NONE;
+
+ return cur_mediator->sci_type;
+}
+
+static int __init sci_init(void)
+{
+ const struct sci_mediator_desc *desc;
+ struct dt_device_node *dt = NULL;
+
+ for ( desc = _sscimediator; desc != _escimediator; desc++ )
+ {
+ if ( likely(dt_host) )
+ {
+ dt = dt_find_matching_node(dt_host, desc->dt_match);
+ if ( !dt )
+ continue;
+ }
+
+ if ( desc->ops->probe(dt) )
+ {
+ printk(XENLOG_INFO "Using SCI mediator for %s\n", desc->name);
+ cur_mediator = desc;
+ return 0;
+ }
+ }
+
+ return 0;
+}
+
+__initcall(sci_init);
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index d5d0792ed4..201de01411 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -50,6 +50,7 @@
#include <asm/cpufeature.h>
#include <asm/platform.h>
#include <asm/procinfo.h>
+#include <asm/sci/sci.h>
#include <asm/setup.h>
#include <xsm/xsm.h>
#include <asm/acpi.h>
diff --git a/xen/arch/arm/xen.lds.S b/xen/arch/arm/xen.lds.S
index 08016948ab..3683f4821f 100644
--- a/xen/arch/arm/xen.lds.S
+++ b/xen/arch/arm/xen.lds.S
@@ -142,6 +142,13 @@ SECTIONS
_eteemediator = .;
} :text
+ . = ALIGN(8);
+ .scimediator.info : {
+ _sscimediator = .;
+ *(.scimediator.info)
+ _escimediator = .;
+ } :text
+
. = ALIGN(PAGE_SIZE); /* Init code and data */
__init_begin = .;
.init.text : {
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 9b3647587a..d9b164017f 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -89,6 +89,10 @@ struct arch_domain
#ifdef CONFIG_TEE
void *tee;
#endif
+
+#ifdef CONFIG_SCI
+ void *sci;
+#endif
} __cacheline_aligned;
struct arch_vcpu
diff --git a/xen/include/asm-arm/sci/sci.h b/xen/include/asm-arm/sci/sci.h
new file mode 100644
index 0000000000..aeff689c72
--- /dev/null
+++ b/xen/include/asm-arm/sci/sci.h
@@ -0,0 +1,162 @@
+/*
+ * xen/include/asm-arm/sci/sci.h
+ *
+ * Generic part of the SCI (System Control Interface) subsystem.
+ *
+ * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
+ * Copyright (C) 2021, EPAM Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ASM_ARM_SCI_H
+#define __ASM_ARM_SCI_H
+
+#include <xen/lib.h>
+#include <xen/types.h>
+#include <xen/device_tree.h>
+
+#ifdef CONFIG_SCI
+
+struct sci_mediator_ops {
+
+ /*
+ * Probe for SCI. Should return true if SCI found and
+ * mediator is initialized.
+ */
+ bool (*probe)(struct dt_device_node *scmi_node);
+
+ /*
+ * Called during domain construction if toolstack requests to enable
+ * SCI support so mediator can inform SCP-firmware about new
+ * guest and create own structures for the new domain.
+ */
+ int (*domain_init)(struct domain *d);
+
+ /*
+ * Called during domain destruction, releases all resources, that
+ * were allocated by the mediator.
+ */
+ void (*domain_destroy)(struct domain *d);
+
+ /*
+ * Called during parsing partial device-sci for the domain.
+ * Passing device_node so mediator could process the device and
+ * mark the device as related to the domain if needed.
+ */
+ int (*add_dt_device)(struct domain *d, struct dt_device_node *dev);
+
+ /*
+ * Called during domain destruction to relinquish resources used
+ * by mediator itself. This function can return -ERESTART to indicate
+ * that it does not finished work and should be called again.
+ */
+ int (*relinquish_resources)(struct domain *d);
+
+ /* Handle call for current domain */
+ bool (*handle_call)(struct domain *d, void *regs);
+
+ /* Gets channel configuration and store it in domainconfig */
+ int (*get_channel_info)(void *sci_ops,
+ struct xen_arch_domainconfig *config);
+
+};
+
+struct sci_mediator_desc {
+ /* Printable name of the SCI. */
+ const char *name;
+
+ /* Mediator callbacks as described above. */
+ const struct sci_mediator_ops *ops;
+
+ /*
+ * ID of SCI. Corresponds to xen_arch_domainconfig.sci_type.
+ * Should be one of XEN_DOMCTL_CONFIG_SCI_xxx
+ */
+ uint16_t sci_type;
+
+ /* Match structure to init mediator */
+ const struct dt_device_match *dt_match;
+
+};
+
+int sci_domain_init(struct domain *d, uint16_t sci_type);
+void sci_domain_destroy(struct domain *d);
+int sci_add_dt_device(struct domain *d, struct dt_device_node *dev);
+int sci_relinquish_resources(struct domain *d);
+bool sci_handle_call(struct domain *d, void *args);
+int sci_get_channel_info(struct domain *d,
+ struct xen_arch_domainconfig *config);
+uint16_t sci_get_type(void);
+
+#define REGISTER_SCI_MEDIATOR(_name, _namestr, _type, _match, _ops) \
+static const struct sci_mediator_desc __sci_desc_##_name __used \
+__section(".scimediator.info") = { \
+ .name = _namestr, \
+ .ops = _ops, \
+ .sci_type = _type, \
+ .dt_match = _match \
+}
+
+#else
+#include <public/errno.h>
+
+static inline int sci_domain_init(struct domain *d, uint16_t sci_type)
+{
+ if ( likely(sci_type == XEN_DOMCTL_CONFIG_SCI_NONE) )
+ return 0;
+
+ return -XEN_ENODEV;
+}
+
+static inline void sci_domain_destroy(struct domain *d)
+{
+}
+
+static inline int sci_add_dt_device(struct domain *d,
+ struct dt_device_node *dev)
+{
+ return 0;
+}
+
+static inline int sci_relinquish_resources(struct domain *d)
+{
+ return 0;
+}
+
+static inline bool sci_handle_call(struct domain *d, void *args)
+{
+ return false;
+}
+
+static inline int sci_get_channel_info(struct domain *d,
+ struct xen_arch_domainconfig *config)
+{
+ return 0;
+}
+
+static inline uint16_t sci_get_type(void)
+{
+ return XEN_DOMCTL_CONFIG_SCI_NONE;
+}
+
+#endif /* CONFIG_SCI */
+
+#endif /* __ASM_ARM_SCI_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 94b31511dd..9180be5e86 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -314,12 +314,16 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
#define XEN_DOMCTL_CONFIG_TEE_NONE 0
#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
+#define XEN_DOMCTL_CONFIG_SCI_NONE 0
+
struct xen_arch_domainconfig {
/* IN/OUT */
uint8_t gic_version;
/* IN */
uint16_t tee_type;
/* IN */
+ uint8_t sci_type;
+ /* IN */
uint32_t nr_spis;
/*
* OUT
@@ -335,6 +339,12 @@ struct xen_arch_domainconfig {
*
*/
uint32_t clock_frequency;
+
+ /* Sets shared address to sw domains.
+ * This information is needed to set correct channel in Domain partial
+ * device-tree
+ */
+ uint64_t sci_agent_paddr;
};
#endif /* __XEN__ || __XEN_TOOLS__ */
--
2.27.0
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 2/5] xen/arm: add generic SCI mediator framework Oleksii Moisieiev
@ 2021-12-14 9:34 ` Oleksii Moisieiev
2021-12-17 11:35 ` Oleksandr
2021-12-18 2:14 ` Stefano Stabellini
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
` (2 subsequent siblings)
5 siblings, 2 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel
Cc: Oleksii Moisieiev, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Bertrand Marquis
This is the implementation of SCI interface, called SCMI-SMC driver,
which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
This allows devices from the Domains to work with clocks, resets and
power-domains without access to CPG.
The following features are implemented:
- request SCMI channels from ATF and pass channels to Domains;
- set device permissions for Domains based on the Domain partial
device-tree. Devices with permissions are able to work with clocks,
resets and power-domains via SCMI;
- redirect scmi messages from Domains to ATF.
Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
---
xen/arch/arm/Kconfig | 2 +
xen/arch/arm/sci/Kconfig | 10 +
xen/arch/arm/sci/Makefile | 1 +
xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
xen/include/public/arch-arm.h | 1 +
5 files changed, 809 insertions(+)
create mode 100644 xen/arch/arm/sci/Kconfig
create mode 100644 xen/arch/arm/sci/scmi_smc.c
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 186e1db389..02d96c6cfc 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -114,6 +114,8 @@ config SCI
support. It allows guests to control system resourcess via one of
SCI mediators implemented in XEN.
+source "arch/arm/sci/Kconfig"
+
endmenu
menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
new file mode 100644
index 0000000000..9563067ddc
--- /dev/null
+++ b/xen/arch/arm/sci/Kconfig
@@ -0,0 +1,10 @@
+config SCMI_SMC
+ bool "Enable SCMI-SMC mediator driver"
+ default n
+ depends on SCI
+ ---help---
+
+ Enables mediator in XEN to pass SCMI requests from Domains to ATF.
+ This feature allows drivers from Domains to work with System
+ Controllers (such as power,resets,clock etc.). SCP is used as transport
+ for communication.
diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
index 837dc7492b..67f2611872 100644
--- a/xen/arch/arm/sci/Makefile
+++ b/xen/arch/arm/sci/Makefile
@@ -1 +1,2 @@
obj-y += sci.o
+obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
new file mode 100644
index 0000000000..2eb01ea82d
--- /dev/null
+++ b/xen/arch/arm/sci/scmi_smc.c
@@ -0,0 +1,795 @@
+/*
+ * xen/arch/arm/sci/scmi_smc.c
+ *
+ * SCMI mediator driver, using SCP as transport.
+ *
+ * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
+ * Copyright (C) 2021, EPAM Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <asm/sci/sci.h>
+#include <asm/smccc.h>
+#include <asm/io.h>
+#include <xen/bitops.h>
+#include <xen/config.h>
+#include <xen/sched.h>
+#include <xen/device_tree.h>
+#include <xen/iocap.h>
+#include <xen/init.h>
+#include <xen/err.h>
+#include <xen/lib.h>
+#include <xen/list.h>
+#include <xen/mm.h>
+#include <xen/string.h>
+#include <xen/time.h>
+#include <xen/vmap.h>
+
+#define SCMI_BASE_PROTOCOL 0x10
+#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
+#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
+#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
+#define SCMI_BASE_DISCOVER_AGENT 0x7
+
+/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
+#define SCMI_SUCCESS 0
+#define SCMI_NOT_SUPPORTED (-1)
+#define SCMI_INVALID_PARAMETERS (-2)
+#define SCMI_DENIED (-3)
+#define SCMI_NOT_FOUND (-4)
+#define SCMI_OUT_OF_RANGE (-5)
+#define SCMI_BUSY (-6)
+#define SCMI_COMMS_ERROR (-7)
+#define SCMI_GENERIC_ERROR (-8)
+#define SCMI_HARDWARE_ERROR (-9)
+#define SCMI_PROTOCOL_ERROR (-10)
+
+#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
+
+#define SCMI_SMC_ID "arm,smc-id"
+#define SCMI_SHARED_MEMORY "linux,scmi_mem"
+#define SCMI_SHMEM "shmem"
+
+#define HYP_CHANNEL 0x0
+
+#define HDR_ID GENMASK(7,0)
+#define HDR_TYPE GENMASK(9, 8)
+#define HDR_PROTO GENMASK(17, 10)
+
+/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
+#define MSG_N_AGENTS_MASK GENMASK(15, 8)
+
+#define FIELD_GET(_mask, _reg)\
+ ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
+#define FIELD_PREP(_mask, _val)\
+ (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
+
+typedef struct scmi_msg_header {
+ uint8_t id;
+ uint8_t type;
+ uint8_t protocol;
+} scmi_msg_header_t;
+
+typedef struct scmi_perms_tx {
+ uint32_t agent_id;
+ uint32_t device_id;
+ uint32_t flags;
+} scmi_perms_tx_t;
+
+#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
+#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
+
+#define SCMI_ALLOW_ACCESS BIT(0, UL)
+
+struct scmi_shared_mem {
+ uint32_t reserved;
+ uint32_t channel_status;
+ uint32_t reserved1[2];
+ uint32_t flags;
+ uint32_t length;
+ uint32_t msg_header;
+ uint8_t msg_payload[];
+};
+
+struct scmi_channel {
+ int chan_id;
+ int agent_id;
+ uint32_t func_id;
+ int domain_id;
+ uint64_t paddr;
+ struct scmi_shared_mem *shmem;
+ spinlock_t lock;
+ struct list_head list;
+};
+
+struct scmi_data {
+ struct list_head channel_list;
+ spinlock_t channel_list_lock;
+ bool initialized;
+ u64 shmem_addr, shmem_size;
+};
+
+static struct scmi_data scmi_data;
+
+/*
+ * pack_scmi_header() - packs and returns 32-bit header
+ *
+ * @hdr: pointer to header containing all the information on message id,
+ * protocol id and type id.
+ *
+ * Return: 32-bit packed message header to be sent to the platform.
+ */
+static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
+{
+ return FIELD_PREP(HDR_ID, hdr->id) |
+ FIELD_PREP(HDR_TYPE, hdr->type) |
+ FIELD_PREP(HDR_PROTO, hdr->protocol);
+}
+
+/*
+ * unpack_scmi_header() - unpacks and records message and protocol id
+ *
+ * @msg_hdr: 32-bit packed message header sent from the platform
+ * @hdr: pointer to header to fetch message and protocol id.
+ */
+static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
+{
+ hdr->id = FIELD_GET(HDR_ID, msg_hdr);
+ hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
+ hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
+}
+
+static inline int channel_is_free(struct scmi_channel *chan_info)
+{
+ return ( chan_info->shmem->channel_status
+ & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
+}
+
+static int send_smc_message(struct scmi_channel *chan_info,
+ scmi_msg_header_t *hdr, void *data, int len)
+{
+ struct arm_smccc_res resp;
+ int ret;
+
+ printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
+ chan_info->shmem->channel_status, len);
+ printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
+ hdr->id, hdr->type, hdr->protocol);
+
+ ret = channel_is_free(chan_info);
+ if ( IS_ERR_VALUE(ret) )
+ return ret;
+
+ chan_info->shmem->channel_status = 0x0;
+ /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
+ chan_info->shmem->flags = 0x0;
+ chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
+ chan_info->shmem->msg_header = pack_scmi_header(hdr);
+
+ printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
+ chan_info->shmem);
+ if ( len > 0 && data )
+ memcpy((void *)(chan_info->shmem->msg_payload), data, len);
+
+ arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
+ &resp);
+
+ printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
+
+ if ( resp.a0 )
+ return -EOPNOTSUPP;
+
+ return 0;
+}
+
+static int check_scmi_status(int scmi_status)
+{
+ if ( scmi_status == SCMI_SUCCESS )
+ return 0;
+
+ printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
+
+ switch ( scmi_status )
+ {
+ case SCMI_NOT_SUPPORTED:
+ return -EOPNOTSUPP;
+ case SCMI_INVALID_PARAMETERS:
+ return -EINVAL;
+ case SCMI_DENIED:
+ return -EACCES;
+ case SCMI_NOT_FOUND:
+ return -ENOENT;
+ case SCMI_OUT_OF_RANGE:
+ return -ERANGE;
+ case SCMI_BUSY:
+ return -EBUSY;
+ case SCMI_COMMS_ERROR:
+ return -ENOTCONN;
+ case SCMI_GENERIC_ERROR:
+ return -EIO;
+ case SCMI_HARDWARE_ERROR:
+ return -ENXIO;
+ case SCMI_PROTOCOL_ERROR:
+ return -EBADMSG;
+ }
+
+ return -EINVAL;
+}
+
+static int get_smc_response(struct scmi_channel *chan_info,
+ scmi_msg_header_t *hdr, void *data, int len)
+{
+ int recv_len;
+ int ret;
+
+ printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
+
+ ret = channel_is_free(chan_info);
+ if ( IS_ERR_VALUE(ret) )
+ return ret;
+
+ recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
+
+ if ( recv_len < 0 )
+ {
+ printk(XENLOG_ERR
+ "scmi: Wrong size of smc message. Data may be invalid\n");
+ return -EINVAL;
+ }
+
+ if ( recv_len > len )
+ {
+ printk(XENLOG_ERR
+ "scmi: Not enough buffer for message %d, expecting %d\n",
+ recv_len, len);
+ return -EINVAL;
+ }
+
+ unpack_scmi_header(chan_info->shmem->msg_header, hdr);
+
+ if ( recv_len > 0 )
+ {
+ memcpy(data, chan_info->shmem->msg_payload, recv_len);
+ }
+
+ return 0;
+}
+
+static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
+ void *rx_data, int rx_size)
+{
+ int ret = 0;
+
+ if ( !hdr )
+ return -EINVAL;
+
+ spin_lock(&channel->lock);
+
+ ret = send_smc_message(channel, hdr, tx_data, tx_size);
+ if ( ret )
+ goto clean;
+
+ ret = get_smc_response(channel, hdr, rx_data, rx_size);
+clean:
+ spin_unlock(&channel->lock);
+
+ return ret;
+}
+
+static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
+{
+ struct scmi_channel *curr;
+ bool found = false;
+
+ spin_lock(&scmi_data.channel_list_lock);
+ list_for_each_entry(curr, &scmi_data.channel_list, list)
+ if ( curr->chan_id == chan_id )
+ {
+ found = true;
+ break;
+ }
+
+ spin_unlock(&scmi_data.channel_list_lock);
+ if ( found )
+ return curr;
+
+ return NULL;
+}
+
+static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
+{
+ struct scmi_channel *curr;
+ bool found = false;
+
+ spin_lock(&scmi_data.channel_list_lock);
+ list_for_each_entry(curr, &scmi_data.channel_list, list)
+ if ( curr->domain_id == domain_id )
+ {
+ found = true;
+ break;
+ }
+
+ spin_unlock(&scmi_data.channel_list_lock);
+ if ( found )
+ return curr;
+
+ return NULL;
+}
+
+static struct scmi_channel *aquire_scmi_channel(int domain_id)
+{
+ struct scmi_channel *curr;
+ bool found = false;
+
+ ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
+
+ spin_lock(&scmi_data.channel_list_lock);
+ list_for_each_entry(curr, &scmi_data.channel_list, list)
+ if ( (curr->domain_id == DOMID_INVALID)
+ && (curr->chan_id != HYP_CHANNEL) )
+ {
+ curr->domain_id = domain_id;
+ found = true;
+ break;
+ }
+
+ spin_unlock(&scmi_data.channel_list_lock);
+ if ( found )
+ return curr;
+
+ return NULL;
+}
+
+static void relinquish_scmi_channel(struct scmi_channel *channel)
+{
+ spin_lock(&scmi_data.channel_list_lock);
+ ASSERT(channel != NULL);
+ channel->domain_id = DOMID_INVALID;
+ spin_unlock(&scmi_data.channel_list_lock);
+}
+
+static struct scmi_channel *smc_create_channel(uint8_t chan_id,
+ uint32_t func_id, uint64_t addr)
+{
+ struct scmi_channel *channel;
+ mfn_t mfn;
+
+ channel = get_channel_by_id(chan_id);
+ if ( channel )
+ return ERR_PTR(EEXIST);
+
+ channel = xmalloc(struct scmi_channel);
+ if ( !channel )
+ return ERR_PTR(ENOMEM);
+
+ channel->chan_id = chan_id;
+ channel->func_id = func_id;
+ channel->domain_id = DOMID_INVALID;
+ mfn = maddr_to_mfn(addr);
+ channel->shmem = vmap(&mfn, 1);
+ if ( !channel->shmem )
+ {
+ xfree(channel);
+ return ERR_PTR(ENOMEM);
+ }
+
+ printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
+ channel->paddr = addr;
+ channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
+ spin_lock_init(&channel->lock);
+ spin_lock(&scmi_data.channel_list_lock);
+ list_add(&channel->list, &scmi_data.channel_list);
+ spin_unlock(&scmi_data.channel_list_lock);
+ return channel;
+}
+
+static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
+{
+ return iomem_permit_access(d, paddr_to_pfn(addr),
+ paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
+}
+
+static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
+ uint64_t len)
+{
+ return iomem_deny_access(d, paddr_to_pfn(addr),
+ paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
+}
+
+static int dt_update_domain_range(struct domain *d, uint64_t addr,
+ uint64_t size)
+{
+ struct dt_device_node *shmem_node;
+ __be32 *hw_reg;
+ const struct dt_property *pp;
+ uint32_t len;
+
+ shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
+
+ if ( !shmem_node )
+ {
+ printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
+ return -EINVAL;
+ }
+
+ pp = dt_find_property(shmem_node, "reg", &len);
+ if ( !pp )
+ {
+ printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
+ return -ENOENT;
+ }
+
+ hw_reg = pp->value;
+ dt_set_range(&hw_reg, shmem_node, addr, size);
+
+ return 0;
+}
+
+static void free_channel_list(void)
+{
+ struct scmi_channel *curr, *_curr;
+
+ spin_lock(&scmi_data.channel_list_lock);
+ list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
+ {
+ vunmap(curr->shmem);
+ list_del(&curr->list);
+ xfree(curr);
+ }
+
+ spin_unlock(&scmi_data.channel_list_lock);
+}
+
+static __init bool scmi_probe(struct dt_device_node *scmi_node)
+{
+ struct dt_device_node *shmem_node;
+ int ret, i;
+ struct scmi_channel *channel, *agent_channel;
+ int n_agents;
+ scmi_msg_header_t hdr;
+ struct rx_t {
+ int32_t status;
+ uint32_t attributes;
+ } rx;
+
+ uint32_t func_id;
+
+ ASSERT(scmi_node != NULL);
+
+ INIT_LIST_HEAD(&scmi_data.channel_list);
+ spin_lock_init(&scmi_data.channel_list_lock);
+
+ if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
+ {
+ printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
+ return false;
+ }
+
+ shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
+ if ( IS_ERR_OR_NULL(shmem_node) )
+ {
+ printk(XENLOG_ERR
+ "scmi: Device tree error, can't parse shmem phandle %ld\n",
+ PTR_ERR(shmem_node));
+ return false;
+ }
+
+ ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
+ &scmi_data.shmem_size);
+ if ( IS_ERR_VALUE(ret) )
+ return false;
+
+ channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
+ if ( IS_ERR(channel) )
+ return false;
+
+ hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
+ hdr.type = 0;
+ hdr.protocol = SCMI_BASE_PROTOCOL;
+
+ ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
+ if ( ret )
+ goto clean;
+
+ ret = check_scmi_status(rx.status);
+ if ( ret )
+ goto clean;
+
+ n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
+ printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
+
+ n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
+ scmi_data.shmem_size / PAGE_SIZE : n_agents;
+
+ for ( i = 1; i < n_agents; i++ )
+ {
+ uint32_t tx_agent_id = 0xFFFFFFFF;
+ struct {
+ int32_t status;
+ uint32_t agent_id;
+ char name[16];
+ } da_rx;
+
+ agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
+ i * PAGE_SIZE);
+ if ( IS_ERR(agent_channel) )
+ {
+ ret = PTR_ERR(agent_channel);
+ goto clean;
+ }
+
+ hdr.id = SCMI_BASE_DISCOVER_AGENT;
+ hdr.type = 0;
+ hdr.protocol = SCMI_BASE_PROTOCOL;
+
+ ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
+ sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
+ if ( ret )
+ goto clean;
+
+ ret = check_scmi_status(da_rx.status);
+ if ( ret )
+ goto clean;
+
+ printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
+ da_rx.status, da_rx.agent_id, da_rx.name);
+
+ agent_channel->agent_id = da_rx.agent_id;
+ }
+
+ scmi_data.initialized = true;
+ return true;
+
+clean:
+ free_channel_list();
+ return ret == 0;
+}
+
+static int scmi_domain_init(struct domain *d)
+{
+ struct scmi_channel *channel;
+ int ret;
+
+ if ( !scmi_data.initialized )
+ return 0;
+
+ channel = aquire_scmi_channel(d->domain_id);
+ if ( IS_ERR_OR_NULL(channel) )
+ return -ENOENT;
+
+ printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
+ "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
+ channel->paddr);
+
+ if ( is_hardware_domain(d) )
+ {
+ ret = map_memory_to_domain(d, scmi_data.shmem_addr,
+ scmi_data.shmem_size);
+ if ( IS_ERR_VALUE(ret) )
+ goto error;
+
+ ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
+ if ( IS_ERR_VALUE(ret) )
+ {
+ int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
+ scmi_data.shmem_size);
+ if ( rc )
+ printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
+
+ goto error;
+ }
+ }
+
+ d->arch.sci = channel;
+
+ return 0;
+error:
+ relinquish_scmi_channel(channel);
+
+ return ret;
+}
+
+static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
+{
+ struct scmi_channel *channel, *agent_channel;
+ scmi_msg_header_t hdr;
+ scmi_perms_tx_t tx;
+ struct rx_t {
+ int32_t status;
+ uint32_t attributes;
+ } rx;
+ int ret;
+
+ if ( !scmi_data.initialized )
+ return 0;
+
+ printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
+
+ agent_channel = get_channel_by_domain(d->domain_id);
+ if ( IS_ERR_OR_NULL(agent_channel) )
+ return PTR_ERR(agent_channel);
+
+ channel = get_channel_by_id(HYP_CHANNEL);
+ if ( IS_ERR_OR_NULL(channel) )
+ return PTR_ERR(channel);
+
+ hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
+ hdr.type = 0;
+ hdr.protocol = SCMI_BASE_PROTOCOL;
+
+ tx.agent_id = agent_channel->agent_id;
+ tx.device_id = scmi_devid;
+ tx.flags = SCMI_ALLOW_ACCESS;
+
+ ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
+ if ( IS_ERR_VALUE(ret) )
+ return ret;
+
+ ret = check_scmi_status(rx.status);
+ if ( IS_ERR_VALUE(ret) )
+ return ret;
+
+ return 0;
+}
+
+static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
+{
+ uint32_t scmi_devid;
+
+ if ( (!scmi_data.initialized) || (!d->arch.sci) )
+ return 0;
+
+ if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
+ return 0;
+
+ printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
+
+ return scmi_add_device_by_devid(d, scmi_devid);
+}
+
+static int scmi_relinquish_resources(struct domain *d)
+{
+ int ret;
+ struct scmi_channel *channel, *agent_channel;
+ scmi_msg_header_t hdr;
+ struct reset_agent_tx {
+ uint32_t agent_id;
+ uint32_t flags;
+ } tx;
+ uint32_t rx;
+
+ if ( !d->arch.sci )
+ return 0;
+
+ agent_channel = d->arch.sci;
+
+ spin_lock(&agent_channel->lock);
+ tx.agent_id = agent_channel->agent_id;
+ spin_unlock(&agent_channel->lock);
+
+ channel = get_channel_by_id(HYP_CHANNEL);
+ if ( !channel )
+ {
+ printk(XENLOG_ERR
+ "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
+ d->domain_id);
+ return -EINVAL;
+ }
+
+ hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
+ hdr.type = 0;
+ hdr.protocol = SCMI_BASE_PROTOCOL;
+
+ tx.flags = 0;
+
+ ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
+ if ( ret )
+ return ret;
+
+ ret = check_scmi_status(rx);
+
+ return ret;
+}
+
+static void scmi_domain_destroy(struct domain *d)
+{
+ struct scmi_channel *channel;
+
+ if ( !d->arch.sci )
+ return;
+
+ channel = d->arch.sci;
+ spin_lock(&channel->lock);
+
+ relinquish_scmi_channel(channel);
+ printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
+
+ d->arch.sci = NULL;
+
+ unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
+ spin_unlock(&channel->lock);
+ return;
+}
+
+static bool scmi_handle_call(struct domain *d, void *args)
+{
+ bool res = false;
+ struct scmi_channel *agent_channel;
+ struct arm_smccc_res resp;
+ struct cpu_user_regs *regs = args;
+
+ if ( !d->arch.sci )
+ return false;
+
+ agent_channel = d->arch.sci;
+ spin_lock(&agent_channel->lock);
+
+ if ( agent_channel->func_id != regs->x0 )
+ {
+ printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
+ goto unlock;
+ }
+
+ arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
+ agent_channel->chan_id, &resp);
+
+ set_user_reg(regs, 0, resp.a0);
+ set_user_reg(regs, 1, resp.a1);
+ set_user_reg(regs, 2, resp.a2);
+ set_user_reg(regs, 3, resp.a3);
+ res = true;
+unlock:
+ spin_unlock(&agent_channel->lock);
+
+ return res;
+}
+
+static int scmi_get_channel_paddr(void *scmi_ops,
+ struct xen_arch_domainconfig *config)
+{
+ struct scmi_channel *agent_channel = scmi_ops;
+
+ if ( !agent_channel )
+ return -EINVAL;
+
+ config->sci_agent_paddr = agent_channel->paddr;
+ return 0;
+}
+
+static const struct dt_device_match scmi_smc_match[] __initconst =
+{
+ DT_MATCH_SCMI_SMC,
+ { /* sentinel */ },
+};
+
+static const struct sci_mediator_ops scmi_ops =
+{
+ .probe = scmi_probe,
+ .domain_init = scmi_domain_init,
+ .domain_destroy = scmi_domain_destroy,
+ .add_dt_device = scmi_add_dt_device,
+ .relinquish_resources = scmi_relinquish_resources,
+ .handle_call = scmi_handle_call,
+ .get_channel_info = scmi_get_channel_paddr
+};
+
+REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
+ scmi_smc_match, &scmi_ops);
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 9180be5e86..a67237942d 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
#define XEN_DOMCTL_CONFIG_SCI_NONE 0
+#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
struct xen_arch_domainconfig {
/* IN/OUT */
--
2.27.0
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
` (2 preceding siblings ...)
2021-12-14 9:34 ` [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver Oleksii Moisieiev
@ 2021-12-14 9:34 ` Oleksii Moisieiev
2021-12-15 21:51 ` Oleksandr
` (2 more replies)
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
2021-12-16 0:33 ` [RFC v1 0/5] Introduce SCI-mediator feature Oleksandr
5 siblings, 3 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel; +Cc: Oleksii Moisieiev, Wei Liu, Anthony PERARD, Juergen Gross
This enumeration sets SCI type for the domain. Currently there is
two possible options: either 'none' or 'scmi_smc'.
'none' is the default value and it disables SCI support at all.
'scmi_smc' enables access to the Firmware from the domains using SCMI
protocol and SMC as transport.
Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
---
docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
tools/include/libxl.h | 5 +++++
tools/libs/light/libxl_types.idl | 6 ++++++
tools/xl/xl_parse.c | 9 +++++++++
4 files changed, 42 insertions(+)
diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..92d0593875 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
=back
+=item B<sci="STRING">
+
+B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
+allows domain to manage various functions that are provided by HW platform.
+
+=over 4
+
+=item B<none>
+
+Don't allow guest to use SCI if present on the platform. This is the default
+value.
+
+=item B<scmi_smc>
+
+Enables SCMI-SMC support for the guest. SCMI is System Control Management
+Inferface - allows domain to manage various functions that are provided by HW
+platform, such as clocks, resets and power-domains. Xen will mediate access to
+clocks, power-domains and resets between Domains and ATF. Disabled by default.
+SMC is used as transport.
+
+=back
+
=back
=head2 Paravirtualised (PV) Guest Specific Options
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 2bbbd21f0b..30e5aee119 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -278,6 +278,11 @@
*/
#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE 1
+/*
+ * libxl_domain_build_info has the arch_arm.sci field.
+ */
+#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_SCI 1
+
/*
* LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing
* 'soft reset' for domains and there is 'soft_reset' shutdown reason
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index 2a42da2f7d..9067b509f4 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -480,6 +480,11 @@ libxl_tee_type = Enumeration("tee_type", [
(1, "optee")
], init_val = "LIBXL_TEE_TYPE_NONE")
+libxl_sci_type = Enumeration("sci_type", [
+ (0, "none"),
+ (1, "scmi_smc")
+ ], init_val = "LIBXL_SCI_TYPE_NONE")
+
libxl_rdm_reserve = Struct("rdm_reserve", [
("strategy", libxl_rdm_reserve_strategy),
("policy", libxl_rdm_reserve_policy),
@@ -564,6 +569,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("apic", libxl_defbool),
("dm_restrict", libxl_defbool),
("tee", libxl_tee_type),
+ ("sci", libxl_sci_type),
("u", KeyedUnion(None, libxl_domain_type, "type",
[("hvm", Struct(None, [("firmware", string),
("bios", libxl_bios_type),
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 117fcdcb2b..c37bf6298b 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -2747,6 +2747,15 @@ skip_usbdev:
}
}
+ if (!xlu_cfg_get_string (config, "sci", &buf, 1)) {
+ e = libxl_sci_type_from_string(buf, &b_info->sci);
+ if (e) {
+ fprintf(stderr,
+ "Unknown sci \"%s\" specified\n", buf);
+ exit(-ERROR_FAIL);
+ }
+ }
+
parse_vkb_list(config, d_config);
xlu_cfg_get_defbool(config, "xend_suspend_evtchn_compat",
--
2.27.0
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
` (3 preceding siblings ...)
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
@ 2021-12-14 9:34 ` Oleksii Moisieiev
2021-12-14 9:41 ` Jan Beulich
` (2 more replies)
2021-12-16 0:33 ` [RFC v1 0/5] Introduce SCI-mediator feature Oleksandr
5 siblings, 3 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-14 9:34 UTC (permalink / raw)
To: xen-devel
Cc: Oleksii Moisieiev, Wei Liu, Juergen Gross, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Anthony PERARD, Volodymyr Babchuk, Bertrand Marquis
Integration of the SCMI mediator with xen libs:
- add hypercalls to aquire SCI channel and set device permissions
for DomUs;
- add SCMI_SMC nodes to DomUs device-tree based on partial device-tree;
- SCI requests redirection from DomUs to Firmware.
Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
---
tools/include/xenctrl.h | 3 +
tools/include/xenguest.h | 2 +
tools/libs/ctrl/xc_domain.c | 23 ++++++
tools/libs/guest/xg_dom_arm.c | 5 +-
tools/libs/light/libxl_arm.c | 122 +++++++++++++++++++++++++++---
tools/libs/light/libxl_create.c | 54 ++++++++++++-
tools/libs/light/libxl_dom.c | 1 +
tools/libs/light/libxl_internal.h | 4 +
xen/arch/arm/domctl.c | 15 ++++
xen/include/public/domctl.h | 9 +++
10 files changed, 223 insertions(+), 15 deletions(-)
diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
index 07b96e6671..cdd14f465f 100644
--- a/tools/include/xenctrl.h
+++ b/tools/include/xenctrl.h
@@ -1238,6 +1238,9 @@ int xc_domain_getvnuma(xc_interface *xch,
int xc_domain_soft_reset(xc_interface *xch,
uint32_t domid);
+int xc_domain_add_sci_device(xc_interface *xch,
+ uint32_t domid, char *path);
+
#if defined(__i386__) || defined(__x86_64__)
/*
* PC BIOS standard E820 types and structure.
diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
index 61d0a82f48..35c611ac73 100644
--- a/tools/include/xenguest.h
+++ b/tools/include/xenguest.h
@@ -242,6 +242,8 @@ struct xc_dom_image {
/* Number of vCPUs */
unsigned int max_vcpus;
+
+ xen_pfn_t sci_shmem_gfn;
};
/* --- arch specific hooks ----------------------------------------- */
diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
index b155d6afd2..07bb390e17 100644
--- a/tools/libs/ctrl/xc_domain.c
+++ b/tools/libs/ctrl/xc_domain.c
@@ -2206,6 +2206,29 @@ int xc_domain_soft_reset(xc_interface *xch,
domctl.domain = domid;
return do_domctl(xch, &domctl);
}
+
+int xc_domain_add_sci_device(xc_interface *xch,
+ uint32_t domid, char *path)
+{
+ size_t size = strlen(path);
+ int rc;
+ DECLARE_DOMCTL;
+ DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+ if ( xc_hypercall_bounce_pre(xch, path) )
+ return -1;
+
+ domctl.cmd = XEN_DOMCTL_add_sci_device;
+ domctl.domain = domid;
+ domctl.u.sci_device_op.size = size;
+ set_xen_guest_handle(domctl.u.sci_device_op.path, path);
+ rc = do_domctl(xch, &domctl);
+
+ xc_hypercall_bounce_post(xch, path);
+
+ return rc;
+}
+
/*
* Local variables:
* mode: C
diff --git a/tools/libs/guest/xg_dom_arm.c b/tools/libs/guest/xg_dom_arm.c
index 5e3b76355e..368a670c46 100644
--- a/tools/libs/guest/xg_dom_arm.c
+++ b/tools/libs/guest/xg_dom_arm.c
@@ -25,11 +25,12 @@
#include "xg_private.h"
-#define NR_MAGIC_PAGES 4
+#define NR_MAGIC_PAGES 5
#define CONSOLE_PFN_OFFSET 0
#define XENSTORE_PFN_OFFSET 1
#define MEMACCESS_PFN_OFFSET 2
#define VUART_PFN_OFFSET 3
+#define SCI_SHMEM_OFFSET 4
#define LPAE_SHIFT 9
@@ -69,11 +70,13 @@ static int alloc_magic_pages(struct xc_dom_image *dom)
dom->console_pfn = base + CONSOLE_PFN_OFFSET;
dom->xenstore_pfn = base + XENSTORE_PFN_OFFSET;
dom->vuart_gfn = base + VUART_PFN_OFFSET;
+ dom->sci_shmem_gfn = base + SCI_SHMEM_OFFSET;
xc_clear_domain_page(dom->xch, dom->guest_domid, dom->console_pfn);
xc_clear_domain_page(dom->xch, dom->guest_domid, dom->xenstore_pfn);
xc_clear_domain_page(dom->xch, dom->guest_domid, base + MEMACCESS_PFN_OFFSET);
xc_clear_domain_page(dom->xch, dom->guest_domid, dom->vuart_gfn);
+ xc_clear_domain_page(dom->xch, dom->guest_domid, dom->sci_shmem_gfn);
xc_hvm_param_set(dom->xch, dom->guest_domid, HVM_PARAM_CONSOLE_PFN,
dom->console_pfn);
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index eef1de0939..73ef591822 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -8,6 +8,8 @@
#include <assert.h>
#include <xen/device_tree_defs.h>
+#define SCMI_NODE_PATH "/firmware/scmi"
+
static const char *gicv_to_string(libxl_gic_version gic_version)
{
switch (gic_version) {
@@ -101,6 +103,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
return ERROR_FAIL;
}
+ switch (d_config->b_info.sci) {
+ case LIBXL_SCI_TYPE_NONE:
+ config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_NONE;
+ break;
+ case LIBXL_SCI_TYPE_SCMI_SMC:
+ config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_SCMI_SMC;
+ break;
+ default:
+ LOG(ERROR, "Unknown SCI type %d",
+ d_config->b_info.sci);
+ return ERROR_FAIL;
+ }
+
return 0;
}
@@ -122,6 +137,7 @@ int libxl__arch_domain_save_config(libxl__gc *gc,
}
state->clock_frequency = config->arch.clock_frequency;
+ state->sci_agent_paddr = config->arch.sci_agent_paddr;
return 0;
}
@@ -502,9 +518,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
int res;
LOG(DEBUG, "Creating OP-TEE node in dtb");
- res = fdt_begin_node(fdt, "firmware");
- if (res) return res;
-
res = fdt_begin_node(fdt, "optee");
if (res) return res;
@@ -517,9 +530,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
res = fdt_end_node(fdt);
if (res) return res;
- res = fdt_end_node(fdt);
- if (res) return res;
-
return 0;
}
@@ -902,10 +912,9 @@ static int copy_node(libxl__gc *gc, void *fdt, void *pfdt,
return 0;
}
-static int copy_node_by_path(libxl__gc *gc, const char *path,
- void *fdt, void *pfdt)
+static int get_path_nodeoff(const char *path, void *pfdt)
{
- int nodeoff, r;
+ int nodeoff;
const char *name = strrchr(path, '/');
if (!name)
@@ -925,12 +934,101 @@ static int copy_node_by_path(libxl__gc *gc, const char *path,
if (strcmp(fdt_get_name(pfdt, nodeoff, NULL), name))
return -FDT_ERR_NOTFOUND;
+ return nodeoff;
+}
+
+static int copy_node_by_path(libxl__gc *gc, const char *path,
+ void *fdt, void *pfdt)
+{
+ int nodeoff, r;
+
+ nodeoff = get_path_nodeoff(path, pfdt);
+ if (nodeoff < 0)
+ return nodeoff;
+
r = copy_node(gc, fdt, pfdt, nodeoff, 0);
if (r) return r;
return 0;
}
+static int get_node_phandle(const char *path, void *pfdt, uint32_t *phandle)
+{
+ int nodeoff;
+ const char *name = strrchr(path, '/');
+
+ if (!name)
+ return -FDT_ERR_INTERNAL;
+
+ name++;
+ nodeoff = fdt_path_offset(pfdt, path);
+ if (nodeoff < 0)
+ return nodeoff;
+
+ *phandle = fdt_get_phandle(pfdt, nodeoff);
+ return 0;
+}
+
+static int make_scmi_shmem_node(libxl__gc *gc, void *fdt, void *pfdt,
+ struct xc_dom_image *dom)
+{
+ int res;
+ char buf[64];
+ uint32_t phandle = 0;
+
+ res = get_node_phandle("/scp-shmem", pfdt, &phandle);
+ if (res) return res;
+
+ snprintf(buf, sizeof(buf), "scp-shmem@%lx",
+ dom->sci_shmem_gfn << XC_PAGE_SHIFT);
+ res = fdt_begin_node(fdt, buf);
+ if (res) return res;
+
+ res = fdt_property_compat(gc, fdt, 1, "arm,scmi-shmem");
+ if (res) return res;
+
+ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 1,
+ dom->sci_shmem_gfn << XC_PAGE_SHIFT, XC_PAGE_SHIFT);
+ if (res) return res;
+
+ LOG(DEBUG, "scmi: setting phandle = %u\n", phandle);
+ res = fdt_property_cell(fdt, "phandle", phandle);
+ if (res) return res;
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
+static int make_firmware_node(libxl__gc *gc, void *fdt, void *pfdt, int tee,
+ int sci)
+{
+ int res;
+
+ if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci != LIBXL_SCI_TYPE_NONE))
+ return 0;
+
+ res = fdt_begin_node(fdt, "firmware");
+ if (res) return res;
+
+ if (tee == LIBXL_TEE_TYPE_OPTEE) {
+ res = make_optee_node(gc, fdt);
+ if (res) return res;
+ }
+
+ if (sci == LIBXL_SCI_TYPE_SCMI_SMC) {
+ res = copy_node_by_path(gc, SCMI_NODE_PATH, fdt, pfdt);
+ if (res) return res;
+ }
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
/*
* The partial device tree is not copied entirely. Only the relevant bits are
* copied to the guest device tree:
@@ -1088,8 +1186,10 @@ next_resize:
if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART)
FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) );
- if (info->tee == LIBXL_TEE_TYPE_OPTEE)
- FDT( make_optee_node(gc, fdt) );
+ FDT( make_firmware_node(gc, fdt, pfdt, info->tee, info->sci) );
+
+ if (info->sci == LIBXL_SCI_TYPE_SCMI_SMC)
+ FDT( make_scmi_shmem_node(gc, fdt, pfdt, dom) );
if (d_config->num_pcidevs)
FDT( make_vpci_node(gc, fdt, ainfo, dom) );
diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
index dcd09d32ba..c7372fd344 100644
--- a/tools/libs/light/libxl_create.c
+++ b/tools/libs/light/libxl_create.c
@@ -596,6 +596,37 @@ out:
return ret;
}
+static int map_sci_page(libxl__gc *gc, uint32_t domid, uint64_t paddr,
+ uint64_t guest_addr)
+{
+ int ret;
+ uint64_t _paddr_pfn = paddr >> XC_PAGE_SHIFT;
+ uint64_t _guest_pfn = guest_addr >> XC_PAGE_SHIFT;
+
+ LOG(DEBUG, "iomem %"PRIx64, _paddr_pfn);
+
+ ret = xc_domain_iomem_permission(CTX->xch, domid, _paddr_pfn, 1, 1);
+ if (ret < 0) {
+ LOG(ERROR,
+ "failed give domain access to iomem page %"PRIx64,
+ _paddr_pfn);
+ return ret;
+ }
+
+ ret = xc_domain_memory_mapping(CTX->xch, domid,
+ _guest_pfn, _paddr_pfn,
+ 1, 1);
+ if (ret < 0) {
+ LOG(ERROR,
+ "failed to map to domain iomem page %"PRIx64
+ " to guest address %"PRIx64,
+ _paddr_pfn, _guest_pfn);
+ return ret;
+ }
+
+ return 0;
+}
+
int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
libxl__domain_build_state *state,
uint32_t *domid, bool soft_reset)
@@ -762,6 +793,16 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
goto out;
}
+ if (state->sci_agent_paddr != 0) {
+ ret = map_sci_page(gc, *domid, state->sci_agent_paddr,
+ state->sci_shmem_gfn << XC_PAGE_SHIFT);
+ if (ret < 0) {
+ LOGED(ERROR, *domid, "map SCI page fail");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
dom_path = libxl__xs_get_dompath(gc, *domid);
if (!dom_path) {
rc = ERROR_FAIL;
@@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
{
AO_GC;
libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
- int i, rc = 0;
+ int i, rc = 0, rc_sci = 0;
for (i = 0; i < d_config->num_dtdevs; i++) {
const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
- if (rc < 0) {
- LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
+ rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
+
+ if ((rc < 0) && (rc_sci < 0)) {
+ LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
+ "xc_domain_add_sci_device failed: %d",
+ rc, rc_sci);
goto out;
}
+
+ if (rc)
+ rc = rc_sci;
}
out:
diff --git a/tools/libs/light/libxl_dom.c b/tools/libs/light/libxl_dom.c
index fe9f760f71..b1d288a8b9 100644
--- a/tools/libs/light/libxl_dom.c
+++ b/tools/libs/light/libxl_dom.c
@@ -710,6 +710,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
state->console_mfn = dom->console_pfn;
state->store_mfn = dom->xenstore_pfn;
state->vuart_gfn = dom->vuart_gfn;
+ state->sci_shmem_gfn = dom->sci_shmem_gfn;
} else {
state->console_mfn = xc_dom_p2m(dom, dom->console_pfn);
state->store_mfn = xc_dom_p2m(dom, dom->xenstore_pfn);
diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
index 0b4671318c..f9f9cc633a 100644
--- a/tools/libs/light/libxl_internal.h
+++ b/tools/libs/light/libxl_internal.h
@@ -1407,6 +1407,10 @@ typedef struct {
/* Whether this domain is being migrated/restored, or booting fresh. Only
* applicable to the primary domain, not support domains (e.g. stub QEMU). */
bool restore;
+
+ /* sci channel paddr to be set to device-tree node */
+ uint64_t sci_agent_paddr;
+ xen_pfn_t sci_shmem_gfn;
} libxl__domain_build_state;
_hidden void libxl__domain_build_state_init(libxl__domain_build_state *s);
diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
index 6245af6d0b..ba200407da 100644
--- a/xen/arch/arm/domctl.c
+++ b/xen/arch/arm/domctl.c
@@ -4,6 +4,7 @@
* Copyright (c) 2012, Citrix Systems
*/
+#include <asm/sci/sci.h>
#include <xen/errno.h>
#include <xen/guest_access.h>
#include <xen/hypercall.h>
@@ -175,6 +176,20 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
return rc;
}
+ case XEN_DOMCTL_add_sci_device:
+ {
+ int rc;
+ struct dt_device_node *dev;
+
+ rc = dt_find_node_by_gpath(domctl->u.sci_device_op.path,
+ domctl->u.sci_device_op.size,
+ &dev);
+ if ( rc )
+ return rc;
+
+ return sci_add_dt_device(d, dev);
+ }
+
default:
{
int rc;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index b85e6170b0..671c72c3e8 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
#define XEN_DOMCTL_vmtrace_get_option 5
#define XEN_DOMCTL_vmtrace_set_option 6
};
+
+/* XEN_DOMCTL_add_sci_device: set sci device permissions */
+struct xen_domctl_sci_device_op {
+ uint32_t size; /* Length of the path */
+ XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
+};
+
typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
@@ -1265,6 +1272,7 @@ struct xen_domctl {
#define XEN_DOMCTL_get_cpu_policy 82
#define XEN_DOMCTL_set_cpu_policy 83
#define XEN_DOMCTL_vmtrace_op 84
+#define XEN_DOMCTL_add_sci_device 85
#define XEN_DOMCTL_gdbsx_guestmemio 1000
#define XEN_DOMCTL_gdbsx_pausevcpu 1001
#define XEN_DOMCTL_gdbsx_unpausevcpu 1002
@@ -1326,6 +1334,7 @@ struct xen_domctl {
struct xen_domctl_psr_alloc psr_alloc;
struct xen_domctl_vuart_op vuart_op;
struct xen_domctl_vmtrace_op vmtrace_op;
+ struct xen_domctl_sci_device_op sci_device_op;
uint8_t pad[128];
} u;
};
--
2.27.0
^ permalink raw reply related [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
@ 2021-12-14 9:41 ` Jan Beulich
2021-12-16 17:36 ` Oleksii Moisieiev
2021-12-16 0:04 ` Oleksandr
2021-12-21 1:37 ` Stefano Stabellini
2 siblings, 1 reply; 95+ messages in thread
From: Jan Beulich @ 2021-12-14 9:41 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis, xen-devel
On 14.12.2021 10:34, Oleksii Moisieiev wrote:
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> #define XEN_DOMCTL_vmtrace_get_option 5
> #define XEN_DOMCTL_vmtrace_set_option 6
> };
> +
> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> +struct xen_domctl_sci_device_op {
> + uint32_t size; /* Length of the path */
> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> +};
This being - aiui - Arm-only, please enclose it by respective #if,
just like we do for certain x86-only ops.
I'm further afraid the term "SCI" is ambiguous with ACPI's System
Control Interrupt, so there's some further tag needed in the names
used here.
Finally please make padding explicit and check that it's zero on
input.
Jan
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
@ 2021-12-15 6:38 ` Oleksandr Andrushchenko
2021-12-15 20:08 ` Oleksii Moisieiev
2021-12-15 9:39 ` Julien Grall
2021-12-15 9:57 ` Oleksandr Tyshchenko
2 siblings, 1 reply; 95+ messages in thread
From: Oleksandr Andrushchenko @ 2021-12-15 6:38 UTC (permalink / raw)
To: Oleksii Moisieiev, xen-devel
Cc: Stefano Stabellini, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi, Oleksii!
On 14.12.21 11:34, Oleksii Moisieiev wrote:
> Implementation includes platform-specific smc handler for rcar3 platform.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
Reviewed-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> ---
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
> 2 files changed, 47 insertions(+)
> create mode 100644 xen/arch/arm/platforms/rcar3.c
>
> diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
> index 8632f4115f..b64c25de6c 100644
> --- a/xen/arch/arm/platforms/Makefile
> +++ b/xen/arch/arm/platforms/Makefile
> @@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
> obj-$(CONFIG_ALL32_PLAT) += midway.o
> obj-$(CONFIG_ALL32_PLAT) += omap5.o
> obj-$(CONFIG_ALL32_PLAT) += rcar2.o
> +obj-$(CONFIG_RCAR3) += rcar3.o
> obj-$(CONFIG_ALL64_PLAT) += seattle.o
> obj-$(CONFIG_ALL_PLAT) += sunxi.o
> obj-$(CONFIG_ALL64_PLAT) += thunderx.o
> diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
> new file mode 100644
> index 0000000000..d740145c71
> --- /dev/null
> +++ b/xen/arch/arm/platforms/rcar3.c
> @@ -0,0 +1,46 @@
> +/*
> + * xen/arch/arm/platforms/rcar3.c
> + *
> + * Renesas R-Car Gen3 specific settings
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021 EPAM Systems
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <asm/platform.h>
> +
> +static bool rcar3_smc(struct cpu_user_regs *regs)
> +{
> + return false;
> +}
> +
> +static const char *const rcar3_dt_compat[] __initconst =
> +{
> + "renesas,r8a7795",
> + "renesas,r8a7796",
> + NULL
> +};
> +
> +PLATFORM_START(rcar3, "Renesas R-Car Gen3")
> + .compatible = rcar3_dt_compat,
> + .smc = rcar3_smc
> +PLATFORM_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
2021-12-15 6:38 ` Oleksandr Andrushchenko
@ 2021-12-15 9:39 ` Julien Grall
2021-12-17 10:48 ` Oleksii Moisieiev
2021-12-15 9:57 ` Oleksandr Tyshchenko
2 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-15 9:39 UTC (permalink / raw)
To: Oleksii Moisieiev, xen-devel
Cc: Stefano Stabellini, Volodymyr Babchuk, Bertrand Marquis
Hi,
On 14/12/2021 09:34, Oleksii Moisieiev wrote:
> Implementation includes platform-specific smc handler for rcar3 platform.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
> 2 files changed, 47 insertions(+)
> create mode 100644 xen/arch/arm/platforms/rcar3.c
>
> diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
> index 8632f4115f..b64c25de6c 100644
> --- a/xen/arch/arm/platforms/Makefile
> +++ b/xen/arch/arm/platforms/Makefile
> @@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
> obj-$(CONFIG_ALL32_PLAT) += midway.o
> obj-$(CONFIG_ALL32_PLAT) += omap5.o
> obj-$(CONFIG_ALL32_PLAT) += rcar2.o
> +obj-$(CONFIG_RCAR3) += rcar3.o
> obj-$(CONFIG_ALL64_PLAT) += seattle.o
> obj-$(CONFIG_ALL_PLAT) += sunxi.o
> obj-$(CONFIG_ALL64_PLAT) += thunderx.o
> diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
> new file mode 100644
> index 0000000000..d740145c71
> --- /dev/null
> +++ b/xen/arch/arm/platforms/rcar3.c
> @@ -0,0 +1,46 @@
> +/*
> + * xen/arch/arm/platforms/rcar3.c
> + *
> + * Renesas R-Car Gen3 specific settings
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021 EPAM Systems
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <asm/platform.h>
> +
> +static bool rcar3_smc(struct cpu_user_regs *regs)
> +{
> + return false;
> +}
Looking at the rest of the series, this is going to be replaced in patch
#2 with:
return sci_handle_call();
SCMI is not specific to RCAR3. So I would expect the function to be
called from common code.
If it still needs some platform specific code, then I think it would be
best to introduce rcar3.c at the end of the series. So we don't
introduce a dummy platform and not hook the code in the middle of
patch#2 which is meant to be generic.
I will have a proper review of the rest of the series in the new year.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
2021-12-15 6:38 ` Oleksandr Andrushchenko
2021-12-15 9:39 ` Julien Grall
@ 2021-12-15 9:57 ` Oleksandr Tyshchenko
2021-12-17 10:52 ` Oleksii Moisieiev
2 siblings, 1 reply; 95+ messages in thread
From: Oleksandr Tyshchenko @ 2021-12-15 9:57 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 2861 bytes --]
On Tue, Dec 14, 2021 at 11:35 AM Oleksii Moisieiev <
Oleksii_Moisieiev@epam.com> wrote:
Hi Oleksii
[sorry for the possible format issues]
Implementation includes platform-specific smc handler for rcar3 platform.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
> 2 files changed, 47 insertions(+)
> create mode 100644 xen/arch/arm/platforms/rcar3.c
>
> diff --git a/xen/arch/arm/platforms/Makefile
> b/xen/arch/arm/platforms/Makefile
> index 8632f4115f..b64c25de6c 100644
> --- a/xen/arch/arm/platforms/Makefile
> +++ b/xen/arch/arm/platforms/Makefile
> @@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
> obj-$(CONFIG_ALL32_PLAT) += midway.o
> obj-$(CONFIG_ALL32_PLAT) += omap5.o
> obj-$(CONFIG_ALL32_PLAT) += rcar2.o
> +obj-$(CONFIG_RCAR3) += rcar3.o
> obj-$(CONFIG_ALL64_PLAT) += seattle.o
> obj-$(CONFIG_ALL_PLAT) += sunxi.o
> obj-$(CONFIG_ALL64_PLAT) += thunderx.o
> diff --git a/xen/arch/arm/platforms/rcar3.c
> b/xen/arch/arm/platforms/rcar3.c
> new file mode 100644
> index 0000000000..d740145c71
> --- /dev/null
> +++ b/xen/arch/arm/platforms/rcar3.c
> @@ -0,0 +1,46 @@
> +/*
> + * xen/arch/arm/platforms/rcar3.c
> + *
> + * Renesas R-Car Gen3 specific settings
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021 EPAM Systems
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <asm/platform.h>
> +
> +static bool rcar3_smc(struct cpu_user_regs *regs)
> +{
> + return false;
> +}
> +
> +static const char *const rcar3_dt_compat[] __initconst =
> +{
> + "renesas,r8a7795",
> + "renesas,r8a7796",
>
Please note that since Linux commit:
"9c9f7891093b02eb64ca4e1c7ab776a4296c058f soc: renesas: Identify R-Car
M3-W+"
the compatible string for R-Car M3-W+ (ES3.0) SoC is "renesas,r8a77961". So
in case we want to have vSCMI feature on this new SoC revision as well we
will need
to extend the compatible list.
+ NULL
> +};
> +
> +PLATFORM_START(rcar3, "Renesas R-Car Gen3")
> + .compatible = rcar3_dt_compat,
> + .smc = rcar3_smc
> +PLATFORM_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> --
> 2.27.0
>
>
--
Regards,
Oleksandr Tyshchenko
[-- Attachment #2: Type: text/html, Size: 4467 bytes --]
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-15 6:38 ` Oleksandr Andrushchenko
@ 2021-12-15 20:08 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-15 20:08 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
Hi Oleksandr,
Thank you for the point.
I will add it to the next patch version.
Best regards,
Oleksii
On Wed, Dec 15, 2021 at 06:38:00AM +0000, Oleksandr Andrushchenko wrote:
> Hi, Oleksii!
>
> On 14.12.21 11:34, Oleksii Moisieiev wrote:
> > Implementation includes platform-specific smc handler for rcar3 platform.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> Reviewed-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>
> > ---
> > xen/arch/arm/platforms/Makefile | 1 +
> > xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
> > 2 files changed, 47 insertions(+)
> > create mode 100644 xen/arch/arm/platforms/rcar3.c
> >
> > diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
> > index 8632f4115f..b64c25de6c 100644
> > --- a/xen/arch/arm/platforms/Makefile
> > +++ b/xen/arch/arm/platforms/Makefile
> > @@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
> > obj-$(CONFIG_ALL32_PLAT) += midway.o
> > obj-$(CONFIG_ALL32_PLAT) += omap5.o
> > obj-$(CONFIG_ALL32_PLAT) += rcar2.o
> > +obj-$(CONFIG_RCAR3) += rcar3.o
> > obj-$(CONFIG_ALL64_PLAT) += seattle.o
> > obj-$(CONFIG_ALL_PLAT) += sunxi.o
> > obj-$(CONFIG_ALL64_PLAT) += thunderx.o
> > diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
> > new file mode 100644
> > index 0000000000..d740145c71
> > --- /dev/null
> > +++ b/xen/arch/arm/platforms/rcar3.c
> > @@ -0,0 +1,46 @@
> > +/*
> > + * xen/arch/arm/platforms/rcar3.c
> > + *
> > + * Renesas R-Car Gen3 specific settings
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021 EPAM Systems
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <asm/platform.h>
> > +
> > +static bool rcar3_smc(struct cpu_user_regs *regs)
> > +{
> > + return false;
> > +}
> > +
> > +static const char *const rcar3_dt_compat[] __initconst =
> > +{
> > + "renesas,r8a7795",
> > + "renesas,r8a7796",
> > + NULL
> > +};
> > +
> > +PLATFORM_START(rcar3, "Renesas R-Car Gen3")
> > + .compatible = rcar3_dt_compat,
> > + .smc = rcar3_smc
> > +PLATFORM_END
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
@ 2021-12-15 21:51 ` Oleksandr
2021-12-17 11:00 ` Oleksii Moisieiev
2021-12-21 0:54 ` Stefano Stabellini
2021-12-21 13:27 ` Anthony PERARD
2 siblings, 1 reply; 95+ messages in thread
From: Oleksandr @ 2021-12-15 21:51 UTC (permalink / raw)
To: Oleksii Moisieiev; +Cc: xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
On 14.12.21 11:34, Oleksii Moisieiev wrote:
Hi Oleksii
> This enumeration sets SCI type for the domain. Currently there is
> two possible options: either 'none' or 'scmi_smc'.
>
> 'none' is the default value and it disables SCI support at all.
>
> 'scmi_smc' enables access to the Firmware from the domains using SCMI
> protocol and SMC as transport.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> tools/include/libxl.h | 5 +++++
> tools/libs/light/libxl_types.idl | 6 ++++++
> tools/xl/xl_parse.c | 9 +++++++++
> 4 files changed, 42 insertions(+)
>
> diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> index b98d161398..92d0593875 100644
> --- a/docs/man/xl.cfg.5.pod.in
> +++ b/docs/man/xl.cfg.5.pod.in
> @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
>
> =back
>
> +=item B<sci="STRING">
> +
> +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> +allows domain to manage various functions that are provided by HW platform.
> +
> +=over 4
> +
> +=item B<none>
> +
> +Don't allow guest to use SCI if present on the platform. This is the default
> +value.
> +
> +=item B<scmi_smc>
> +
> +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> +Inferface - allows domain to manage various functions that are provided by HW
> +platform, such as clocks, resets and power-domains. Xen will mediate access to
> +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> +SMC is used as transport.
> +
> +=back
> +
> =back
>
> =head2 Paravirtualised (PV) Guest Specific Options
> diff --git a/tools/include/libxl.h b/tools/include/libxl.h
> index 2bbbd21f0b..30e5aee119 100644
> --- a/tools/include/libxl.h
> +++ b/tools/include/libxl.h
> @@ -278,6 +278,11 @@
> */
> #define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE 1
>
> +/*
> + * libxl_domain_build_info has the arch_arm.sci field.
> + */
> +#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_SCI 1
> +
> /*
> * LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing
> * 'soft reset' for domains and there is 'soft_reset' shutdown reason
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 2a42da2f7d..9067b509f4 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
I assume that at least goland bindings want updating.
> @@ -480,6 +480,11 @@ libxl_tee_type = Enumeration("tee_type", [
> (1, "optee")
> ], init_val = "LIBXL_TEE_TYPE_NONE")
>
> +libxl_sci_type = Enumeration("sci_type", [
> + (0, "none"),
> + (1, "scmi_smc")
> + ], init_val = "LIBXL_SCI_TYPE_NONE")
> +
> libxl_rdm_reserve = Struct("rdm_reserve", [
> ("strategy", libxl_rdm_reserve_strategy),
> ("policy", libxl_rdm_reserve_policy),
> @@ -564,6 +569,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
> ("apic", libxl_defbool),
> ("dm_restrict", libxl_defbool),
> ("tee", libxl_tee_type),
> + ("sci", libxl_sci_type),
> ("u", KeyedUnion(None, libxl_domain_type, "type",
> [("hvm", Struct(None, [("firmware", string),
> ("bios", libxl_bios_type),
> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
> index 117fcdcb2b..c37bf6298b 100644
> --- a/tools/xl/xl_parse.c
> +++ b/tools/xl/xl_parse.c
> @@ -2747,6 +2747,15 @@ skip_usbdev:
> }
> }
>
> + if (!xlu_cfg_get_string (config, "sci", &buf, 1)) {
> + e = libxl_sci_type_from_string(buf, &b_info->sci);
> + if (e) {
> + fprintf(stderr,
> + "Unknown sci \"%s\" specified\n", buf);
> + exit(-ERROR_FAIL);
> + }
> + }
> +
> parse_vkb_list(config, d_config);
>
> xlu_cfg_get_defbool(config, "xend_suspend_evtchn_compat",
--
Regards,
Oleksandr Tyshchenko
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
2021-12-14 9:41 ` Jan Beulich
@ 2021-12-16 0:04 ` Oleksandr
2021-12-17 12:15 ` Oleksii Moisieiev
2021-12-21 1:37 ` Stefano Stabellini
2 siblings, 1 reply; 95+ messages in thread
From: Oleksandr @ 2021-12-16 0:04 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis
On 14.12.21 11:34, Oleksii Moisieiev wrote:
Hi Oleksii
> Integration of the SCMI mediator with xen libs:
> - add hypercalls to aquire SCI channel and set device permissions
> for DomUs;
> - add SCMI_SMC nodes to DomUs device-tree based on partial device-tree;
> - SCI requests redirection from DomUs to Firmware.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> tools/include/xenctrl.h | 3 +
> tools/include/xenguest.h | 2 +
> tools/libs/ctrl/xc_domain.c | 23 ++++++
> tools/libs/guest/xg_dom_arm.c | 5 +-
> tools/libs/light/libxl_arm.c | 122 +++++++++++++++++++++++++++---
> tools/libs/light/libxl_create.c | 54 ++++++++++++-
> tools/libs/light/libxl_dom.c | 1 +
> tools/libs/light/libxl_internal.h | 4 +
> xen/arch/arm/domctl.c | 15 ++++
> xen/include/public/domctl.h | 9 +++
> 10 files changed, 223 insertions(+), 15 deletions(-)
>
> diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
> index 07b96e6671..cdd14f465f 100644
> --- a/tools/include/xenctrl.h
> +++ b/tools/include/xenctrl.h
> @@ -1238,6 +1238,9 @@ int xc_domain_getvnuma(xc_interface *xch,
> int xc_domain_soft_reset(xc_interface *xch,
> uint32_t domid);
>
> +int xc_domain_add_sci_device(xc_interface *xch,
> + uint32_t domid, char *path);
> +
> #if defined(__i386__) || defined(__x86_64__)
> /*
> * PC BIOS standard E820 types and structure.
> diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
> index 61d0a82f48..35c611ac73 100644
> --- a/tools/include/xenguest.h
> +++ b/tools/include/xenguest.h
> @@ -242,6 +242,8 @@ struct xc_dom_image {
>
> /* Number of vCPUs */
> unsigned int max_vcpus;
> +
> + xen_pfn_t sci_shmem_gfn;
> };
>
> /* --- arch specific hooks ----------------------------------------- */
> diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
> index b155d6afd2..07bb390e17 100644
> --- a/tools/libs/ctrl/xc_domain.c
> +++ b/tools/libs/ctrl/xc_domain.c
> @@ -2206,6 +2206,29 @@ int xc_domain_soft_reset(xc_interface *xch,
> domctl.domain = domid;
> return do_domctl(xch, &domctl);
> }
> +
> +int xc_domain_add_sci_device(xc_interface *xch,
> + uint32_t domid, char *path)
> +{
> + size_t size = strlen(path);
> + int rc;
> + DECLARE_DOMCTL;
> + DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
> +
> + if ( xc_hypercall_bounce_pre(xch, path) )
> + return -1;
> +
> + domctl.cmd = XEN_DOMCTL_add_sci_device;
> + domctl.domain = domid;
> + domctl.u.sci_device_op.size = size;
> + set_xen_guest_handle(domctl.u.sci_device_op.path, path);
> + rc = do_domctl(xch, &domctl);
> +
> + xc_hypercall_bounce_post(xch, path);
> +
> + return rc;
> +}
> +
> /*
> * Local variables:
> * mode: C
> diff --git a/tools/libs/guest/xg_dom_arm.c b/tools/libs/guest/xg_dom_arm.c
> index 5e3b76355e..368a670c46 100644
> --- a/tools/libs/guest/xg_dom_arm.c
> +++ b/tools/libs/guest/xg_dom_arm.c
> @@ -25,11 +25,12 @@
>
> #include "xg_private.h"
>
> -#define NR_MAGIC_PAGES 4
> +#define NR_MAGIC_PAGES 5
> #define CONSOLE_PFN_OFFSET 0
> #define XENSTORE_PFN_OFFSET 1
> #define MEMACCESS_PFN_OFFSET 2
> #define VUART_PFN_OFFSET 3
> +#define SCI_SHMEM_OFFSET 4
>
> #define LPAE_SHIFT 9
>
> @@ -69,11 +70,13 @@ static int alloc_magic_pages(struct xc_dom_image *dom)
> dom->console_pfn = base + CONSOLE_PFN_OFFSET;
> dom->xenstore_pfn = base + XENSTORE_PFN_OFFSET;
> dom->vuart_gfn = base + VUART_PFN_OFFSET;
> + dom->sci_shmem_gfn = base + SCI_SHMEM_OFFSET;
>
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->console_pfn);
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->xenstore_pfn);
> xc_clear_domain_page(dom->xch, dom->guest_domid, base + MEMACCESS_PFN_OFFSET);
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->vuart_gfn);
> + xc_clear_domain_page(dom->xch, dom->guest_domid, dom->sci_shmem_gfn);
>
> xc_hvm_param_set(dom->xch, dom->guest_domid, HVM_PARAM_CONSOLE_PFN,
> dom->console_pfn);
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index eef1de0939..73ef591822 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -8,6 +8,8 @@
> #include <assert.h>
> #include <xen/device_tree_defs.h>
>
> +#define SCMI_NODE_PATH "/firmware/scmi"
> +
> static const char *gicv_to_string(libxl_gic_version gic_version)
> {
> switch (gic_version) {
> @@ -101,6 +103,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
> return ERROR_FAIL;
> }
>
> + switch (d_config->b_info.sci) {
> + case LIBXL_SCI_TYPE_NONE:
> + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_NONE;
> + break;
> + case LIBXL_SCI_TYPE_SCMI_SMC:
> + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_SCMI_SMC;
> + break;
> + default:
> + LOG(ERROR, "Unknown SCI type %d",
> + d_config->b_info.sci);
> + return ERROR_FAIL;
> + }
> +
> return 0;
> }
>
> @@ -122,6 +137,7 @@ int libxl__arch_domain_save_config(libxl__gc *gc,
> }
>
> state->clock_frequency = config->arch.clock_frequency;
> + state->sci_agent_paddr = config->arch.sci_agent_paddr;
>
> return 0;
> }
> @@ -502,9 +518,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> int res;
> LOG(DEBUG, "Creating OP-TEE node in dtb");
>
> - res = fdt_begin_node(fdt, "firmware");
> - if (res) return res;
> -
> res = fdt_begin_node(fdt, "optee");
> if (res) return res;
>
> @@ -517,9 +530,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> res = fdt_end_node(fdt);
> if (res) return res;
>
> - res = fdt_end_node(fdt);
> - if (res) return res;
> -
> return 0;
> }
>
> @@ -902,10 +912,9 @@ static int copy_node(libxl__gc *gc, void *fdt, void *pfdt,
> return 0;
> }
>
> -static int copy_node_by_path(libxl__gc *gc, const char *path,
> - void *fdt, void *pfdt)
> +static int get_path_nodeoff(const char *path, void *pfdt)
> {
> - int nodeoff, r;
> + int nodeoff;
> const char *name = strrchr(path, '/');
>
> if (!name)
> @@ -925,12 +934,101 @@ static int copy_node_by_path(libxl__gc *gc, const char *path,
> if (strcmp(fdt_get_name(pfdt, nodeoff, NULL), name))
> return -FDT_ERR_NOTFOUND;
>
> + return nodeoff;
> +}
> +
> +static int copy_node_by_path(libxl__gc *gc, const char *path,
> + void *fdt, void *pfdt)
> +{
> + int nodeoff, r;
> +
> + nodeoff = get_path_nodeoff(path, pfdt);
> + if (nodeoff < 0)
> + return nodeoff;
> +
> r = copy_node(gc, fdt, pfdt, nodeoff, 0);
> if (r) return r;
>
> return 0;
> }
>
> +static int get_node_phandle(const char *path, void *pfdt, uint32_t *phandle)
> +{
> + int nodeoff;
> + const char *name = strrchr(path, '/');
> +
> + if (!name)
> + return -FDT_ERR_INTERNAL;
> +
> + name++;
> + nodeoff = fdt_path_offset(pfdt, path);
> + if (nodeoff < 0)
> + return nodeoff;
> +
> + *phandle = fdt_get_phandle(pfdt, nodeoff);
> + return 0;
> +}
> +
> +static int make_scmi_shmem_node(libxl__gc *gc, void *fdt, void *pfdt,
> + struct xc_dom_image *dom)
> +{
> + int res;
> + char buf[64];
> + uint32_t phandle = 0;
> +
> + res = get_node_phandle("/scp-shmem", pfdt, &phandle);
> + if (res) return res;
> +
> + snprintf(buf, sizeof(buf), "scp-shmem@%lx",
> + dom->sci_shmem_gfn << XC_PAGE_SHIFT);
> + res = fdt_begin_node(fdt, buf);
> + if (res) return res;
> +
> + res = fdt_property_compat(gc, fdt, 1, "arm,scmi-shmem");
> + if (res) return res;
> +
> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> + GUEST_ROOT_SIZE_CELLS, 1,
> + dom->sci_shmem_gfn << XC_PAGE_SHIFT, XC_PAGE_SHIFT);
> + if (res) return res;
> +
> + LOG(DEBUG, "scmi: setting phandle = %u\n", phandle);
> + res = fdt_property_cell(fdt, "phandle", phandle);
> + if (res) return res;
> +
> + res = fdt_end_node(fdt);
> + if (res) return res;
> +
> + return 0;
> +}
> +
> +static int make_firmware_node(libxl__gc *gc, void *fdt, void *pfdt, int tee,
> + int sci)
> +{
> + int res;
> +
> + if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci != LIBXL_SCI_TYPE_NONE))
> + return 0;
> +
> + res = fdt_begin_node(fdt, "firmware");
> + if (res) return res;
> +
> + if (tee == LIBXL_TEE_TYPE_OPTEE) {
> + res = make_optee_node(gc, fdt);
> + if (res) return res;
> + }
> +
> + if (sci == LIBXL_SCI_TYPE_SCMI_SMC) {
> + res = copy_node_by_path(gc, SCMI_NODE_PATH, fdt, pfdt);
> + if (res) return res;
> + }
> +
> + res = fdt_end_node(fdt);
> + if (res) return res;
> +
> + return 0;
> +}
> +
> /*
> * The partial device tree is not copied entirely. Only the relevant bits are
> * copied to the guest device tree:
> @@ -1088,8 +1186,10 @@ next_resize:
> if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART)
> FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) );
>
> - if (info->tee == LIBXL_TEE_TYPE_OPTEE)
> - FDT( make_optee_node(gc, fdt) );
> + FDT( make_firmware_node(gc, fdt, pfdt, info->tee, info->sci) );
> +
> + if (info->sci == LIBXL_SCI_TYPE_SCMI_SMC)
> + FDT( make_scmi_shmem_node(gc, fdt, pfdt, dom) );
>
> if (d_config->num_pcidevs)
> FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
> index dcd09d32ba..c7372fd344 100644
> --- a/tools/libs/light/libxl_create.c
> +++ b/tools/libs/light/libxl_create.c
> @@ -596,6 +596,37 @@ out:
> return ret;
> }
>
> +static int map_sci_page(libxl__gc *gc, uint32_t domid, uint64_t paddr,
> + uint64_t guest_addr)
> +{
> + int ret;
> + uint64_t _paddr_pfn = paddr >> XC_PAGE_SHIFT;
> + uint64_t _guest_pfn = guest_addr >> XC_PAGE_SHIFT;
> +
> + LOG(DEBUG, "iomem %"PRIx64, _paddr_pfn);
> +
> + ret = xc_domain_iomem_permission(CTX->xch, domid, _paddr_pfn, 1, 1);
> + if (ret < 0) {
> + LOG(ERROR,
> + "failed give domain access to iomem page %"PRIx64,
> + _paddr_pfn);
> + return ret;
> + }
> +
> + ret = xc_domain_memory_mapping(CTX->xch, domid,
> + _guest_pfn, _paddr_pfn,
> + 1, 1);
> + if (ret < 0) {
> + LOG(ERROR,
> + "failed to map to domain iomem page %"PRIx64
> + " to guest address %"PRIx64,
> + _paddr_pfn, _guest_pfn);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> libxl__domain_build_state *state,
> uint32_t *domid, bool soft_reset)
> @@ -762,6 +793,16 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> goto out;
> }
>
> + if (state->sci_agent_paddr != 0) {
> + ret = map_sci_page(gc, *domid, state->sci_agent_paddr,
> + state->sci_shmem_gfn << XC_PAGE_SHIFT);
> + if (ret < 0) {
> + LOGED(ERROR, *domid, "map SCI page fail");
> + rc = ERROR_FAIL;
> + goto out;
> + }
> + }
> +
> dom_path = libxl__xs_get_dompath(gc, *domid);
> if (!dom_path) {
> rc = ERROR_FAIL;
> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> {
> AO_GC;
> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> - int i, rc = 0;
> + int i, rc = 0, rc_sci = 0;
>
> for (i = 0; i < d_config->num_dtdevs; i++) {
> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>
> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> - if (rc < 0) {
> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> +
> + if ((rc < 0) && (rc_sci < 0)) {
> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> + "xc_domain_add_sci_device failed: %d",
> + rc, rc_sci);
> goto out;
> }
> +
> + if (rc)
> + rc = rc_sci;
If I get this code right, it sounds like the dom.cfg's dtdev property is
reused to describe sci devices as well, but the libxl__add_dtdevs() does
not (and can not) differentiate them. So it has no option but to send
two domctls for each device in dtdevs[] hoping to at least one domctl to
succeeded. Or I really missed something?
It feels to me that:
- either new dom.cfg's property could be introduced (scidev?) to
describe sci devices together with new parsing logic/management code, so
you will end up having new libxl__add_scidevs() to invoke
XEN_DOMCTL_add_sci_device,
so no mixing things.
- or indeed dtdev logic could be *completely* reused including
extending XEN_DOMCTL_assign_device to cover your use-case, although
sounds generic, it is used to describe devices for the passthrough (to
configure an IOMMU for the device), in that case you wouldn't need an
extra XEN_DOMCTL_add_sci_device introduced by current patch.
Personally I would use the first option as I am not sure that second
option is conceptually correct && possible. I would leave this for the
maintainers to clarify.
> }
>
> out:
> diff --git a/tools/libs/light/libxl_dom.c b/tools/libs/light/libxl_dom.c
> index fe9f760f71..b1d288a8b9 100644
> --- a/tools/libs/light/libxl_dom.c
> +++ b/tools/libs/light/libxl_dom.c
> @@ -710,6 +710,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
> state->console_mfn = dom->console_pfn;
> state->store_mfn = dom->xenstore_pfn;
> state->vuart_gfn = dom->vuart_gfn;
> + state->sci_shmem_gfn = dom->sci_shmem_gfn;
> } else {
> state->console_mfn = xc_dom_p2m(dom, dom->console_pfn);
> state->store_mfn = xc_dom_p2m(dom, dom->xenstore_pfn);
> diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
> index 0b4671318c..f9f9cc633a 100644
> --- a/tools/libs/light/libxl_internal.h
> +++ b/tools/libs/light/libxl_internal.h
> @@ -1407,6 +1407,10 @@ typedef struct {
> /* Whether this domain is being migrated/restored, or booting fresh. Only
> * applicable to the primary domain, not support domains (e.g. stub QEMU). */
> bool restore;
> +
> + /* sci channel paddr to be set to device-tree node */
> + uint64_t sci_agent_paddr;
> + xen_pfn_t sci_shmem_gfn;
> } libxl__domain_build_state;
>
> _hidden void libxl__domain_build_state_init(libxl__domain_build_state *s);
> diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
> index 6245af6d0b..ba200407da 100644
> --- a/xen/arch/arm/domctl.c
> +++ b/xen/arch/arm/domctl.c
> @@ -4,6 +4,7 @@
> * Copyright (c) 2012, Citrix Systems
> */
>
> +#include <asm/sci/sci.h>
> #include <xen/errno.h>
> #include <xen/guest_access.h>
> #include <xen/hypercall.h>
> @@ -175,6 +176,20 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
>
> return rc;
> }
> + case XEN_DOMCTL_add_sci_device:
> + {
> + int rc;
> + struct dt_device_node *dev;
> +
> + rc = dt_find_node_by_gpath(domctl->u.sci_device_op.path,
> + domctl->u.sci_device_op.size,
> + &dev);
> + if ( rc )
> + return rc;
> +
> + return sci_add_dt_device(d, dev);
> + }
> +
> default:
> {
> int rc;
> diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> index b85e6170b0..671c72c3e8 100644
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> #define XEN_DOMCTL_vmtrace_get_option 5
> #define XEN_DOMCTL_vmtrace_set_option 6
> };
> +
> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> +struct xen_domctl_sci_device_op {
> + uint32_t size; /* Length of the path */
> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> +};
> +
> typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
> DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
>
> @@ -1265,6 +1272,7 @@ struct xen_domctl {
> #define XEN_DOMCTL_get_cpu_policy 82
> #define XEN_DOMCTL_set_cpu_policy 83
> #define XEN_DOMCTL_vmtrace_op 84
> +#define XEN_DOMCTL_add_sci_device 85
> #define XEN_DOMCTL_gdbsx_guestmemio 1000
> #define XEN_DOMCTL_gdbsx_pausevcpu 1001
> #define XEN_DOMCTL_gdbsx_unpausevcpu 1002
> @@ -1326,6 +1334,7 @@ struct xen_domctl {
> struct xen_domctl_psr_alloc psr_alloc;
> struct xen_domctl_vuart_op vuart_op;
> struct xen_domctl_vmtrace_op vmtrace_op;
> + struct xen_domctl_sci_device_op sci_device_op;
> uint8_t pad[128];
> } u;
> };
I assume the XSM needs updating (adding new hook, etc).
--
Regards,
Oleksandr Tyshchenko
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 0/5] Introduce SCI-mediator feature
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
` (4 preceding siblings ...)
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
@ 2021-12-16 0:33 ` Oleksandr
2021-12-17 12:24 ` Oleksii Moisieiev
5 siblings, 1 reply; 95+ messages in thread
From: Oleksandr @ 2021-12-16 0:33 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Anthony PERARD, Juergen Gross
On 14.12.21 11:34, Oleksii Moisieiev wrote:
Hi Oleksii
> Introducing the feature, called SCI mediator.
> It's purpose is to redirect SCMI requests from the domains to firmware
> (SCP, ATF etc), which controls the power/clock/resets etc.
> The idea is to make SCP firmware (or similar, such as AT-F) responsible for
> control power/clock/resets and provide SCMI interface so controls can be shared
> between the Domains.
> Originally, we've met a problem, that the devices, shared between different
> Domains, can't have an access to HW registers to work with clocks/resets/power
> etc. You have to pass cpg to the Domain, so the devices can access HW directly.
> The solution for this is to move HW controls over power/clock/resets to
> SCP firmware and use Linux-kernel SCMI drivers to pass requests to SCP.
> Xen is responsible for permissions setting, so Domain can access only to
> power/clock/resets which are related to this Domain. Also XEN is the mediator
> which redirects SCMI requests, adding agentID so firmware should know the
> sender.
> SMC is currently used as transport, but this should be configurable.
>
> Here is the high level design:
>
> SCI (System Control Interface) feature can be enabled in xen_config:
>> CONFIG_SCI=y
> Mediator can be configured:
>> CONFIG_SCMI_SMC=y
> Currently, only SCMI_SMC mediator is implemented, which using shared memory
> region to communicate with firmware and SMC as transport.
>
> Xen scmi should be configured in the device-tree.
> Format is the following:
> cpu_scp_shm: scp-shmem@0x53FF0000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF0000 0x0 0x1000>;
> };
>
> firmware {
> scmi {
> compatible = "arm,scmi-smc";
> arm,smc-id = <0x82000002>;
> shmem = <&cpu_scp_shm>;
> #address-cells = <1>;
> #size-cells = <0>;
>
> scmi_power: protocol@11 {
> reg = <0x11>;
> #power-domain-cells = <1>;
> };
>
> scmi_clock: protocol@14 {
> reg = <0x14>;
> #clock-cells = <1>;
> };
>
> scmi_reset: protocol@16 {
> reg = <0x16>;
> #reset-cells = <1>;
> };
> };
> };
>
> Where:
> &cpu_scp_shm is the shared memory for scmi buffers;
> 0x53FF0000, size 0x1000 is the platform specific free address, which provide
> space for the communication.
> &scmi node, which should be copied to Dom0 device-tree.
>
> Device configured to use scmi:
> &avb {
> scmi_devid = <0>;
> clocks = <&scmi_clock 0>;
> power-domains = <&scmi_power 0>;
> resets = <&scmi_reset 0>;
> };
>
> Where:
> scmi_devid - id from the firmware, which is assigned for AVB.
>
> During initialization, XEN scans probes the first SCI-mediator driver which has
> matching node in the device-tree. If no device-tree was provided, then the
> first registered mediator driver should be probed.
>
> DomX should be configured:
> Device-tree should include the same nodes, described above.
> &cpu_scp_shm should be altered during domain creation. Xen allocates free page
> from the memory region, provided in &cpu_scp_shm in XEN device-tree, so each
> domain should have unique page. Nodes &cpu_scp_shm and /firmware/scmi should be
> copied from partial device-tree to domain device-tree, so kernel can initialize
> scmi driver.
>
> SCI mediator can be enabled in dom.cfg the following way:
>> sci = "scmi_smc"
> which sets scmi_smc to be used for the domain.
Great work! I can imagine this is going to be nice feature once upstreamed.
I am wondering, would the Xen (with the required updates of course) also
be able to send it's own requests to the SCP? For example, to control
overall system performance (CPU frequency)
or other let's say important power management task.
>
> Oleksii Moisieiev (5):
> xen/arm: add support for Renesas R-Car Gen3 platform
> xen/arm: add generic SCI mediator framework
> xen/arm: introduce SCMI-SMC mediator driver
> tools/arm: add "scmi_smc" option to xl.cfg
> xen/arm: add SCI mediator support for DomUs
>
> MAINTAINERS | 6 +
> docs/man/xl.cfg.5.pod.in | 22 +
> tools/include/libxl.h | 5 +
> tools/include/xenctrl.h | 3 +
> tools/include/xenguest.h | 2 +
> tools/libs/ctrl/xc_domain.c | 23 +
> tools/libs/guest/xg_dom_arm.c | 5 +-
> tools/libs/light/libxl_arm.c | 122 ++++-
> tools/libs/light/libxl_create.c | 54 +-
> tools/libs/light/libxl_dom.c | 1 +
> tools/libs/light/libxl_internal.h | 4 +
> tools/libs/light/libxl_types.idl | 6 +
> tools/xl/xl_parse.c | 9 +
> xen/arch/arm/Kconfig | 10 +
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 24 +
> xen/arch/arm/domain_build.c | 11 +
> xen/arch/arm/domctl.c | 15 +
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/rcar3.c | 47 ++
> xen/arch/arm/sci/Kconfig | 10 +
> xen/arch/arm/sci/Makefile | 2 +
> xen/arch/arm/sci/sci.c | 128 +++++
> xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++
> xen/arch/arm/setup.c | 1 +
> xen/arch/arm/xen.lds.S | 7 +
> xen/include/asm-arm/domain.h | 4 +
> xen/include/asm-arm/sci/sci.h | 162 ++++++
> xen/include/public/arch-arm.h | 11 +
> xen/include/public/domctl.h | 9 +
> 30 files changed, 1485 insertions(+), 15 deletions(-)
> create mode 100644 xen/arch/arm/platforms/rcar3.c
> create mode 100644 xen/arch/arm/sci/Kconfig
> create mode 100644 xen/arch/arm/sci/Makefile
> create mode 100644 xen/arch/arm/sci/sci.c
> create mode 100644 xen/arch/arm/sci/scmi_smc.c
> create mode 100644 xen/include/asm-arm/sci/sci.h
>
--
Regards,
Oleksandr Tyshchenko
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-14 9:41 ` Jan Beulich
@ 2021-12-16 17:36 ` Oleksii Moisieiev
2021-12-17 7:12 ` Jan Beulich
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-16 17:36 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis, xen-devel
Hi Jan,
On Tue, Dec 14, 2021 at 10:41:30AM +0100, Jan Beulich wrote:
> On 14.12.2021 10:34, Oleksii Moisieiev wrote:
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> > #define XEN_DOMCTL_vmtrace_get_option 5
> > #define XEN_DOMCTL_vmtrace_set_option 6
> > };
> > +
> > +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> > +struct xen_domctl_sci_device_op {
> > + uint32_t size; /* Length of the path */
> > + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> > +};
>
> This being - aiui - Arm-only, please enclose it by respective #if,
> just like we do for certain x86-only ops.
>
I agree. I will add #ifdefs in v2.
> I'm further afraid the term "SCI" is ambiguous with ACPI's System
> Control Interrupt, so there's some further tag needed in the names
> used here.
>
Thank you for remark. I'm thinking about SC as System Control.
What do you think?
> Finally please make padding explicit and check that it's zero on
> input.
>
I will align the comments in functions and structures in v2.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 2/5] xen/arm: add generic SCI mediator framework
2021-12-14 9:34 ` [RFC v1 2/5] xen/arm: add generic SCI mediator framework Oleksii Moisieiev
@ 2021-12-17 2:45 ` Stefano Stabellini
2021-12-17 12:50 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-17 2:45 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Andrew Cooper, George Dunlap, Jan Beulich,
Julien Grall, Stefano Stabellini, Wei Liu, Volodymyr Babchuk,
Bertrand Marquis
On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> This patch adds the basic framework for SCI mediator.
> SCI is System Control Interface, which is designed to redirect
> requests for the Hardware (such as power-domain/clock/resets etc)
> from the Domains to the firmware. Originally, cpg should be passed
> to the domain so it can work with power-domains/clocks/resets etc.
> Considering that cpg can't be split between the Domains, we get the
> limitation that the devices, which are using power-domains/clocks/resets
> etc, couldn't be split between the domains.
> The solution is to move the power-domain/clock/resets etc to the
> Firmware (such as SCP firmware or ATF) and provide interface for the Domains.
> XEN shoud have an entity, caled SCI-Mediator, which is responsible for messages
> redirection between Domains and Firmware and for permission handling.
>
> This is how it works: user can build XEN with multiple SCI mediators.
> See the next patches, where SCMI-SMC mediator is introduced.
> SCI mediator register itself with REGISTER_SCI_MEDIATOR() macro.
>
> At run-time, during initialization, framework calls probe for the first
> matching device in the device-tree. When no device-tree is present - the
> first registered mediator should be probed.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> MAINTAINERS | 6 ++
> xen/arch/arm/Kconfig | 8 ++
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 24 +++++
> xen/arch/arm/domain_build.c | 11 +++
> xen/arch/arm/platforms/rcar3.c | 3 +-
> xen/arch/arm/sci/Makefile | 1 +
> xen/arch/arm/sci/sci.c | 128 ++++++++++++++++++++++++++
> xen/arch/arm/setup.c | 1 +
> xen/arch/arm/xen.lds.S | 7 ++
> xen/include/asm-arm/domain.h | 4 +
> xen/include/asm-arm/sci/sci.h | 162 +++++++++++++++++++++++++++++++++
> xen/include/public/arch-arm.h | 10 ++
> 13 files changed, 365 insertions(+), 1 deletion(-)
> create mode 100644 xen/arch/arm/sci/Makefile
> create mode 100644 xen/arch/arm/sci/sci.c
> create mode 100644 xen/include/asm-arm/sci/sci.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index e43dc0edce..5f96ea35ba 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -512,6 +512,12 @@ S: Supported
> F: xen/arch/arm/tee/
> F: xen/include/asm-arm/tee
>
> +SCI MEDIATORS
> +M: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> +S: Supported
> +F: xen/arch/arm/sci
> +F: xen/include/asm-arm/sci
> +
> TOOLSTACK
> M: Wei Liu <wl@xen.org>
> S: Supported
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index ecfa6822e4..186e1db389 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -106,6 +106,14 @@ config TEE
>
> source "arch/arm/tee/Kconfig"
>
> +config SCI
> + bool "Enable SCI mediators support"
> + default n
> + help
> + This option enables generic SCI (System Control Interface) mediators
> + support. It allows guests to control system resourcess via one of
> + SCI mediators implemented in XEN.
> +
> endmenu
>
> menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 07f634508e..6366ff55e5 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -8,6 +8,7 @@ obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> obj-$(CONFIG_HAS_VPCI) += vpci.o
> +obj-$(CONFIG_SCI) += sci/
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 96e1b23550..80d0a23767 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -34,6 +34,7 @@
> #include <asm/platform.h>
> #include <asm/procinfo.h>
> #include <asm/regs.h>
> +#include <asm/sci/sci.h>
> #include <asm/tee/tee.h>
> #include <asm/vfp.h>
> #include <asm/vgic.h>
> @@ -688,6 +689,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> return -EINVAL;
> }
>
> + if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE &&
> + config->arch.sci_type != sci_get_type() )
> + {
> + dprintk(XENLOG_INFO, "Unsupported SCI type\n");
> + return -EINVAL;
> + }
> +
> return 0;
> }
>
> @@ -764,6 +772,15 @@ int arch_domain_create(struct domain *d,
> /* At this stage vgic_reserve_virq should never fail */
> if ( !vgic_reserve_virq(d, GUEST_EVTCHN_PPI) )
> BUG();
> +
> + if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE )
> + {
> + if ( (rc = sci_domain_init(d, config->arch.sci_type)) != 0)
> + goto fail;
> +
> + if ( (rc = sci_get_channel_info(d, &config->arch)) != 0)
> + goto fail;
> + }
> }
>
> /*
> @@ -796,6 +813,7 @@ void arch_domain_destroy(struct domain *d)
> domain_vgic_free(d);
> domain_vuart_free(d);
> free_xenheap_page(d->shared_info);
> + sci_domain_destroy(d);
> #ifdef CONFIG_ACPI
> free_xenheap_pages(d->arch.efi_acpi_table,
> get_order_from_bytes(d->arch.efi_acpi_len));
> @@ -996,6 +1014,7 @@ enum {
> PROG_xen,
> PROG_page,
> PROG_mapping,
> + PROG_sci,
> PROG_done,
> };
>
> @@ -1056,6 +1075,11 @@ int domain_relinquish_resources(struct domain *d)
> if ( ret )
> return ret;
>
> + PROGRESS(sci):
> + ret = sci_relinquish_resources(d);
> + if ( ret )
> + return ret;
> +
> PROGRESS(done):
> break;
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index d02bacbcd1..38874615dd 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -22,6 +22,7 @@
> #include <asm/kernel.h>
> #include <asm/setup.h>
> #include <asm/tee/tee.h>
> +#include <asm/sci/sci.h>
> #include <asm/platform.h>
> #include <asm/psci.h>
> #include <asm/setup.h>
> @@ -1894,6 +1895,10 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
> return res;
> }
> }
> +
> + res = sci_add_dt_device(d, dev);
> + if ( res < 0 )
> + return res;
> }
It would be good to add the call to dom0less domUs too
> res = handle_device_interrupts(d, dev, need_mapping);
> @@ -3075,6 +3080,10 @@ static int __init construct_dom0(struct domain *d)
> if ( rc < 0 )
> return rc;
>
> + rc = sci_domain_init(d, sci_get_type());
> + if ( rc < 0 )
> + return rc;
same here
> if ( acpi_disabled )
> rc = prepare_dtb_hwdom(d, &kinfo);
> else
> @@ -3109,6 +3118,8 @@ void __init create_dom0(void)
> dom0_cfg.arch.tee_type = tee_get_type();
> dom0_cfg.max_vcpus = dom0_max_vcpus();
>
> + dom0_cfg.arch.sci_type = sci_get_type();
and here
> if ( iommu_enabled )
> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>
> diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
> index d740145c71..a268b09454 100644
> --- a/xen/arch/arm/platforms/rcar3.c
> +++ b/xen/arch/arm/platforms/rcar3.c
> @@ -18,10 +18,11 @@
> */
>
> #include <asm/platform.h>
> +#include <asm/sci/sci.h>
>
> static bool rcar3_smc(struct cpu_user_regs *regs)
> {
> - return false;
> + return sci_handle_call(current->domain, regs);
> }
>
> static const char *const rcar3_dt_compat[] __initconst =
> diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> new file mode 100644
> index 0000000000..837dc7492b
> --- /dev/null
> +++ b/xen/arch/arm/sci/Makefile
> @@ -0,0 +1 @@
> +obj-y += sci.o
> diff --git a/xen/arch/arm/sci/sci.c b/xen/arch/arm/sci/sci.c
> new file mode 100644
> index 0000000000..5961b4cd5d
> --- /dev/null
> +++ b/xen/arch/arm/sci/sci.c
> @@ -0,0 +1,128 @@
> +/*
> + * xen/arch/arm/sci/sci.c
> + *
> + * Generic part of SCI mediator driver
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021, EPAM Systems.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/errno.h>
> +#include <xen/init.h>
> +#include <xen/sched.h>
> +#include <xen/types.h>
> +
> +#include <asm/sci/sci.h>
> +
> +extern const struct sci_mediator_desc _sscimediator[], _escimediator[];
> +static const struct sci_mediator_desc __read_mostly *cur_mediator;
> +
> +bool sci_handle_call(struct domain *d, void *args)
> +{
> + if ( unlikely(!cur_mediator) )
> + return false;
> +
> + return cur_mediator->ops->handle_call(d, args);
> +}
> +
> +int sci_domain_init(struct domain *d, uint16_t sci_type)
> +{
> + if ( sci_type == XEN_DOMCTL_CONFIG_SCI_NONE )
> + return 0;
> +
> + if ( unlikely(!cur_mediator) )
> + return -ENODEV;
> +
> + if ( cur_mediator->sci_type != sci_type )
> + return -EINVAL;
> +
> + return cur_mediator->ops->domain_init(d);
> +}
> +
> +void sci_domain_destroy(struct domain *d)
> +{
> + if ( unlikely(!cur_mediator) )
> + return;
> +
> + cur_mediator->ops->domain_destroy(d);
> +}
> +
> +int sci_relinquish_resources(struct domain *d)
> +{
> + if ( unlikely(!cur_mediator) )
> + return 0;
> +
> + return cur_mediator->ops->relinquish_resources(d);
> +}
> +
> +
> +int sci_add_dt_device(struct domain *d, struct dt_device_node *dev)
> +{
> + if ( unlikely(!cur_mediator) )
> + return 0;
> +
> + return cur_mediator->ops->add_dt_device(d, dev);
> +}
> +
> +int sci_get_channel_info(struct domain *d,
> + struct xen_arch_domainconfig *config)
> +{
> + if ( unlikely(!cur_mediator) )
> + return 0;
> +
> + return cur_mediator->ops->get_channel_info(d->arch.sci, config);
> +}
> +
> +uint16_t sci_get_type(void)
> +{
> + if ( unlikely(!cur_mediator) )
> + return XEN_DOMCTL_CONFIG_SCI_NONE;
> +
> + return cur_mediator->sci_type;
> +}
> +
> +static int __init sci_init(void)
> +{
> + const struct sci_mediator_desc *desc;
> + struct dt_device_node *dt = NULL;
> +
> + for ( desc = _sscimediator; desc != _escimediator; desc++ )
> + {
> + if ( likely(dt_host) )
I think you can assume that dt_host is correctly set, right?
Instead of this check, I would just add:
if ( !acpi_disabled )
return -ENODEV;
at the beginning of sci_init
> + {
> + dt = dt_find_matching_node(dt_host, desc->dt_match);
> + if ( !dt )
> + continue;
> + }
> +
> + if ( desc->ops->probe(dt) )
> + {
> + printk(XENLOG_INFO "Using SCI mediator for %s\n", desc->name);
> + cur_mediator = desc;
> + return 0;
> + }
> + }
> +
> + return 0;
> +}
> +
> +__initcall(sci_init);
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index d5d0792ed4..201de01411 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -50,6 +50,7 @@
> #include <asm/cpufeature.h>
> #include <asm/platform.h>
> #include <asm/procinfo.h>
> +#include <asm/sci/sci.h>
> #include <asm/setup.h>
> #include <xsm/xsm.h>
> #include <asm/acpi.h>
> diff --git a/xen/arch/arm/xen.lds.S b/xen/arch/arm/xen.lds.S
> index 08016948ab..3683f4821f 100644
> --- a/xen/arch/arm/xen.lds.S
> +++ b/xen/arch/arm/xen.lds.S
> @@ -142,6 +142,13 @@ SECTIONS
> _eteemediator = .;
> } :text
>
> + . = ALIGN(8);
> + .scimediator.info : {
> + _sscimediator = .;
> + *(.scimediator.info)
> + _escimediator = .;
> + } :text
> +
> . = ALIGN(PAGE_SIZE); /* Init code and data */
> __init_begin = .;
> .init.text : {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 9b3647587a..d9b164017f 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -89,6 +89,10 @@ struct arch_domain
> #ifdef CONFIG_TEE
> void *tee;
> #endif
> +
> +#ifdef CONFIG_SCI
> + void *sci;
> +#endif
> } __cacheline_aligned;
>
> struct arch_vcpu
> diff --git a/xen/include/asm-arm/sci/sci.h b/xen/include/asm-arm/sci/sci.h
> new file mode 100644
> index 0000000000..aeff689c72
> --- /dev/null
> +++ b/xen/include/asm-arm/sci/sci.h
> @@ -0,0 +1,162 @@
> +/*
> + * xen/include/asm-arm/sci/sci.h
> + *
> + * Generic part of the SCI (System Control Interface) subsystem.
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021, EPAM Systems.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ASM_ARM_SCI_H
> +#define __ASM_ARM_SCI_H
> +
> +#include <xen/lib.h>
> +#include <xen/types.h>
> +#include <xen/device_tree.h>
> +
> +#ifdef CONFIG_SCI
> +
> +struct sci_mediator_ops {
> +
> + /*
> + * Probe for SCI. Should return true if SCI found and
> + * mediator is initialized.
> + */
> + bool (*probe)(struct dt_device_node *scmi_node);
> +
> + /*
> + * Called during domain construction if toolstack requests to enable
> + * SCI support so mediator can inform SCP-firmware about new
> + * guest and create own structures for the new domain.
> + */
> + int (*domain_init)(struct domain *d);
> +
> + /*
> + * Called during domain destruction, releases all resources, that
> + * were allocated by the mediator.
> + */
> + void (*domain_destroy)(struct domain *d);
> +
> + /*
> + * Called during parsing partial device-sci for the domain.
> + * Passing device_node so mediator could process the device and
> + * mark the device as related to the domain if needed.
> + */
> + int (*add_dt_device)(struct domain *d, struct dt_device_node *dev);
> +
> + /*
> + * Called during domain destruction to relinquish resources used
> + * by mediator itself. This function can return -ERESTART to indicate
> + * that it does not finished work and should be called again.
> + */
> + int (*relinquish_resources)(struct domain *d);
> +
> + /* Handle call for current domain */
> + bool (*handle_call)(struct domain *d, void *regs);
> +
> + /* Gets channel configuration and store it in domainconfig */
> + int (*get_channel_info)(void *sci_ops,
> + struct xen_arch_domainconfig *config);
get_channel_info is the only function I don't understand among these...
what is supposed to be stored in struct xen_arch_domainconfig, just
sci_agent_paddr?
Also, it seems to be only called right after sci_domain_init, so can't
the "get_channel_info" operation just be done as part of domain_init?
> +};
> +
> +struct sci_mediator_desc {
> + /* Printable name of the SCI. */
> + const char *name;
> +
> + /* Mediator callbacks as described above. */
> + const struct sci_mediator_ops *ops;
> +
> + /*
> + * ID of SCI. Corresponds to xen_arch_domainconfig.sci_type.
> + * Should be one of XEN_DOMCTL_CONFIG_SCI_xxx
> + */
> + uint16_t sci_type;
> +
> + /* Match structure to init mediator */
> + const struct dt_device_match *dt_match;
> +
> +};
> +
> +int sci_domain_init(struct domain *d, uint16_t sci_type);
> +void sci_domain_destroy(struct domain *d);
> +int sci_add_dt_device(struct domain *d, struct dt_device_node *dev);
> +int sci_relinquish_resources(struct domain *d);
> +bool sci_handle_call(struct domain *d, void *args);
> +int sci_get_channel_info(struct domain *d,
> + struct xen_arch_domainconfig *config);
> +uint16_t sci_get_type(void);
> +
> +#define REGISTER_SCI_MEDIATOR(_name, _namestr, _type, _match, _ops) \
> +static const struct sci_mediator_desc __sci_desc_##_name __used \
> +__section(".scimediator.info") = { \
> + .name = _namestr, \
> + .ops = _ops, \
> + .sci_type = _type, \
> + .dt_match = _match \
> +}
> +
> +#else
> +#include <public/errno.h>
> +
> +static inline int sci_domain_init(struct domain *d, uint16_t sci_type)
> +{
> + if ( likely(sci_type == XEN_DOMCTL_CONFIG_SCI_NONE) )
> + return 0;
> +
> + return -XEN_ENODEV;
> +}
> +
> +static inline void sci_domain_destroy(struct domain *d)
> +{
> +}
> +
> +static inline int sci_add_dt_device(struct domain *d,
> + struct dt_device_node *dev)
> +{
> + return 0;
> +}
> +
> +static inline int sci_relinquish_resources(struct domain *d)
> +{
> + return 0;
> +}
> +
> +static inline bool sci_handle_call(struct domain *d, void *args)
> +{
> + return false;
> +}
> +
> +static inline int sci_get_channel_info(struct domain *d,
> + struct xen_arch_domainconfig *config)
> +{
> + return 0;
> +}
> +
> +static inline uint16_t sci_get_type(void)
> +{
> + return XEN_DOMCTL_CONFIG_SCI_NONE;
> +}
> +
> +#endif /* CONFIG_SCI */
> +
> +#endif /* __ASM_ARM_SCI_H */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 94b31511dd..9180be5e86 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -314,12 +314,16 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_TEE_NONE 0
> #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
>
> +#define XEN_DOMCTL_CONFIG_SCI_NONE 0
> +
> struct xen_arch_domainconfig {
> /* IN/OUT */
> uint8_t gic_version;
> /* IN */
> uint16_t tee_type;
> /* IN */
> + uint8_t sci_type;
sci_type is uint16_t everywhere else
> + /* IN */
> uint32_t nr_spis;
> /*
> * OUT
> @@ -335,6 +339,12 @@ struct xen_arch_domainconfig {
> *
> */
> uint32_t clock_frequency;
> +
> + /* Sets shared address to sw domains.
> + * This information is needed to set correct channel in Domain partial
> + * device-tree
> + */
> + uint64_t sci_agent_paddr;
> };
> #endif /* __XEN__ || __XEN_TOOLS__ */
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-16 17:36 ` Oleksii Moisieiev
@ 2021-12-17 7:12 ` Jan Beulich
2021-12-17 7:16 ` Jan Beulich
0 siblings, 1 reply; 95+ messages in thread
From: Jan Beulich @ 2021-12-17 7:12 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis, xen-devel
On 16.12.2021 18:36, Oleksii Moisieiev wrote:
> On Tue, Dec 14, 2021 at 10:41:30AM +0100, Jan Beulich wrote:
>> On 14.12.2021 10:34, Oleksii Moisieiev wrote:
>>> --- a/xen/include/public/domctl.h
>>> +++ b/xen/include/public/domctl.h
>>> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
>>> #define XEN_DOMCTL_vmtrace_get_option 5
>>> #define XEN_DOMCTL_vmtrace_set_option 6
>>> };
>>> +
>>> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
>>> +struct xen_domctl_sci_device_op {
>>> + uint32_t size; /* Length of the path */
>>> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
>>> +};
>>
>> This being - aiui - Arm-only, please enclose it by respective #if,
>> just like we do for certain x86-only ops.
>>
>
> I agree. I will add #ifdefs in v2.
>
>> I'm further afraid the term "SCI" is ambiguous with ACPI's System
>> Control Interrupt, so there's some further tag needed in the names
>> used here.
>>
>
> Thank you for remark. I'm thinking about SC as System Control.
> What do you think?
I guess "SC" could even more so stand for various things. Even the
spelled out "System Control" looks overly generic. If this isn't
Arm-specific (in which case adding "arm" into the name might at least
help the situation a little), then I guess some further disambiguation
is going to be wanted. Since I don't know any of the context of this,
I'm afraid you're in a far better position than me to come up with a
non-ambiguous name.
Jan
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-17 7:12 ` Jan Beulich
@ 2021-12-17 7:16 ` Jan Beulich
2021-12-17 13:40 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Jan Beulich @ 2021-12-17 7:16 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis, xen-devel
On 17.12.2021 08:12, Jan Beulich wrote:
> On 16.12.2021 18:36, Oleksii Moisieiev wrote:
>> On Tue, Dec 14, 2021 at 10:41:30AM +0100, Jan Beulich wrote:
>>> On 14.12.2021 10:34, Oleksii Moisieiev wrote:
>>>> --- a/xen/include/public/domctl.h
>>>> +++ b/xen/include/public/domctl.h
>>>> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
>>>> #define XEN_DOMCTL_vmtrace_get_option 5
>>>> #define XEN_DOMCTL_vmtrace_set_option 6
>>>> };
>>>> +
>>>> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
>>>> +struct xen_domctl_sci_device_op {
>>>> + uint32_t size; /* Length of the path */
>>>> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
>>>> +};
>>>
>>> This being - aiui - Arm-only, please enclose it by respective #if,
>>> just like we do for certain x86-only ops.
>>>
>>
>> I agree. I will add #ifdefs in v2.
>>
>>> I'm further afraid the term "SCI" is ambiguous with ACPI's System
>>> Control Interrupt, so there's some further tag needed in the names
>>> used here.
>>>
>>
>> Thank you for remark. I'm thinking about SC as System Control.
>> What do you think?
>
> I guess "SC" could even more so stand for various things. Even the
> spelled out "System Control" looks overly generic. If this isn't
> Arm-specific (in which case adding "arm" into the name might at least
> help the situation a little), then I guess some further disambiguation
> is going to be wanted. Since I don't know any of the context of this,
> I'm afraid you're in a far better position than me to come up with a
> non-ambiguous name.
Actually, looking at the title again - any reason not to add "mediator"
into the name? While I have no idea whether there could be other
mediators with an ambiguous acronym, this would at least address the
ACPI related concern (I don't expect anything mediator-like to appear
there, but then again I might be wrong).
Jan
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-15 9:39 ` Julien Grall
@ 2021-12-17 10:48 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 10:48 UTC (permalink / raw)
To: Julien Grall
Cc: xen-devel, Stefano Stabellini, Volodymyr Babchuk, Bertrand Marquis
Hi Julien,
On Wed, Dec 15, 2021 at 09:39:32AM +0000, Julien Grall wrote:
> Hi,
>
> Looking at the rest of the series, this is going to be replaced in patch #2
> with:
>
> return sci_handle_call();
>
> SCMI is not specific to RCAR3. So I would expect the function to be called
> from common code.
>
> If it still needs some platform specific code, then I think it would be best
> to introduce rcar3.c at the end of the series. So we don't introduce a dummy
> platform and not hook the code in the middle of patch#2 which is meant to be
> generic.
>
> I will have a proper review of the rest of the series in the new year.
>
> Cheers,
>
> --
> Julien Grall
That's sound reasonable. My first thought was to move SCM func_id to the
different section, such as Arm Architecture Service (see Section 6
of DEN0028D). But I think that SiP service func_id fits best in this
case because from guest standpoint all clocks\resets\power-domain ids
are SoC implementation specific.
I'm going to leave SMC func_id in SiP range, but refactor SIP smc
handler, so no RCAR3 specific code will be needed. So there will be no
need to introduce rcar3.c in this patch series.
What do you think about that?
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform
2021-12-15 9:57 ` Oleksandr Tyshchenko
@ 2021-12-17 10:52 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 10:52 UTC (permalink / raw)
To: Oleksandr Tyshchenko
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
Hi Oleksandr,
On Wed, Dec 15, 2021 at 11:57:29AM +0200, Oleksandr Tyshchenko wrote:
> On Tue, Dec 14, 2021 at 11:35 AM Oleksii Moisieiev <
> Oleksii_Moisieiev@epam.com> wrote:
>
> Hi Oleksii
>
> [sorry for the possible format issues]
>
> Implementation includes platform-specific smc handler for rcar3 platform.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > xen/arch/arm/platforms/Makefile | 1 +
> > xen/arch/arm/platforms/rcar3.c | 46 +++++++++++++++++++++++++++++++++
> > 2 files changed, 47 insertions(+)
> > create mode 100644 xen/arch/arm/platforms/rcar3.c
> >
> > diff --git a/xen/arch/arm/platforms/Makefile
> > b/xen/arch/arm/platforms/Makefile
> > index 8632f4115f..b64c25de6c 100644
> > --- a/xen/arch/arm/platforms/Makefile
> > +++ b/xen/arch/arm/platforms/Makefile
> > @@ -4,6 +4,7 @@ obj-$(CONFIG_ALL32_PLAT) += exynos5.o
> > obj-$(CONFIG_ALL32_PLAT) += midway.o
> > obj-$(CONFIG_ALL32_PLAT) += omap5.o
> > obj-$(CONFIG_ALL32_PLAT) += rcar2.o
> > +obj-$(CONFIG_RCAR3) += rcar3.o
> > obj-$(CONFIG_ALL64_PLAT) += seattle.o
> > obj-$(CONFIG_ALL_PLAT) += sunxi.o
> > obj-$(CONFIG_ALL64_PLAT) += thunderx.o
> > diff --git a/xen/arch/arm/platforms/rcar3.c
> > b/xen/arch/arm/platforms/rcar3.c
> > new file mode 100644
> > index 0000000000..d740145c71
> > --- /dev/null
> > +++ b/xen/arch/arm/platforms/rcar3.c
> > @@ -0,0 +1,46 @@
> > +/*
> > + * xen/arch/arm/platforms/rcar3.c
> > + *
> > + * Renesas R-Car Gen3 specific settings
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021 EPAM Systems
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <asm/platform.h>
> > +
> > +static bool rcar3_smc(struct cpu_user_regs *regs)
> > +{
> > + return false;
> > +}
> > +
> > +static const char *const rcar3_dt_compat[] __initconst =
> > +{
> > + "renesas,r8a7795",
> > + "renesas,r8a7796",
> >
>
>
> Please note that since Linux commit:
> "9c9f7891093b02eb64ca4e1c7ab776a4296c058f soc: renesas: Identify R-Car
> M3-W+"
> the compatible string for R-Car M3-W+ (ES3.0) SoC is "renesas,r8a77961". So
> in case we want to have vSCMI feature on this new SoC revision as well we
> will need
> to extend the compatible list.
>
Thank you for the point, this will be fixed in v2.
Best regards,
Oleksii
>
> + NULL
> > +};
> > +
> > +PLATFORM_START(rcar3, "Renesas R-Car Gen3")
> > + .compatible = rcar3_dt_compat,
> > + .smc = rcar3_smc
> > +PLATFORM_END
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > --
> > 2.27.0
> >
> >
>
> --
> Regards,
>
> Oleksandr Tyshchenko
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-15 21:51 ` Oleksandr
@ 2021-12-17 11:00 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 11:00 UTC (permalink / raw)
To: Oleksandr; +Cc: xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
Hi Oleksandr,
On Wed, Dec 15, 2021 at 11:51:01PM +0200, Oleksandr wrote:
>
> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>
>
> Hi Oleksii
>
> > This enumeration sets SCI type for the domain. Currently there is
> > two possible options: either 'none' or 'scmi_smc'.
> >
> > 'none' is the default value and it disables SCI support at all.
> >
> > 'scmi_smc' enables access to the Firmware from the domains using SCMI
> > protocol and SMC as transport.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> > tools/include/libxl.h | 5 +++++
> > tools/libs/light/libxl_types.idl | 6 ++++++
> > tools/xl/xl_parse.c | 9 +++++++++
> > 4 files changed, 42 insertions(+)
> >
> > diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> > index b98d161398..92d0593875 100644
> > --- a/docs/man/xl.cfg.5.pod.in
> > +++ b/docs/man/xl.cfg.5.pod.in
> > @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
> > =back
> > +=item B<sci="STRING">
> > +
> > +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> > +allows domain to manage various functions that are provided by HW platform.
> > +
> > +=over 4
> > +
> > +=item B<none>
> > +
> > +Don't allow guest to use SCI if present on the platform. This is the default
> > +value.
> > +
> > +=item B<scmi_smc>
> > +
> > +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> > +Inferface - allows domain to manage various functions that are provided by HW
> > +platform, such as clocks, resets and power-domains. Xen will mediate access to
> > +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> > +SMC is used as transport.
> > +
> > +=back
> > +
> > =back
> > =head2 Paravirtualised (PV) Guest Specific Options
> > diff --git a/tools/include/libxl.h b/tools/include/libxl.h
> > index 2bbbd21f0b..30e5aee119 100644
> > --- a/tools/include/libxl.h
> > +++ b/tools/include/libxl.h
> > @@ -278,6 +278,11 @@
> > */
> > #define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE 1
> > +/*
> > + * libxl_domain_build_info has the arch_arm.sci field.
> > + */
> > +#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_SCI 1
> > +
> > /*
> > * LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing
> > * 'soft reset' for domains and there is 'soft_reset' shutdown reason
> > diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> > index 2a42da2f7d..9067b509f4 100644
> > --- a/tools/libs/light/libxl_types.idl
> > +++ b/tools/libs/light/libxl_types.idl
>
> I assume that at least goland bindings want updating.
>
Thanks for the notice. I will fix that in v2.
>
> > @@ -480,6 +480,11 @@ libxl_tee_type = Enumeration("tee_type", [
> > (1, "optee")
> > ], init_val = "LIBXL_TEE_TYPE_NONE")
> > +libxl_sci_type = Enumeration("sci_type", [
> > + (0, "none"),
> > + (1, "scmi_smc")
> > + ], init_val = "LIBXL_SCI_TYPE_NONE")
> > +
> > libxl_rdm_reserve = Struct("rdm_reserve", [
> > ("strategy", libxl_rdm_reserve_strategy),
> > ("policy", libxl_rdm_reserve_policy),
> > @@ -564,6 +569,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
> > ("apic", libxl_defbool),
> > ("dm_restrict", libxl_defbool),
> > ("tee", libxl_tee_type),
> > + ("sci", libxl_sci_type),
> > ("u", KeyedUnion(None, libxl_domain_type, "type",
> > [("hvm", Struct(None, [("firmware", string),
> > ("bios", libxl_bios_type),
> > diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
> > index 117fcdcb2b..c37bf6298b 100644
> > --- a/tools/xl/xl_parse.c
> > +++ b/tools/xl/xl_parse.c
> > @@ -2747,6 +2747,15 @@ skip_usbdev:
> > }
> > }
> > + if (!xlu_cfg_get_string (config, "sci", &buf, 1)) {
> > + e = libxl_sci_type_from_string(buf, &b_info->sci);
> > + if (e) {
> > + fprintf(stderr,
> > + "Unknown sci \"%s\" specified\n", buf);
> > + exit(-ERROR_FAIL);
> > + }
> > + }
> > +
> > parse_vkb_list(config, d_config);
> > xlu_cfg_get_defbool(config, "xend_suspend_evtchn_compat",
>
> --
> Regards,
>
> Oleksandr Tyshchenko
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-14 9:34 ` [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver Oleksii Moisieiev
@ 2021-12-17 11:35 ` Oleksandr
2021-12-17 13:23 ` Oleksii Moisieiev
2021-12-18 2:14 ` Stefano Stabellini
1 sibling, 1 reply; 95+ messages in thread
From: Oleksandr @ 2021-12-17 11:35 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On 14.12.21 11:34, Oleksii Moisieiev wrote:
Hi Oleksii
> This is the implementation of SCI interface, called SCMI-SMC driver,
> which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> This allows devices from the Domains to work with clocks, resets and
> power-domains without access to CPG.
>
> The following features are implemented:
> - request SCMI channels from ATF and pass channels to Domains;
> - set device permissions for Domains based on the Domain partial
> device-tree. Devices with permissions are able to work with clocks,
> resets and power-domains via SCMI;
> - redirect scmi messages from Domains to ATF.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> xen/arch/arm/Kconfig | 2 +
> xen/arch/arm/sci/Kconfig | 10 +
> xen/arch/arm/sci/Makefile | 1 +
> xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> xen/include/public/arch-arm.h | 1 +
> 5 files changed, 809 insertions(+)
> create mode 100644 xen/arch/arm/sci/Kconfig
> create mode 100644 xen/arch/arm/sci/scmi_smc.c
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 186e1db389..02d96c6cfc 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -114,6 +114,8 @@ config SCI
> support. It allows guests to control system resourcess via one of
> SCI mediators implemented in XEN.
>
> +source "arch/arm/sci/Kconfig"
> +
> endmenu
>
> menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> new file mode 100644
> index 0000000000..9563067ddc
> --- /dev/null
> +++ b/xen/arch/arm/sci/Kconfig
> @@ -0,0 +1,10 @@
> +config SCMI_SMC
> + bool "Enable SCMI-SMC mediator driver"
> + default n
> + depends on SCI
> + ---help---
> +
> + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> + This feature allows drivers from Domains to work with System
> + Controllers (such as power,resets,clock etc.). SCP is used as transport
> + for communication.
> diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> index 837dc7492b..67f2611872 100644
> --- a/xen/arch/arm/sci/Makefile
> +++ b/xen/arch/arm/sci/Makefile
> @@ -1 +1,2 @@
> obj-y += sci.o
> +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> new file mode 100644
> index 0000000000..2eb01ea82d
> --- /dev/null
> +++ b/xen/arch/arm/sci/scmi_smc.c
> @@ -0,0 +1,795 @@
> +/*
> + * xen/arch/arm/sci/scmi_smc.c
> + *
> + * SCMI mediator driver, using SCP as transport.
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021, EPAM Systems.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <asm/sci/sci.h>
> +#include <asm/smccc.h>
> +#include <asm/io.h>
> +#include <xen/bitops.h>
> +#include <xen/config.h>
> +#include <xen/sched.h>
> +#include <xen/device_tree.h>
> +#include <xen/iocap.h>
> +#include <xen/init.h>
> +#include <xen/err.h>
> +#include <xen/lib.h>
> +#include <xen/list.h>
> +#include <xen/mm.h>
> +#include <xen/string.h>
> +#include <xen/time.h>
> +#include <xen/vmap.h>
> +
> +#define SCMI_BASE_PROTOCOL 0x10
> +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> +#define SCMI_BASE_DISCOVER_AGENT 0x7
> +
> +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> +#define SCMI_SUCCESS 0
> +#define SCMI_NOT_SUPPORTED (-1)
> +#define SCMI_INVALID_PARAMETERS (-2)
> +#define SCMI_DENIED (-3)
> +#define SCMI_NOT_FOUND (-4)
> +#define SCMI_OUT_OF_RANGE (-5)
> +#define SCMI_BUSY (-6)
> +#define SCMI_COMMS_ERROR (-7)
> +#define SCMI_GENERIC_ERROR (-8)
> +#define SCMI_HARDWARE_ERROR (-9)
> +#define SCMI_PROTOCOL_ERROR (-10)
> +
> +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> +
> +#define SCMI_SMC_ID "arm,smc-id"
> +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> +#define SCMI_SHMEM "shmem"
> +
> +#define HYP_CHANNEL 0x0
> +
> +#define HDR_ID GENMASK(7,0)
> +#define HDR_TYPE GENMASK(9, 8)
> +#define HDR_PROTO GENMASK(17, 10)
> +
> +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> +
> +#define FIELD_GET(_mask, _reg)\
> + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> +#define FIELD_PREP(_mask, _val)\
> + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> +
> +typedef struct scmi_msg_header {
> + uint8_t id;
> + uint8_t type;
> + uint8_t protocol;
> +} scmi_msg_header_t;
> +
> +typedef struct scmi_perms_tx {
> + uint32_t agent_id;
> + uint32_t device_id;
> + uint32_t flags;
> +} scmi_perms_tx_t;
> +
> +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> +
> +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> +
> +struct scmi_shared_mem {
> + uint32_t reserved;
> + uint32_t channel_status;
> + uint32_t reserved1[2];
> + uint32_t flags;
> + uint32_t length;
> + uint32_t msg_header;
> + uint8_t msg_payload[];
> +};
> +
> +struct scmi_channel {
> + int chan_id;
> + int agent_id;
> + uint32_t func_id;
> + int domain_id;
> + uint64_t paddr;
> + struct scmi_shared_mem *shmem;
> + spinlock_t lock;
> + struct list_head list;
> +};
> +
> +struct scmi_data {
> + struct list_head channel_list;
> + spinlock_t channel_list_lock;
> + bool initialized;
> + u64 shmem_addr, shmem_size;
> +};
> +
> +static struct scmi_data scmi_data;
> +
> +/*
> + * pack_scmi_header() - packs and returns 32-bit header
> + *
> + * @hdr: pointer to header containing all the information on message id,
> + * protocol id and type id.
> + *
> + * Return: 32-bit packed message header to be sent to the platform.
> + */
> +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> +{
> + return FIELD_PREP(HDR_ID, hdr->id) |
> + FIELD_PREP(HDR_TYPE, hdr->type) |
> + FIELD_PREP(HDR_PROTO, hdr->protocol);
> +}
> +
> +/*
> + * unpack_scmi_header() - unpacks and records message and protocol id
> + *
> + * @msg_hdr: 32-bit packed message header sent from the platform
> + * @hdr: pointer to header to fetch message and protocol id.
> + */
> +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> +{
> + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> +}
> +
> +static inline int channel_is_free(struct scmi_channel *chan_info)
> +{
> + return ( chan_info->shmem->channel_status
> + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
> +}
> +
> +static int send_smc_message(struct scmi_channel *chan_info,
> + scmi_msg_header_t *hdr, void *data, int len)
> +{
> + struct arm_smccc_res resp;
> + int ret;
> +
> + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> + chan_info->shmem->channel_status, len);
> + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> + hdr->id, hdr->type, hdr->protocol);
> +
> + ret = channel_is_free(chan_info);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + chan_info->shmem->channel_status = 0x0;
> + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> + chan_info->shmem->flags = 0x0;
> + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> +
> + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> + chan_info->shmem);
> + if ( len > 0 && data )
> + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
> +
> + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> + &resp);
> +
> + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> +
> + if ( resp.a0 )
> + return -EOPNOTSUPP;
> +
> + return 0;
> +}
> +
> +static int check_scmi_status(int scmi_status)
> +{
> + if ( scmi_status == SCMI_SUCCESS )
> + return 0;
> +
> + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> +
> + switch ( scmi_status )
> + {
> + case SCMI_NOT_SUPPORTED:
> + return -EOPNOTSUPP;
> + case SCMI_INVALID_PARAMETERS:
> + return -EINVAL;
> + case SCMI_DENIED:
> + return -EACCES;
> + case SCMI_NOT_FOUND:
> + return -ENOENT;
> + case SCMI_OUT_OF_RANGE:
> + return -ERANGE;
> + case SCMI_BUSY:
> + return -EBUSY;
> + case SCMI_COMMS_ERROR:
> + return -ENOTCONN;
> + case SCMI_GENERIC_ERROR:
> + return -EIO;
> + case SCMI_HARDWARE_ERROR:
> + return -ENXIO;
> + case SCMI_PROTOCOL_ERROR:
> + return -EBADMSG;
Probably we should add a default: here?
> + }
> +
> + return -EINVAL;
> +}
> +
> +static int get_smc_response(struct scmi_channel *chan_info,
> + scmi_msg_header_t *hdr, void *data, int len)
> +{
> + int recv_len;
> + int ret;
> +
> + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> +
> + ret = channel_is_free(chan_info);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> +
> + if ( recv_len < 0 )
> + {
> + printk(XENLOG_ERR
> + "scmi: Wrong size of smc message. Data may be invalid\n");
> + return -EINVAL;
> + }
> +
> + if ( recv_len > len )
> + {
> + printk(XENLOG_ERR
> + "scmi: Not enough buffer for message %d, expecting %d\n",
> + recv_len, len);
> + return -EINVAL;
> + }
> +
> + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> +
> + if ( recv_len > 0 )
> + {
> + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> + }
> +
> + return 0;
> +}
> +
> +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> + void *rx_data, int rx_size)
> +{
> + int ret = 0;
> +
> + if ( !hdr )
> + return -EINVAL;
> +
> + spin_lock(&channel->lock);
> +
> + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> + if ( ret )
> + goto clean;
> +
> + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> +clean:
> + spin_unlock(&channel->lock);
> +
> + return ret;
> +}
> +
> +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
> + if ( curr->chan_id == chan_id )
> + {
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
> + if ( curr->domain_id == domain_id )
> + {
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
> + if ( (curr->domain_id == DOMID_INVALID)
> + && (curr->chan_id != HYP_CHANNEL) )
> + {
> + curr->domain_id = domain_id;
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static void relinquish_scmi_channel(struct scmi_channel *channel)
> +{
> + spin_lock(&scmi_data.channel_list_lock);
> + ASSERT(channel != NULL);
> + channel->domain_id = DOMID_INVALID;
> + spin_unlock(&scmi_data.channel_list_lock);
> +}
> +
> +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> + uint32_t func_id, uint64_t addr)
> +{
> + struct scmi_channel *channel;
> + mfn_t mfn;
> +
> + channel = get_channel_by_id(chan_id);
> + if ( channel )
> + return ERR_PTR(EEXIST);
> +
> + channel = xmalloc(struct scmi_channel);
> + if ( !channel )
> + return ERR_PTR(ENOMEM);
> +
> + channel->chan_id = chan_id;
> + channel->func_id = func_id;
> + channel->domain_id = DOMID_INVALID;
> + mfn = maddr_to_mfn(addr);
> + channel->shmem = vmap(&mfn, 1);
> + if ( !channel->shmem )
> + {
> + xfree(channel);
> + return ERR_PTR(ENOMEM);
> + }
> +
> + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> + channel->paddr = addr;
> + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> + spin_lock_init(&channel->lock);
> + spin_lock(&scmi_data.channel_list_lock);
> + list_add(&channel->list, &scmi_data.channel_list);
> + spin_unlock(&scmi_data.channel_list_lock);
> + return channel;
> +}
> +
> +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> +{
> + return iomem_permit_access(d, paddr_to_pfn(addr),
> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> +}
> +
> +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> + uint64_t len)
> +{
> + return iomem_deny_access(d, paddr_to_pfn(addr),
> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> +}
I wonder, why we need an extra level of indirection here. And if this is
really needed, I wonder why map(unmap)_memory* name was chosen, as there
is no memory mapping/unmapping really happens here.
> +
> +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> + uint64_t size)
Looks like *d is not used in this function.
> +{
> + struct dt_device_node *shmem_node;
> + __be32 *hw_reg;
> + const struct dt_property *pp;
> + uint32_t len;
> +
> + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
> +
> + if ( !shmem_node )
> + {
> + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> + return -EINVAL;
> + }
> +
> + pp = dt_find_property(shmem_node, "reg", &len);
> + if ( !pp )
> + {
> + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> + return -ENOENT;
> + }
> +
> + hw_reg = pp->value;
> + dt_set_range(&hw_reg, shmem_node, addr, size);
> +
> + return 0;
> +}
> +
> +static void free_channel_list(void)
> +{
> + struct scmi_channel *curr, *_curr;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> + {
> + vunmap(curr->shmem);
> + list_del(&curr->list);
> + xfree(curr);
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> +}
> +
> +static __init bool scmi_probe(struct dt_device_node *scmi_node)
Generic question to consider ...
If probe fails for some reason (so we cannot use mediator in Xen) what
should we do with SCMI nodes in domain's device-tree (leave as is, drop,
etc)?
> +{
> + struct dt_device_node *shmem_node;
> + int ret, i;
> + struct scmi_channel *channel, *agent_channel;
> + int n_agents;
> + scmi_msg_header_t hdr;
> + struct rx_t {
> + int32_t status;
> + uint32_t attributes;
> + } rx;
> +
> + uint32_t func_id;
> +
> + ASSERT(scmi_node != NULL);
> +
> + INIT_LIST_HEAD(&scmi_data.channel_list);
> + spin_lock_init(&scmi_data.channel_list_lock);
> +
> + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> + {
> + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> + return false;
> + }
> +
> + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
> + if ( IS_ERR_OR_NULL(shmem_node) )
> + {
> + printk(XENLOG_ERR
> + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> + PTR_ERR(shmem_node));
> + return false;
> + }
> +
> + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> + &scmi_data.shmem_size);
> + if ( IS_ERR_VALUE(ret) )
> + return false;
> +
> + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> + if ( IS_ERR(channel) )
> + return false;
> +
> + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> + if ( ret )
> + goto clean;
> +
> + ret = check_scmi_status(rx.status);
> + if ( ret )
> + goto clean;
> +
> + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> +
> + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> +
> + for ( i = 1; i < n_agents; i++ )
> + {
> + uint32_t tx_agent_id = 0xFFFFFFFF;
> + struct {
> + int32_t status;
> + uint32_t agent_id;
> + char name[16];
> + } da_rx;
> +
> + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> + i * PAGE_SIZE);
> + if ( IS_ERR(agent_channel) )
> + {
> + ret = PTR_ERR(agent_channel);
> + goto clean;
> + }
> +
> + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> + if ( ret )
> + goto clean;
> +
> + ret = check_scmi_status(da_rx.status);
> + if ( ret )
> + goto clean;
> +
> + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> + da_rx.status, da_rx.agent_id, da_rx.name);
> +
> + agent_channel->agent_id = da_rx.agent_id;
> + }
> +
> + scmi_data.initialized = true;
> + return true;
> +
> +clean:
> + free_channel_list();
> + return ret == 0;
> +}
> +
> +static int scmi_domain_init(struct domain *d)
> +{
> + struct scmi_channel *channel;
> + int ret;
> +
> + if ( !scmi_data.initialized )
> + return 0;
> +
> + channel = aquire_scmi_channel(d->domain_id);
> + if ( IS_ERR_OR_NULL(channel) )
> + return -ENOENT;
> +
> + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> + channel->paddr);
It seems this breaks build on Arm32:
scmi_smc.c: In function ‘scmi_domain_init’:
/home/otyshchenko/xen/xen/include/xen/config.h:53:24: error: format
‘%lx’ expects argument of type ‘long unsigned int’, but argument 4 has
type ‘uint64_t {aka long long unsigned int}’ [-Werror=format=]
#define XENLOG_INFO "<2>"
^
scmi_smc.c:569:12: note: in expansion of macro ‘XENLOG_INFO’
printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x ,
domain_id = %d"
^~~~~~~~~~~
scmi_smc.c:570:25: note: format string is defined here
"paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
~~^
%llx
> +
> + if ( is_hardware_domain(d) )
> + {
> + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> + scmi_data.shmem_size);
> + if ( IS_ERR_VALUE(ret) )
> + goto error;
> +
> + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> + if ( IS_ERR_VALUE(ret) )
> + {
> + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> + scmi_data.shmem_size);
> + if ( rc )
> + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> +
> + goto error;
> + }
> + }
> +
> + d->arch.sci = channel;
> +
> + return 0;
> +error:
> + relinquish_scmi_channel(channel);
> +
> + return ret;
> +}
> +
> +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> +{
> + struct scmi_channel *channel, *agent_channel;
> + scmi_msg_header_t hdr;
> + scmi_perms_tx_t tx;
> + struct rx_t {
> + int32_t status;
> + uint32_t attributes;
> + } rx;
> + int ret;
> +
> + if ( !scmi_data.initialized )
> + return 0;
> +
> + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> +
> + agent_channel = get_channel_by_domain(d->domain_id);
> + if ( IS_ERR_OR_NULL(agent_channel) )
> + return PTR_ERR(agent_channel);
> +
> + channel = get_channel_by_id(HYP_CHANNEL);
> + if ( IS_ERR_OR_NULL(channel) )
> + return PTR_ERR(channel);
> +
> + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + tx.agent_id = agent_channel->agent_id;
> + tx.device_id = scmi_devid;
> + tx.flags = SCMI_ALLOW_ACCESS;
> +
> + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + ret = check_scmi_status(rx.status);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + return 0;
> +}
> +
> +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> +{
> + uint32_t scmi_devid;
> +
> + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> + return 0;
> +
> + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> + return 0;
> +
> + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> +
> + return scmi_add_device_by_devid(d, scmi_devid);
> +}
> +
> +static int scmi_relinquish_resources(struct domain *d)
> +{
> + int ret;
> + struct scmi_channel *channel, *agent_channel;
> + scmi_msg_header_t hdr;
> + struct reset_agent_tx {
> + uint32_t agent_id;
> + uint32_t flags;
> + } tx;
> + uint32_t rx;
> +
> + if ( !d->arch.sci )
> + return 0;
> +
> + agent_channel = d->arch.sci;
> +
> + spin_lock(&agent_channel->lock);
> + tx.agent_id = agent_channel->agent_id;
> + spin_unlock(&agent_channel->lock);
> +
> + channel = get_channel_by_id(HYP_CHANNEL);
> + if ( !channel )
> + {
> + printk(XENLOG_ERR
> + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> + d->domain_id);
> + return -EINVAL;
> + }
> +
> + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + tx.flags = 0;
> +
> + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> + if ( ret )
> + return ret;
> +
> + ret = check_scmi_status(rx);
> +
> + return ret;
> +}
> +
> +static void scmi_domain_destroy(struct domain *d)
> +{
> + struct scmi_channel *channel;
> +
> + if ( !d->arch.sci )
> + return;
> +
> + channel = d->arch.sci;
> + spin_lock(&channel->lock);
> +
> + relinquish_scmi_channel(channel);
> + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> +
> + d->arch.sci = NULL;
> +
> + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
I didn't manage to find where corresponding map_memory_from_domain() is
called for a non-hardware domain (it seems that scmi_domain_init() only
directly handles hardware domain case).
Or perhaps, it is called indirectly at do_domctl(): case
XEN_DOMCTL_iomem_permission:?
I wonder, do we really need to call this here? Taking into the account
that unmap_memory_from_domain() doesn't actually unmap anything, but
only removes a range from the iomem_caps rangeset
for the domain to be destroyed and all involved rangesets (including
iomem_caps) will be removed soon at rangeset_domain_destroy() anyway. Or
I missed something?
> + spin_unlock(&channel->lock);
> + return;
empty return could be dropped, I think.
> +}
> +
> +static bool scmi_handle_call(struct domain *d, void *args)
> +{
> + bool res = false;
> + struct scmi_channel *agent_channel;
> + struct arm_smccc_res resp;
> + struct cpu_user_regs *regs = args;
> +
> + if ( !d->arch.sci )
> + return false;
> +
> + agent_channel = d->arch.sci;
> + spin_lock(&agent_channel->lock);
> +
> + if ( agent_channel->func_id != regs->x0 )
This also breaks build on Arm32:
scmi_smc.c: In function ‘scmi_handle_call’:
scmi_smc.c:736:42: error: ‘struct cpu_user_regs’ has no member named
‘x0’; did you mean ‘r0’?
if ( agent_channel->func_id != regs->x0 )
^~
r0
cc1: all warnings being treated as errors
***
BTW, I noticed that xen/arch/arm/traps.c contains the following
construct, probably we might want something similar here?
#ifdef CONFIG_ARM_64
#define HYPERCALL_RESULT_REG(r) (r)->x0
[snip]
#else
#define HYPERCALL_RESULT_REG(r) (r)->r0
[snip]
#endif
This RFC patch series, so I think, there is no serious issues at the
moment, this is rather to let you know for the future (when you drop RFC
tag).
I have to admit, I often forget to build-test on Arm32 also))
> + {
> + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> + goto unlock;
> + }
> +
> + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> + agent_channel->chan_id, &resp);
> +
> + set_user_reg(regs, 0, resp.a0);
> + set_user_reg(regs, 1, resp.a1);
> + set_user_reg(regs, 2, resp.a2);
> + set_user_reg(regs, 3, resp.a3);
> + res = true;
> +unlock:
> + spin_unlock(&agent_channel->lock);
> +
> + return res;
> +}
> +
> +static int scmi_get_channel_paddr(void *scmi_ops,
> + struct xen_arch_domainconfig *config)
> +{
> + struct scmi_channel *agent_channel = scmi_ops;
> +
> + if ( !agent_channel )
> + return -EINVAL;
> +
> + config->sci_agent_paddr = agent_channel->paddr;
> + return 0;
> +}
> +
> +static const struct dt_device_match scmi_smc_match[] __initconst =
> +{
> + DT_MATCH_SCMI_SMC,
> + { /* sentinel */ },
> +};
> +
> +static const struct sci_mediator_ops scmi_ops =
> +{
> + .probe = scmi_probe,
> + .domain_init = scmi_domain_init,
> + .domain_destroy = scmi_domain_destroy,
> + .add_dt_device = scmi_add_dt_device,
> + .relinquish_resources = scmi_relinquish_resources,
> + .handle_call = scmi_handle_call,
> + .get_channel_info = scmi_get_channel_paddr
> +};
> +
> +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> + scmi_smc_match, &scmi_ops);
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 9180be5e86..a67237942d 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
>
> #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
>
> struct xen_arch_domainconfig {
> /* IN/OUT */
--
Regards,
Oleksandr Tyshchenko
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-16 0:04 ` Oleksandr
@ 2021-12-17 12:15 ` Oleksii Moisieiev
2021-12-21 14:45 ` Anthony PERARD
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 12:15 UTC (permalink / raw)
To: Oleksandr
Cc: xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis
Hi Oleksandr,
On Thu, Dec 16, 2021 at 02:04:35AM +0200, Oleksandr wrote:
>
> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>
> Hi Oleksii
>
> > Integration of the SCMI mediator with xen libs:
> > - add hypercalls to aquire SCI channel and set device permissions
> > for DomUs;
> > - add SCMI_SMC nodes to DomUs device-tree based on partial device-tree;
> > - SCI requests redirection from DomUs to Firmware.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > tools/include/xenctrl.h | 3 +
> > tools/include/xenguest.h | 2 +
> > tools/libs/ctrl/xc_domain.c | 23 ++++++
> > tools/libs/guest/xg_dom_arm.c | 5 +-
> > tools/libs/light/libxl_arm.c | 122 +++++++++++++++++++++++++++---
> > tools/libs/light/libxl_create.c | 54 ++++++++++++-
> > tools/libs/light/libxl_dom.c | 1 +
> > tools/libs/light/libxl_internal.h | 4 +
> > xen/arch/arm/domctl.c | 15 ++++
> > xen/include/public/domctl.h | 9 +++
> > 10 files changed, 223 insertions(+), 15 deletions(-)
> >
> > diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
> > index 07b96e6671..cdd14f465f 100644
> > --- a/tools/include/xenctrl.h
> > +++ b/tools/include/xenctrl.h
> > @@ -1238,6 +1238,9 @@ int xc_domain_getvnuma(xc_interface *xch,
> > int xc_domain_soft_reset(xc_interface *xch,
> > uint32_t domid);
> > +int xc_domain_add_sci_device(xc_interface *xch,
> > + uint32_t domid, char *path);
> > +
> > #if defined(__i386__) || defined(__x86_64__)
> > /*
> > * PC BIOS standard E820 types and structure.
> > diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
> > index 61d0a82f48..35c611ac73 100644
> > --- a/tools/include/xenguest.h
> > +++ b/tools/include/xenguest.h
> > @@ -242,6 +242,8 @@ struct xc_dom_image {
> > /* Number of vCPUs */
> > unsigned int max_vcpus;
> > +
> > + xen_pfn_t sci_shmem_gfn;
> > };
> > /* --- arch specific hooks ----------------------------------------- */
> > diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
> > index b155d6afd2..07bb390e17 100644
> > --- a/tools/libs/ctrl/xc_domain.c
> > +++ b/tools/libs/ctrl/xc_domain.c
> > @@ -2206,6 +2206,29 @@ int xc_domain_soft_reset(xc_interface *xch,
> > domctl.domain = domid;
> > return do_domctl(xch, &domctl);
> > }
> > +
> > +int xc_domain_add_sci_device(xc_interface *xch,
> > + uint32_t domid, char *path)
> > +{
> > + size_t size = strlen(path);
> > + int rc;
> > + DECLARE_DOMCTL;
> > + DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
> > +
> > + if ( xc_hypercall_bounce_pre(xch, path) )
> > + return -1;
> > +
> > + domctl.cmd = XEN_DOMCTL_add_sci_device;
> > + domctl.domain = domid;
> > + domctl.u.sci_device_op.size = size;
> > + set_xen_guest_handle(domctl.u.sci_device_op.path, path);
> > + rc = do_domctl(xch, &domctl);
> > +
> > + xc_hypercall_bounce_post(xch, path);
> > +
> > + return rc;
> > +}
> > +
> > /*
> > * Local variables:
> > * mode: C
> > diff --git a/tools/libs/guest/xg_dom_arm.c b/tools/libs/guest/xg_dom_arm.c
> > index 5e3b76355e..368a670c46 100644
> > --- a/tools/libs/guest/xg_dom_arm.c
> > +++ b/tools/libs/guest/xg_dom_arm.c
> > @@ -25,11 +25,12 @@
> > #include "xg_private.h"
> > -#define NR_MAGIC_PAGES 4
> > +#define NR_MAGIC_PAGES 5
> > #define CONSOLE_PFN_OFFSET 0
> > #define XENSTORE_PFN_OFFSET 1
> > #define MEMACCESS_PFN_OFFSET 2
> > #define VUART_PFN_OFFSET 3
> > +#define SCI_SHMEM_OFFSET 4
> > #define LPAE_SHIFT 9
> > @@ -69,11 +70,13 @@ static int alloc_magic_pages(struct xc_dom_image *dom)
> > dom->console_pfn = base + CONSOLE_PFN_OFFSET;
> > dom->xenstore_pfn = base + XENSTORE_PFN_OFFSET;
> > dom->vuart_gfn = base + VUART_PFN_OFFSET;
> > + dom->sci_shmem_gfn = base + SCI_SHMEM_OFFSET;
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->console_pfn);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->xenstore_pfn);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, base + MEMACCESS_PFN_OFFSET);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->vuart_gfn);
> > + xc_clear_domain_page(dom->xch, dom->guest_domid, dom->sci_shmem_gfn);
> > xc_hvm_param_set(dom->xch, dom->guest_domid, HVM_PARAM_CONSOLE_PFN,
> > dom->console_pfn);
> > diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> > index eef1de0939..73ef591822 100644
> > --- a/tools/libs/light/libxl_arm.c
> > +++ b/tools/libs/light/libxl_arm.c
> > @@ -8,6 +8,8 @@
> > #include <assert.h>
> > #include <xen/device_tree_defs.h>
> > +#define SCMI_NODE_PATH "/firmware/scmi"
> > +
> > static const char *gicv_to_string(libxl_gic_version gic_version)
> > {
> > switch (gic_version) {
> > @@ -101,6 +103,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
> > return ERROR_FAIL;
> > }
> > + switch (d_config->b_info.sci) {
> > + case LIBXL_SCI_TYPE_NONE:
> > + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_NONE;
> > + break;
> > + case LIBXL_SCI_TYPE_SCMI_SMC:
> > + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_SCMI_SMC;
> > + break;
> > + default:
> > + LOG(ERROR, "Unknown SCI type %d",
> > + d_config->b_info.sci);
> > + return ERROR_FAIL;
> > + }
> > +
> > return 0;
> > }
> > @@ -122,6 +137,7 @@ int libxl__arch_domain_save_config(libxl__gc *gc,
> > }
> > state->clock_frequency = config->arch.clock_frequency;
> > + state->sci_agent_paddr = config->arch.sci_agent_paddr;
> > return 0;
> > }
> > @@ -502,9 +518,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> > int res;
> > LOG(DEBUG, "Creating OP-TEE node in dtb");
> > - res = fdt_begin_node(fdt, "firmware");
> > - if (res) return res;
> > -
> > res = fdt_begin_node(fdt, "optee");
> > if (res) return res;
> > @@ -517,9 +530,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> > res = fdt_end_node(fdt);
> > if (res) return res;
> > - res = fdt_end_node(fdt);
> > - if (res) return res;
> > -
> > return 0;
> > }
> > @@ -902,10 +912,9 @@ static int copy_node(libxl__gc *gc, void *fdt, void *pfdt,
> > return 0;
> > }
> > -static int copy_node_by_path(libxl__gc *gc, const char *path,
> > - void *fdt, void *pfdt)
> > +static int get_path_nodeoff(const char *path, void *pfdt)
> > {
> > - int nodeoff, r;
> > + int nodeoff;
> > const char *name = strrchr(path, '/');
> > if (!name)
> > @@ -925,12 +934,101 @@ static int copy_node_by_path(libxl__gc *gc, const char *path,
> > if (strcmp(fdt_get_name(pfdt, nodeoff, NULL), name))
> > return -FDT_ERR_NOTFOUND;
> > + return nodeoff;
> > +}
> > +
> > +static int copy_node_by_path(libxl__gc *gc, const char *path,
> > + void *fdt, void *pfdt)
> > +{
> > + int nodeoff, r;
> > +
> > + nodeoff = get_path_nodeoff(path, pfdt);
> > + if (nodeoff < 0)
> > + return nodeoff;
> > +
> > r = copy_node(gc, fdt, pfdt, nodeoff, 0);
> > if (r) return r;
> > return 0;
> > }
> > +static int get_node_phandle(const char *path, void *pfdt, uint32_t *phandle)
> > +{
> > + int nodeoff;
> > + const char *name = strrchr(path, '/');
> > +
> > + if (!name)
> > + return -FDT_ERR_INTERNAL;
> > +
> > + name++;
> > + nodeoff = fdt_path_offset(pfdt, path);
> > + if (nodeoff < 0)
> > + return nodeoff;
> > +
> > + *phandle = fdt_get_phandle(pfdt, nodeoff);
> > + return 0;
> > +}
> > +
> > +static int make_scmi_shmem_node(libxl__gc *gc, void *fdt, void *pfdt,
> > + struct xc_dom_image *dom)
> > +{
> > + int res;
> > + char buf[64];
> > + uint32_t phandle = 0;
> > +
> > + res = get_node_phandle("/scp-shmem", pfdt, &phandle);
> > + if (res) return res;
> > +
> > + snprintf(buf, sizeof(buf), "scp-shmem@%lx",
> > + dom->sci_shmem_gfn << XC_PAGE_SHIFT);
> > + res = fdt_begin_node(fdt, buf);
> > + if (res) return res;
> > +
> > + res = fdt_property_compat(gc, fdt, 1, "arm,scmi-shmem");
> > + if (res) return res;
> > +
> > + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> > + GUEST_ROOT_SIZE_CELLS, 1,
> > + dom->sci_shmem_gfn << XC_PAGE_SHIFT, XC_PAGE_SHIFT);
> > + if (res) return res;
> > +
> > + LOG(DEBUG, "scmi: setting phandle = %u\n", phandle);
> > + res = fdt_property_cell(fdt, "phandle", phandle);
> > + if (res) return res;
> > +
> > + res = fdt_end_node(fdt);
> > + if (res) return res;
> > +
> > + return 0;
> > +}
> > +
> > +static int make_firmware_node(libxl__gc *gc, void *fdt, void *pfdt, int tee,
> > + int sci)
> > +{
> > + int res;
> > +
> > + if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci != LIBXL_SCI_TYPE_NONE))
> > + return 0;
> > +
> > + res = fdt_begin_node(fdt, "firmware");
> > + if (res) return res;
> > +
> > + if (tee == LIBXL_TEE_TYPE_OPTEE) {
> > + res = make_optee_node(gc, fdt);
> > + if (res) return res;
> > + }
> > +
> > + if (sci == LIBXL_SCI_TYPE_SCMI_SMC) {
> > + res = copy_node_by_path(gc, SCMI_NODE_PATH, fdt, pfdt);
> > + if (res) return res;
> > + }
> > +
> > + res = fdt_end_node(fdt);
> > + if (res) return res;
> > +
> > + return 0;
> > +}
> > +
> > /*
> > * The partial device tree is not copied entirely. Only the relevant bits are
> > * copied to the guest device tree:
> > @@ -1088,8 +1186,10 @@ next_resize:
> > if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART)
> > FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) );
> > - if (info->tee == LIBXL_TEE_TYPE_OPTEE)
> > - FDT( make_optee_node(gc, fdt) );
> > + FDT( make_firmware_node(gc, fdt, pfdt, info->tee, info->sci) );
> > +
> > + if (info->sci == LIBXL_SCI_TYPE_SCMI_SMC)
> > + FDT( make_scmi_shmem_node(gc, fdt, pfdt, dom) );
> > if (d_config->num_pcidevs)
> > FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> > diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
> > index dcd09d32ba..c7372fd344 100644
> > --- a/tools/libs/light/libxl_create.c
> > +++ b/tools/libs/light/libxl_create.c
> > @@ -596,6 +596,37 @@ out:
> > return ret;
> > }
> > +static int map_sci_page(libxl__gc *gc, uint32_t domid, uint64_t paddr,
> > + uint64_t guest_addr)
> > +{
> > + int ret;
> > + uint64_t _paddr_pfn = paddr >> XC_PAGE_SHIFT;
> > + uint64_t _guest_pfn = guest_addr >> XC_PAGE_SHIFT;
> > +
> > + LOG(DEBUG, "iomem %"PRIx64, _paddr_pfn);
> > +
> > + ret = xc_domain_iomem_permission(CTX->xch, domid, _paddr_pfn, 1, 1);
> > + if (ret < 0) {
> > + LOG(ERROR,
> > + "failed give domain access to iomem page %"PRIx64,
> > + _paddr_pfn);
> > + return ret;
> > + }
> > +
> > + ret = xc_domain_memory_mapping(CTX->xch, domid,
> > + _guest_pfn, _paddr_pfn,
> > + 1, 1);
> > + if (ret < 0) {
> > + LOG(ERROR,
> > + "failed to map to domain iomem page %"PRIx64
> > + " to guest address %"PRIx64,
> > + _paddr_pfn, _guest_pfn);
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> > libxl__domain_build_state *state,
> > uint32_t *domid, bool soft_reset)
> > @@ -762,6 +793,16 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> > goto out;
> > }
> > + if (state->sci_agent_paddr != 0) {
> > + ret = map_sci_page(gc, *domid, state->sci_agent_paddr,
> > + state->sci_shmem_gfn << XC_PAGE_SHIFT);
> > + if (ret < 0) {
> > + LOGED(ERROR, *domid, "map SCI page fail");
> > + rc = ERROR_FAIL;
> > + goto out;
> > + }
> > + }
> > +
> > dom_path = libxl__xs_get_dompath(gc, *domid);
> > if (!dom_path) {
> > rc = ERROR_FAIL;
> > @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> > {
> > AO_GC;
> > libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> > - int i, rc = 0;
> > + int i, rc = 0, rc_sci = 0;
> > for (i = 0; i < d_config->num_dtdevs; i++) {
> > const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
> > LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> > rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> > - if (rc < 0) {
> > - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> > + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> > +
> > + if ((rc < 0) && (rc_sci < 0)) {
> > + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> > + "xc_domain_add_sci_device failed: %d",
> > + rc, rc_sci);
> > goto out;
> > }
> > +
> > + if (rc)
> > + rc = rc_sci;
>
>
> If I get this code right, it sounds like the dom.cfg's dtdev property is
> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
> (and can not) differentiate them. So it has no option but to send two
> domctls for each device in dtdevs[] hoping to at least one domctl to
> succeeded. Or I really missed something?
>
> It feels to me that:
> - either new dom.cfg's property could be introduced (scidev?) to describe
> sci devices together with new parsing logic/management code, so you will end
> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
> so no mixing things.
> - or indeed dtdev logic could be *completely* reused including extending
> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
> is used to describe devices for the passthrough (to configure an IOMMU for
> the device), in that case you wouldn't need an extra
> XEN_DOMCTL_add_sci_device introduced by current patch.
>
> Personally I would use the first option as I am not sure that second option
> is conceptually correct && possible. I would leave this for the maintainers
> to clarify.
>
Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
seems not to be conceptually correct. Introducing new dom.cfg property
seems to be the only way to avoid extra Domctl calls.
I will handle this in v2 if there will be no complains from Maintainers.
---
Oleksii.
>
>
> > }
> > out:
> > diff --git a/tools/libs/light/libxl_dom.c b/tools/libs/light/libxl_dom.c
> > index fe9f760f71..b1d288a8b9 100644
> > --- a/tools/libs/light/libxl_dom.c
> > +++ b/tools/libs/light/libxl_dom.c
> > @@ -710,6 +710,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
> > state->console_mfn = dom->console_pfn;
> > state->store_mfn = dom->xenstore_pfn;
> > state->vuart_gfn = dom->vuart_gfn;
> > + state->sci_shmem_gfn = dom->sci_shmem_gfn;
> > } else {
> > state->console_mfn = xc_dom_p2m(dom, dom->console_pfn);
> > state->store_mfn = xc_dom_p2m(dom, dom->xenstore_pfn);
> > diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
> > index 0b4671318c..f9f9cc633a 100644
> > --- a/tools/libs/light/libxl_internal.h
> > +++ b/tools/libs/light/libxl_internal.h
> > @@ -1407,6 +1407,10 @@ typedef struct {
> > /* Whether this domain is being migrated/restored, or booting fresh. Only
> > * applicable to the primary domain, not support domains (e.g. stub QEMU). */
> > bool restore;
> > +
> > + /* sci channel paddr to be set to device-tree node */
> > + uint64_t sci_agent_paddr;
> > + xen_pfn_t sci_shmem_gfn;
> > } libxl__domain_build_state;
> > _hidden void libxl__domain_build_state_init(libxl__domain_build_state *s);
> > diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
> > index 6245af6d0b..ba200407da 100644
> > --- a/xen/arch/arm/domctl.c
> > +++ b/xen/arch/arm/domctl.c
> > @@ -4,6 +4,7 @@
> > * Copyright (c) 2012, Citrix Systems
> > */
> > +#include <asm/sci/sci.h>
> > #include <xen/errno.h>
> > #include <xen/guest_access.h>
> > #include <xen/hypercall.h>
> > @@ -175,6 +176,20 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
> > return rc;
> > }
> > + case XEN_DOMCTL_add_sci_device:
> > + {
> > + int rc;
> > + struct dt_device_node *dev;
> > +
> > + rc = dt_find_node_by_gpath(domctl->u.sci_device_op.path,
> > + domctl->u.sci_device_op.size,
> > + &dev);
> > + if ( rc )
> > + return rc;
> > +
> > + return sci_add_dt_device(d, dev);
> > + }
> > +
> > default:
> > {
> > int rc;
> > diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> > index b85e6170b0..671c72c3e8 100644
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> > #define XEN_DOMCTL_vmtrace_get_option 5
> > #define XEN_DOMCTL_vmtrace_set_option 6
> > };
> > +
> > +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> > +struct xen_domctl_sci_device_op {
> > + uint32_t size; /* Length of the path */
> > + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> > +};
> > +
> > typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
> > DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
> > @@ -1265,6 +1272,7 @@ struct xen_domctl {
> > #define XEN_DOMCTL_get_cpu_policy 82
> > #define XEN_DOMCTL_set_cpu_policy 83
> > #define XEN_DOMCTL_vmtrace_op 84
> > +#define XEN_DOMCTL_add_sci_device 85
> > #define XEN_DOMCTL_gdbsx_guestmemio 1000
> > #define XEN_DOMCTL_gdbsx_pausevcpu 1001
> > #define XEN_DOMCTL_gdbsx_unpausevcpu 1002
> > @@ -1326,6 +1334,7 @@ struct xen_domctl {
> > struct xen_domctl_psr_alloc psr_alloc;
> > struct xen_domctl_vuart_op vuart_op;
> > struct xen_domctl_vmtrace_op vmtrace_op;
> > + struct xen_domctl_sci_device_op sci_device_op;
> > uint8_t pad[128];
> > } u;
> > };
>
> I assume the XSM needs updating (adding new hook, etc).
>
>
> --
> Regards,
>
> Oleksandr Tyshchenko
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 0/5] Introduce SCI-mediator feature
2021-12-16 0:33 ` [RFC v1 0/5] Introduce SCI-mediator feature Oleksandr
@ 2021-12-17 12:24 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 12:24 UTC (permalink / raw)
To: Oleksandr
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Anthony PERARD, Juergen Gross
Hi Oleksandr,
On Thu, Dec 16, 2021 at 02:33:28AM +0200, Oleksandr wrote:
>
> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>
>
> Hi Oleksii
>
> > Introducing the feature, called SCI mediator.
> > It's purpose is to redirect SCMI requests from the domains to firmware
> > (SCP, ATF etc), which controls the power/clock/resets etc.
> > The idea is to make SCP firmware (or similar, such as AT-F) responsible for
> > control power/clock/resets and provide SCMI interface so controls can be shared
> > between the Domains.
> > Originally, we've met a problem, that the devices, shared between different
> > Domains, can't have an access to HW registers to work with clocks/resets/power
> > etc. You have to pass cpg to the Domain, so the devices can access HW directly.
> > The solution for this is to move HW controls over power/clock/resets to
> > SCP firmware and use Linux-kernel SCMI drivers to pass requests to SCP.
> > Xen is responsible for permissions setting, so Domain can access only to
> > power/clock/resets which are related to this Domain. Also XEN is the mediator
> > which redirects SCMI requests, adding agentID so firmware should know the
> > sender.
> > SMC is currently used as transport, but this should be configurable.
> >
> > Here is the high level design:
> >
> > SCI (System Control Interface) feature can be enabled in xen_config:
> > > CONFIG_SCI=y
> > Mediator can be configured:
> > > CONFIG_SCMI_SMC=y
> > Currently, only SCMI_SMC mediator is implemented, which using shared memory
> > region to communicate with firmware and SMC as transport.
> >
> > Xen scmi should be configured in the device-tree.
> > Format is the following:
> > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > };
> >
> > firmware {
> > scmi {
> > compatible = "arm,scmi-smc";
> > arm,smc-id = <0x82000002>;
> > shmem = <&cpu_scp_shm>;
> > #address-cells = <1>;
> > #size-cells = <0>;
> >
> > scmi_power: protocol@11 {
> > reg = <0x11>;
> > #power-domain-cells = <1>;
> > };
> >
> > scmi_clock: protocol@14 {
> > reg = <0x14>;
> > #clock-cells = <1>;
> > };
> >
> > scmi_reset: protocol@16 {
> > reg = <0x16>;
> > #reset-cells = <1>;
> > };
> > };
> > };
> >
> > Where:
> > &cpu_scp_shm is the shared memory for scmi buffers;
> > 0x53FF0000, size 0x1000 is the platform specific free address, which provide
> > space for the communication.
> > &scmi node, which should be copied to Dom0 device-tree.
> >
> > Device configured to use scmi:
> > &avb {
> > scmi_devid = <0>;
> > clocks = <&scmi_clock 0>;
> > power-domains = <&scmi_power 0>;
> > resets = <&scmi_reset 0>;
> > };
> >
> > Where:
> > scmi_devid - id from the firmware, which is assigned for AVB.
> >
> > During initialization, XEN scans probes the first SCI-mediator driver which has
> > matching node in the device-tree. If no device-tree was provided, then the
> > first registered mediator driver should be probed.
> >
> > DomX should be configured:
> > Device-tree should include the same nodes, described above.
> > &cpu_scp_shm should be altered during domain creation. Xen allocates free page
> > from the memory region, provided in &cpu_scp_shm in XEN device-tree, so each
> > domain should have unique page. Nodes &cpu_scp_shm and /firmware/scmi should be
> > copied from partial device-tree to domain device-tree, so kernel can initialize
> > scmi driver.
> >
> > SCI mediator can be enabled in dom.cfg the following way:
> > > sci = "scmi_smc"
> > which sets scmi_smc to be used for the domain.
>
>
> Great work! I can imagine this is going to be nice feature once upstreamed.
>
> I am wondering, would the Xen (with the required updates of course) also be
> able to send it's own requests to the SCP? For example, to control overall
> system performance (CPU frequency)
>
> or other let's say important power management task.
>
I think yes. In current implementation Xen register itself as
privilleged agent and use it's own channel to request
which is used to get agent configuration and process device permissions.
I think this channel can also be used to do some configuration tasks
via SCMI. But this will require updates.
--
Oleksii.
>
> >
> > Oleksii Moisieiev (5):
> > xen/arm: add support for Renesas R-Car Gen3 platform
> > xen/arm: add generic SCI mediator framework
> > xen/arm: introduce SCMI-SMC mediator driver
> > tools/arm: add "scmi_smc" option to xl.cfg
> > xen/arm: add SCI mediator support for DomUs
> >
> > MAINTAINERS | 6 +
> > docs/man/xl.cfg.5.pod.in | 22 +
> > tools/include/libxl.h | 5 +
> > tools/include/xenctrl.h | 3 +
> > tools/include/xenguest.h | 2 +
> > tools/libs/ctrl/xc_domain.c | 23 +
> > tools/libs/guest/xg_dom_arm.c | 5 +-
> > tools/libs/light/libxl_arm.c | 122 ++++-
> > tools/libs/light/libxl_create.c | 54 +-
> > tools/libs/light/libxl_dom.c | 1 +
> > tools/libs/light/libxl_internal.h | 4 +
> > tools/libs/light/libxl_types.idl | 6 +
> > tools/xl/xl_parse.c | 9 +
> > xen/arch/arm/Kconfig | 10 +
> > xen/arch/arm/Makefile | 1 +
> > xen/arch/arm/domain.c | 24 +
> > xen/arch/arm/domain_build.c | 11 +
> > xen/arch/arm/domctl.c | 15 +
> > xen/arch/arm/platforms/Makefile | 1 +
> > xen/arch/arm/platforms/rcar3.c | 47 ++
> > xen/arch/arm/sci/Kconfig | 10 +
> > xen/arch/arm/sci/Makefile | 2 +
> > xen/arch/arm/sci/sci.c | 128 +++++
> > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++
> > xen/arch/arm/setup.c | 1 +
> > xen/arch/arm/xen.lds.S | 7 +
> > xen/include/asm-arm/domain.h | 4 +
> > xen/include/asm-arm/sci/sci.h | 162 ++++++
> > xen/include/public/arch-arm.h | 11 +
> > xen/include/public/domctl.h | 9 +
> > 30 files changed, 1485 insertions(+), 15 deletions(-)
> > create mode 100644 xen/arch/arm/platforms/rcar3.c
> > create mode 100644 xen/arch/arm/sci/Kconfig
> > create mode 100644 xen/arch/arm/sci/Makefile
> > create mode 100644 xen/arch/arm/sci/sci.c
> > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > create mode 100644 xen/include/asm-arm/sci/sci.h
> >
> --
> Regards,
>
> Oleksandr Tyshchenko
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 2/5] xen/arm: add generic SCI mediator framework
2021-12-17 2:45 ` Stefano Stabellini
@ 2021-12-17 12:50 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 12:50 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Andrew Cooper, George Dunlap, Jan Beulich,
Julien Grall, Wei Liu, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Thu, Dec 16, 2021 at 06:45:11PM -0800, Stefano Stabellini wrote:
> On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > This patch adds the basic framework for SCI mediator.
> > SCI is System Control Interface, which is designed to redirect
> > requests for the Hardware (such as power-domain/clock/resets etc)
> > from the Domains to the firmware. Originally, cpg should be passed
> > to the domain so it can work with power-domains/clocks/resets etc.
> > Considering that cpg can't be split between the Domains, we get the
> > limitation that the devices, which are using power-domains/clocks/resets
> > etc, couldn't be split between the domains.
> > The solution is to move the power-domain/clock/resets etc to the
> > Firmware (such as SCP firmware or ATF) and provide interface for the Domains.
> > XEN shoud have an entity, caled SCI-Mediator, which is responsible for messages
> > redirection between Domains and Firmware and for permission handling.
> >
> > This is how it works: user can build XEN with multiple SCI mediators.
> > See the next patches, where SCMI-SMC mediator is introduced.
> > SCI mediator register itself with REGISTER_SCI_MEDIATOR() macro.
> >
> > At run-time, during initialization, framework calls probe for the first
> > matching device in the device-tree. When no device-tree is present - the
> > first registered mediator should be probed.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > MAINTAINERS | 6 ++
> > xen/arch/arm/Kconfig | 8 ++
> > xen/arch/arm/Makefile | 1 +
> > xen/arch/arm/domain.c | 24 +++++
> > xen/arch/arm/domain_build.c | 11 +++
> > xen/arch/arm/platforms/rcar3.c | 3 +-
> > xen/arch/arm/sci/Makefile | 1 +
> > xen/arch/arm/sci/sci.c | 128 ++++++++++++++++++++++++++
> > xen/arch/arm/setup.c | 1 +
> > xen/arch/arm/xen.lds.S | 7 ++
> > xen/include/asm-arm/domain.h | 4 +
> > xen/include/asm-arm/sci/sci.h | 162 +++++++++++++++++++++++++++++++++
> > xen/include/public/arch-arm.h | 10 ++
> > 13 files changed, 365 insertions(+), 1 deletion(-)
> > create mode 100644 xen/arch/arm/sci/Makefile
> > create mode 100644 xen/arch/arm/sci/sci.c
> > create mode 100644 xen/include/asm-arm/sci/sci.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index e43dc0edce..5f96ea35ba 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -512,6 +512,12 @@ S: Supported
> > F: xen/arch/arm/tee/
> > F: xen/include/asm-arm/tee
> >
> > +SCI MEDIATORS
> > +M: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > +S: Supported
> > +F: xen/arch/arm/sci
> > +F: xen/include/asm-arm/sci
> > +
> > TOOLSTACK
> > M: Wei Liu <wl@xen.org>
> > S: Supported
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index ecfa6822e4..186e1db389 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -106,6 +106,14 @@ config TEE
> >
> > source "arch/arm/tee/Kconfig"
> >
> > +config SCI
> > + bool "Enable SCI mediators support"
> > + default n
> > + help
> > + This option enables generic SCI (System Control Interface) mediators
> > + support. It allows guests to control system resourcess via one of
> > + SCI mediators implemented in XEN.
> > +
> > endmenu
> >
> > menu "ARM errata workaround via the alternative framework"
> > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> > index 07f634508e..6366ff55e5 100644
> > --- a/xen/arch/arm/Makefile
> > +++ b/xen/arch/arm/Makefile
> > @@ -8,6 +8,7 @@ obj-y += platforms/
> > endif
> > obj-$(CONFIG_TEE) += tee/
> > obj-$(CONFIG_HAS_VPCI) += vpci.o
> > +obj-$(CONFIG_SCI) += sci/
> >
> > obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> > obj-y += bootfdt.init.o
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index 96e1b23550..80d0a23767 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -34,6 +34,7 @@
> > #include <asm/platform.h>
> > #include <asm/procinfo.h>
> > #include <asm/regs.h>
> > +#include <asm/sci/sci.h>
> > #include <asm/tee/tee.h>
> > #include <asm/vfp.h>
> > #include <asm/vgic.h>
> > @@ -688,6 +689,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> > return -EINVAL;
> > }
> >
> > + if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE &&
> > + config->arch.sci_type != sci_get_type() )
> > + {
> > + dprintk(XENLOG_INFO, "Unsupported SCI type\n");
> > + return -EINVAL;
> > + }
> > +
> > return 0;
> > }
> >
> > @@ -764,6 +772,15 @@ int arch_domain_create(struct domain *d,
> > /* At this stage vgic_reserve_virq should never fail */
> > if ( !vgic_reserve_virq(d, GUEST_EVTCHN_PPI) )
> > BUG();
> > +
> > + if ( config->arch.sci_type != XEN_DOMCTL_CONFIG_SCI_NONE )
> > + {
> > + if ( (rc = sci_domain_init(d, config->arch.sci_type)) != 0)
> > + goto fail;
> > +
> > + if ( (rc = sci_get_channel_info(d, &config->arch)) != 0)
> > + goto fail;
> > + }
> > }
> >
> > /*
> > @@ -796,6 +813,7 @@ void arch_domain_destroy(struct domain *d)
> > domain_vgic_free(d);
> > domain_vuart_free(d);
> > free_xenheap_page(d->shared_info);
> > + sci_domain_destroy(d);
> > #ifdef CONFIG_ACPI
> > free_xenheap_pages(d->arch.efi_acpi_table,
> > get_order_from_bytes(d->arch.efi_acpi_len));
> > @@ -996,6 +1014,7 @@ enum {
> > PROG_xen,
> > PROG_page,
> > PROG_mapping,
> > + PROG_sci,
> > PROG_done,
> > };
> >
> > @@ -1056,6 +1075,11 @@ int domain_relinquish_resources(struct domain *d)
> > if ( ret )
> > return ret;
> >
> > + PROGRESS(sci):
> > + ret = sci_relinquish_resources(d);
> > + if ( ret )
> > + return ret;
> > +
> > PROGRESS(done):
> > break;
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index d02bacbcd1..38874615dd 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -22,6 +22,7 @@
> > #include <asm/kernel.h>
> > #include <asm/setup.h>
> > #include <asm/tee/tee.h>
> > +#include <asm/sci/sci.h>
> > #include <asm/platform.h>
> > #include <asm/psci.h>
> > #include <asm/setup.h>
> > @@ -1894,6 +1895,10 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
> > return res;
> > }
> > }
> > +
> > + res = sci_add_dt_device(d, dev);
> > + if ( res < 0 )
> > + return res;
> > }
>
> It would be good to add the call to dom0less domUs too
>
Thank you, I will fix all mentioned places in v2.
>
> > res = handle_device_interrupts(d, dev, need_mapping);
> > @@ -3075,6 +3080,10 @@ static int __init construct_dom0(struct domain *d)
> > if ( rc < 0 )
> > return rc;
> >
> > + rc = sci_domain_init(d, sci_get_type());
> > + if ( rc < 0 )
> > + return rc;
>
> same here
>
TBD in v2
>
> > if ( acpi_disabled )
> > rc = prepare_dtb_hwdom(d, &kinfo);
> > else
> > @@ -3109,6 +3118,8 @@ void __init create_dom0(void)
> > dom0_cfg.arch.tee_type = tee_get_type();
> > dom0_cfg.max_vcpus = dom0_max_vcpus();
> >
> > + dom0_cfg.arch.sci_type = sci_get_type();
>
> and here
>
TBD in v2
>
> > if ( iommu_enabled )
> > dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> >
> > diff --git a/xen/arch/arm/platforms/rcar3.c b/xen/arch/arm/platforms/rcar3.c
> > index d740145c71..a268b09454 100644
> > --- a/xen/arch/arm/platforms/rcar3.c
> > +++ b/xen/arch/arm/platforms/rcar3.c
> > @@ -18,10 +18,11 @@
> > */
> >
> > #include <asm/platform.h>
> > +#include <asm/sci/sci.h>
> >
> > static bool rcar3_smc(struct cpu_user_regs *regs)
> > {
> > - return false;
> > + return sci_handle_call(current->domain, regs);
> > }
> >
> > static const char *const rcar3_dt_compat[] __initconst =
> > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > new file mode 100644
> > index 0000000000..837dc7492b
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/Makefile
> > @@ -0,0 +1 @@
> > +obj-y += sci.o
> > diff --git a/xen/arch/arm/sci/sci.c b/xen/arch/arm/sci/sci.c
> > new file mode 100644
> > index 0000000000..5961b4cd5d
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/sci.c
> > @@ -0,0 +1,128 @@
> > +/*
> > + * xen/arch/arm/sci/sci.c
> > + *
> > + * Generic part of SCI mediator driver
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021, EPAM Systems.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <xen/errno.h>
> > +#include <xen/init.h>
> > +#include <xen/sched.h>
> > +#include <xen/types.h>
> > +
> > +#include <asm/sci/sci.h>
> > +
> > +extern const struct sci_mediator_desc _sscimediator[], _escimediator[];
> > +static const struct sci_mediator_desc __read_mostly *cur_mediator;
> > +
> > +bool sci_handle_call(struct domain *d, void *args)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return false;
> > +
> > + return cur_mediator->ops->handle_call(d, args);
> > +}
> > +
> > +int sci_domain_init(struct domain *d, uint16_t sci_type)
> > +{
> > + if ( sci_type == XEN_DOMCTL_CONFIG_SCI_NONE )
> > + return 0;
> > +
> > + if ( unlikely(!cur_mediator) )
> > + return -ENODEV;
> > +
> > + if ( cur_mediator->sci_type != sci_type )
> > + return -EINVAL;
> > +
> > + return cur_mediator->ops->domain_init(d);
> > +}
> > +
> > +void sci_domain_destroy(struct domain *d)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return;
> > +
> > + cur_mediator->ops->domain_destroy(d);
> > +}
> > +
> > +int sci_relinquish_resources(struct domain *d)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return 0;
> > +
> > + return cur_mediator->ops->relinquish_resources(d);
> > +}
> > +
> > +
> > +int sci_add_dt_device(struct domain *d, struct dt_device_node *dev)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return 0;
> > +
> > + return cur_mediator->ops->add_dt_device(d, dev);
> > +}
> > +
> > +int sci_get_channel_info(struct domain *d,
> > + struct xen_arch_domainconfig *config)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return 0;
> > +
> > + return cur_mediator->ops->get_channel_info(d->arch.sci, config);
> > +}
> > +
> > +uint16_t sci_get_type(void)
> > +{
> > + if ( unlikely(!cur_mediator) )
> > + return XEN_DOMCTL_CONFIG_SCI_NONE;
> > +
> > + return cur_mediator->sci_type;
> > +}
> > +
> > +static int __init sci_init(void)
> > +{
> > + const struct sci_mediator_desc *desc;
> > + struct dt_device_node *dt = NULL;
> > +
> > + for ( desc = _sscimediator; desc != _escimediator; desc++ )
> > + {
> > + if ( likely(dt_host) )
>
> I think you can assume that dt_host is correctly set, right?
> Instead of this check, I would just add:
>
> if ( !acpi_disabled )
> return -ENODEV;
>
> at the beginning of sci_init
>
Good point! I will it in v2. Thanks.
>
> > + {
> > + dt = dt_find_matching_node(dt_host, desc->dt_match);
> > + if ( !dt )
> > + continue;
> > + }
> > +
> > + if ( desc->ops->probe(dt) )
> > + {
> > + printk(XENLOG_INFO "Using SCI mediator for %s\n", desc->name);
> > + cur_mediator = desc;
> > + return 0;
> > + }
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +__initcall(sci_init);
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> > index d5d0792ed4..201de01411 100644
> > --- a/xen/arch/arm/setup.c
> > +++ b/xen/arch/arm/setup.c
> > @@ -50,6 +50,7 @@
> > #include <asm/cpufeature.h>
> > #include <asm/platform.h>
> > #include <asm/procinfo.h>
> > +#include <asm/sci/sci.h>
> > #include <asm/setup.h>
> > #include <xsm/xsm.h>
> > #include <asm/acpi.h>
> > diff --git a/xen/arch/arm/xen.lds.S b/xen/arch/arm/xen.lds.S
> > index 08016948ab..3683f4821f 100644
> > --- a/xen/arch/arm/xen.lds.S
> > +++ b/xen/arch/arm/xen.lds.S
> > @@ -142,6 +142,13 @@ SECTIONS
> > _eteemediator = .;
> > } :text
> >
> > + . = ALIGN(8);
> > + .scimediator.info : {
> > + _sscimediator = .;
> > + *(.scimediator.info)
> > + _escimediator = .;
> > + } :text
> > +
> > . = ALIGN(PAGE_SIZE); /* Init code and data */
> > __init_begin = .;
> > .init.text : {
> > diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> > index 9b3647587a..d9b164017f 100644
> > --- a/xen/include/asm-arm/domain.h
> > +++ b/xen/include/asm-arm/domain.h
> > @@ -89,6 +89,10 @@ struct arch_domain
> > #ifdef CONFIG_TEE
> > void *tee;
> > #endif
> > +
> > +#ifdef CONFIG_SCI
> > + void *sci;
> > +#endif
> > } __cacheline_aligned;
> >
> > struct arch_vcpu
> > diff --git a/xen/include/asm-arm/sci/sci.h b/xen/include/asm-arm/sci/sci.h
> > new file mode 100644
> > index 0000000000..aeff689c72
> > --- /dev/null
> > +++ b/xen/include/asm-arm/sci/sci.h
> > @@ -0,0 +1,162 @@
> > +/*
> > + * xen/include/asm-arm/sci/sci.h
> > + *
> > + * Generic part of the SCI (System Control Interface) subsystem.
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021, EPAM Systems.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#ifndef __ASM_ARM_SCI_H
> > +#define __ASM_ARM_SCI_H
> > +
> > +#include <xen/lib.h>
> > +#include <xen/types.h>
> > +#include <xen/device_tree.h>
> > +
> > +#ifdef CONFIG_SCI
> > +
> > +struct sci_mediator_ops {
> > +
> > + /*
> > + * Probe for SCI. Should return true if SCI found and
> > + * mediator is initialized.
> > + */
> > + bool (*probe)(struct dt_device_node *scmi_node);
> > +
> > + /*
> > + * Called during domain construction if toolstack requests to enable
> > + * SCI support so mediator can inform SCP-firmware about new
> > + * guest and create own structures for the new domain.
> > + */
> > + int (*domain_init)(struct domain *d);
> > +
> > + /*
> > + * Called during domain destruction, releases all resources, that
> > + * were allocated by the mediator.
> > + */
> > + void (*domain_destroy)(struct domain *d);
> > +
> > + /*
> > + * Called during parsing partial device-sci for the domain.
> > + * Passing device_node so mediator could process the device and
> > + * mark the device as related to the domain if needed.
> > + */
> > + int (*add_dt_device)(struct domain *d, struct dt_device_node *dev);
> > +
> > + /*
> > + * Called during domain destruction to relinquish resources used
> > + * by mediator itself. This function can return -ERESTART to indicate
> > + * that it does not finished work and should be called again.
> > + */
> > + int (*relinquish_resources)(struct domain *d);
> > +
> > + /* Handle call for current domain */
> > + bool (*handle_call)(struct domain *d, void *regs);
> > +
> > + /* Gets channel configuration and store it in domainconfig */
> > + int (*get_channel_info)(void *sci_ops,
> > + struct xen_arch_domainconfig *config);
>
> get_channel_info is the only function I don't understand among these...
> what is supposed to be stored in struct xen_arch_domainconfig, just
> sci_agent_paddr?
>
> Also, it seems to be only called right after sci_domain_init, so can't
> the "get_channel_info" operation just be done as part of domain_init?
>
We need sci_agent_paddr for Domains. It represents shmem address which
should be mapped to guest Domain. We also don't need paddr when we do
sci_domain_init for dom0. That's why I've made it as separate call.
Also I named it get_channel_info, not get_channel_paddr, because
potentially, the different information can be needed from different SCI
implementation, which doesn't need shared memory to pass data for
exapmle.
>
>
> > +};
> > +
> > +struct sci_mediator_desc {
> > + /* Printable name of the SCI. */
> > + const char *name;
> > +
> > + /* Mediator callbacks as described above. */
> > + const struct sci_mediator_ops *ops;
> > +
> > + /*
> > + * ID of SCI. Corresponds to xen_arch_domainconfig.sci_type.
> > + * Should be one of XEN_DOMCTL_CONFIG_SCI_xxx
> > + */
> > + uint16_t sci_type;
> > +
> > + /* Match structure to init mediator */
> > + const struct dt_device_match *dt_match;
> > +
> > +};
> > +
> > +int sci_domain_init(struct domain *d, uint16_t sci_type);
> > +void sci_domain_destroy(struct domain *d);
> > +int sci_add_dt_device(struct domain *d, struct dt_device_node *dev);
> > +int sci_relinquish_resources(struct domain *d);
> > +bool sci_handle_call(struct domain *d, void *args);
> > +int sci_get_channel_info(struct domain *d,
> > + struct xen_arch_domainconfig *config);
> > +uint16_t sci_get_type(void);
> > +
> > +#define REGISTER_SCI_MEDIATOR(_name, _namestr, _type, _match, _ops) \
> > +static const struct sci_mediator_desc __sci_desc_##_name __used \
> > +__section(".scimediator.info") = { \
> > + .name = _namestr, \
> > + .ops = _ops, \
> > + .sci_type = _type, \
> > + .dt_match = _match \
> > +}
> > +
> > +#else
> > +#include <public/errno.h>
> > +
> > +static inline int sci_domain_init(struct domain *d, uint16_t sci_type)
> > +{
> > + if ( likely(sci_type == XEN_DOMCTL_CONFIG_SCI_NONE) )
> > + return 0;
> > +
> > + return -XEN_ENODEV;
> > +}
> > +
> > +static inline void sci_domain_destroy(struct domain *d)
> > +{
> > +}
> > +
> > +static inline int sci_add_dt_device(struct domain *d,
> > + struct dt_device_node *dev)
> > +{
> > + return 0;
> > +}
> > +
> > +static inline int sci_relinquish_resources(struct domain *d)
> > +{
> > + return 0;
> > +}
> > +
> > +static inline bool sci_handle_call(struct domain *d, void *args)
> > +{
> > + return false;
> > +}
> > +
> > +static inline int sci_get_channel_info(struct domain *d,
> > + struct xen_arch_domainconfig *config)
> > +{
> > + return 0;
> > +}
> > +
> > +static inline uint16_t sci_get_type(void)
> > +{
> > + return XEN_DOMCTL_CONFIG_SCI_NONE;
> > +}
> > +
> > +#endif /* CONFIG_SCI */
> > +
> > +#endif /* __ASM_ARM_SCI_H */
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > index 94b31511dd..9180be5e86 100644
> > --- a/xen/include/public/arch-arm.h
> > +++ b/xen/include/public/arch-arm.h
> > @@ -314,12 +314,16 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> > #define XEN_DOMCTL_CONFIG_TEE_NONE 0
> > #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> >
> > +#define XEN_DOMCTL_CONFIG_SCI_NONE 0
> > +
> > struct xen_arch_domainconfig {
> > /* IN/OUT */
> > uint8_t gic_version;
> > /* IN */
> > uint16_t tee_type;
> > /* IN */
> > + uint8_t sci_type;
>
> sci_type is uint16_t everywhere else
>
>
> > + /* IN */
> > uint32_t nr_spis;
> > /*
> > * OUT
> > @@ -335,6 +339,12 @@ struct xen_arch_domainconfig {
> > *
> > */
> > uint32_t clock_frequency;
> > +
> > + /* Sets shared address to sw domains.
> > + * This information is needed to set correct channel in Domain partial
> > + * device-tree
> > + */
> > + uint64_t sci_agent_paddr;
> > };
> > #endif /* __XEN__ || __XEN_TOOLS__ */
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-17 11:35 ` Oleksandr
@ 2021-12-17 13:23 ` Oleksii Moisieiev
2021-12-17 13:37 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 13:23 UTC (permalink / raw)
To: Oleksandr
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
Hi Oleksandr,
On Fri, Dec 17, 2021 at 01:35:35PM +0200, Oleksandr wrote:
>
> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>
>
> Hi Oleksii
>
> > This is the implementation of SCI interface, called SCMI-SMC driver,
> > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > This allows devices from the Domains to work with clocks, resets and
> > power-domains without access to CPG.
> >
> > The following features are implemented:
> > - request SCMI channels from ATF and pass channels to Domains;
> > - set device permissions for Domains based on the Domain partial
> > device-tree. Devices with permissions are able to work with clocks,
> > resets and power-domains via SCMI;
> > - redirect scmi messages from Domains to ATF.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > xen/arch/arm/Kconfig | 2 +
> > xen/arch/arm/sci/Kconfig | 10 +
> > xen/arch/arm/sci/Makefile | 1 +
> > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > xen/include/public/arch-arm.h | 1 +
> > 5 files changed, 809 insertions(+)
> > create mode 100644 xen/arch/arm/sci/Kconfig
> > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index 186e1db389..02d96c6cfc 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -114,6 +114,8 @@ config SCI
> > support. It allows guests to control system resourcess via one of
> > SCI mediators implemented in XEN.
> > +source "arch/arm/sci/Kconfig"
> > +
> > endmenu
> > menu "ARM errata workaround via the alternative framework"
> > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > new file mode 100644
> > index 0000000000..9563067ddc
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/Kconfig
> > @@ -0,0 +1,10 @@
> > +config SCMI_SMC
> > + bool "Enable SCMI-SMC mediator driver"
> > + default n
> > + depends on SCI
> > + ---help---
> > +
> > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > + This feature allows drivers from Domains to work with System
> > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > + for communication.
> > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > index 837dc7492b..67f2611872 100644
> > --- a/xen/arch/arm/sci/Makefile
> > +++ b/xen/arch/arm/sci/Makefile
> > @@ -1 +1,2 @@
> > obj-y += sci.o
> > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > new file mode 100644
> > index 0000000000..2eb01ea82d
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/scmi_smc.c
> > @@ -0,0 +1,795 @@
> > +/*
> > + * xen/arch/arm/sci/scmi_smc.c
> > + *
> > + * SCMI mediator driver, using SCP as transport.
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021, EPAM Systems.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <asm/sci/sci.h>
> > +#include <asm/smccc.h>
> > +#include <asm/io.h>
> > +#include <xen/bitops.h>
> > +#include <xen/config.h>
> > +#include <xen/sched.h>
> > +#include <xen/device_tree.h>
> > +#include <xen/iocap.h>
> > +#include <xen/init.h>
> > +#include <xen/err.h>
> > +#include <xen/lib.h>
> > +#include <xen/list.h>
> > +#include <xen/mm.h>
> > +#include <xen/string.h>
> > +#include <xen/time.h>
> > +#include <xen/vmap.h>
> > +
> > +#define SCMI_BASE_PROTOCOL 0x10
> > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > +
> > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > +#define SCMI_SUCCESS 0
> > +#define SCMI_NOT_SUPPORTED (-1)
> > +#define SCMI_INVALID_PARAMETERS (-2)
> > +#define SCMI_DENIED (-3)
> > +#define SCMI_NOT_FOUND (-4)
> > +#define SCMI_OUT_OF_RANGE (-5)
> > +#define SCMI_BUSY (-6)
> > +#define SCMI_COMMS_ERROR (-7)
> > +#define SCMI_GENERIC_ERROR (-8)
> > +#define SCMI_HARDWARE_ERROR (-9)
> > +#define SCMI_PROTOCOL_ERROR (-10)
> > +
> > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > +
> > +#define SCMI_SMC_ID "arm,smc-id"
> > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > +#define SCMI_SHMEM "shmem"
> > +
> > +#define HYP_CHANNEL 0x0
> > +
> > +#define HDR_ID GENMASK(7,0)
> > +#define HDR_TYPE GENMASK(9, 8)
> > +#define HDR_PROTO GENMASK(17, 10)
> > +
> > +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> > +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> > +
> > +#define FIELD_GET(_mask, _reg)\
> > + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> > +#define FIELD_PREP(_mask, _val)\
> > + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> > +
> > +typedef struct scmi_msg_header {
> > + uint8_t id;
> > + uint8_t type;
> > + uint8_t protocol;
> > +} scmi_msg_header_t;
> > +
> > +typedef struct scmi_perms_tx {
> > + uint32_t agent_id;
> > + uint32_t device_id;
> > + uint32_t flags;
> > +} scmi_perms_tx_t;
> > +
> > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> > +
> > +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> > +
> > +struct scmi_shared_mem {
> > + uint32_t reserved;
> > + uint32_t channel_status;
> > + uint32_t reserved1[2];
> > + uint32_t flags;
> > + uint32_t length;
> > + uint32_t msg_header;
> > + uint8_t msg_payload[];
> > +};
> > +
> > +struct scmi_channel {
> > + int chan_id;
> > + int agent_id;
> > + uint32_t func_id;
> > + int domain_id;
> > + uint64_t paddr;
> > + struct scmi_shared_mem *shmem;
> > + spinlock_t lock;
> > + struct list_head list;
> > +};
> > +
> > +struct scmi_data {
> > + struct list_head channel_list;
> > + spinlock_t channel_list_lock;
> > + bool initialized;
> > + u64 shmem_addr, shmem_size;
> > +};
> > +
> > +static struct scmi_data scmi_data;
> > +
> > +/*
> > + * pack_scmi_header() - packs and returns 32-bit header
> > + *
> > + * @hdr: pointer to header containing all the information on message id,
> > + * protocol id and type id.
> > + *
> > + * Return: 32-bit packed message header to be sent to the platform.
> > + */
> > +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> > +{
> > + return FIELD_PREP(HDR_ID, hdr->id) |
> > + FIELD_PREP(HDR_TYPE, hdr->type) |
> > + FIELD_PREP(HDR_PROTO, hdr->protocol);
> > +}
> > +
> > +/*
> > + * unpack_scmi_header() - unpacks and records message and protocol id
> > + *
> > + * @msg_hdr: 32-bit packed message header sent from the platform
> > + * @hdr: pointer to header to fetch message and protocol id.
> > + */
> > +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> > +{
> > + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> > + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> > + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> > +}
> > +
> > +static inline int channel_is_free(struct scmi_channel *chan_info)
> > +{
> > + return ( chan_info->shmem->channel_status
> > + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
> > +}
> > +
> > +static int send_smc_message(struct scmi_channel *chan_info,
> > + scmi_msg_header_t *hdr, void *data, int len)
> > +{
> > + struct arm_smccc_res resp;
> > + int ret;
> > +
> > + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> > + chan_info->shmem->channel_status, len);
> > + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> > + hdr->id, hdr->type, hdr->protocol);
> > +
> > + ret = channel_is_free(chan_info);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + chan_info->shmem->channel_status = 0x0;
> > + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> > + chan_info->shmem->flags = 0x0;
> > + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> > + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> > +
> > + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> > + chan_info->shmem);
> > + if ( len > 0 && data )
> > + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
> > +
> > + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> > + &resp);
> > +
> > + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> > +
> > + if ( resp.a0 )
> > + return -EOPNOTSUPP;
> > +
> > + return 0;
> > +}
> > +
> > +static int check_scmi_status(int scmi_status)
> > +{
> > + if ( scmi_status == SCMI_SUCCESS )
> > + return 0;
> > +
> > + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> > +
> > + switch ( scmi_status )
> > + {
> > + case SCMI_NOT_SUPPORTED:
> > + return -EOPNOTSUPP;
> > + case SCMI_INVALID_PARAMETERS:
> > + return -EINVAL;
> > + case SCMI_DENIED:
> > + return -EACCES;
> > + case SCMI_NOT_FOUND:
> > + return -ENOENT;
> > + case SCMI_OUT_OF_RANGE:
> > + return -ERANGE;
> > + case SCMI_BUSY:
> > + return -EBUSY;
> > + case SCMI_COMMS_ERROR:
> > + return -ENOTCONN;
> > + case SCMI_GENERIC_ERROR:
> > + return -EIO;
> > + case SCMI_HARDWARE_ERROR:
> > + return -ENXIO;
> > + case SCMI_PROTOCOL_ERROR:
> > + return -EBADMSG;
>
> Probably we should add a default: here?
>
Thanks, I will add the default, so fucntion will look like this:
static int check_scmi_status(int scmi_status)
{
...
switch ( scmi_status )
{
...
default:
return -EINVAL;
}
}
>
> > + }
> > +
> > + return -EINVAL;
> > +}
> > +
> > +static int get_smc_response(struct scmi_channel *chan_info,
> > + scmi_msg_header_t *hdr, void *data, int len)
> > +{
> > + int recv_len;
> > + int ret;
> > +
> > + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> > +
> > + ret = channel_is_free(chan_info);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> > +
> > + if ( recv_len < 0 )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Wrong size of smc message. Data may be invalid\n");
> > + return -EINVAL;
> > + }
> > +
> > + if ( recv_len > len )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Not enough buffer for message %d, expecting %d\n",
> > + recv_len, len);
> > + return -EINVAL;
> > + }
> > +
> > + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> > +
> > + if ( recv_len > 0 )
> > + {
> > + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> > + void *rx_data, int rx_size)
> > +{
> > + int ret = 0;
> > +
> > + if ( !hdr )
> > + return -EINVAL;
> > +
> > + spin_lock(&channel->lock);
> > +
> > + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> > +clean:
> > + spin_unlock(&channel->lock);
> > +
> > + return ret;
> > +}
> > +
> > +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > + if ( curr->chan_id == chan_id )
> > + {
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > + if ( curr->domain_id == domain_id )
> > + {
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > + if ( (curr->domain_id == DOMID_INVALID)
> > + && (curr->chan_id != HYP_CHANNEL) )
> > + {
> > + curr->domain_id = domain_id;
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static void relinquish_scmi_channel(struct scmi_channel *channel)
> > +{
> > + spin_lock(&scmi_data.channel_list_lock);
> > + ASSERT(channel != NULL);
> > + channel->domain_id = DOMID_INVALID;
> > + spin_unlock(&scmi_data.channel_list_lock);
> > +}
> > +
> > +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> > + uint32_t func_id, uint64_t addr)
> > +{
> > + struct scmi_channel *channel;
> > + mfn_t mfn;
> > +
> > + channel = get_channel_by_id(chan_id);
> > + if ( channel )
> > + return ERR_PTR(EEXIST);
> > +
> > + channel = xmalloc(struct scmi_channel);
> > + if ( !channel )
> > + return ERR_PTR(ENOMEM);
> > +
> > + channel->chan_id = chan_id;
> > + channel->func_id = func_id;
> > + channel->domain_id = DOMID_INVALID;
> > + mfn = maddr_to_mfn(addr);
> > + channel->shmem = vmap(&mfn, 1);
> > + if ( !channel->shmem )
> > + {
> > + xfree(channel);
> > + return ERR_PTR(ENOMEM);
> > + }
> > +
> > + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> > + channel->paddr = addr;
> > + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> > + spin_lock_init(&channel->lock);
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_add(&channel->list, &scmi_data.channel_list);
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + return channel;
> > +}
> > +
> > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > +{
> > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > +}
> > +
> > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > + uint64_t len)
> > +{
> > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > +}
>
> I wonder, why we need an extra level of indirection here. And if this is
> really needed, I wonder why map(unmap)_memory* name was chosen, as there is
> no memory mapping/unmapping really happens here.
>
I've added extra indirection to hide math like
paddr_to_pfn(PAGE_ALIGN(addr + len -1)
so you don't have to math in each call. unmap_memory_from_domain called
from 2 places, so I moved both calls to separate function.
Although, I agree that map/unmap is not perfect name. I consider
renaming it to mem_permit_acces and mam_deny_access.
>
> > +
> > +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> > + uint64_t size)
>
> Looks like *d is not used in this function.
>
Yep I will remove it. Thanks.
>
> > +{
> > + struct dt_device_node *shmem_node;
> > + __be32 *hw_reg;
> > + const struct dt_property *pp;
> > + uint32_t len;
> > +
> > + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
> > +
> > + if ( !shmem_node )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> > + return -EINVAL;
> > + }
> > +
> > + pp = dt_find_property(shmem_node, "reg", &len);
> > + if ( !pp )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> > + return -ENOENT;
> > + }
> > +
> > + hw_reg = pp->value;
> > + dt_set_range(&hw_reg, shmem_node, addr, size);
> > +
> > + return 0;
> > +}
> > +
> > +static void free_channel_list(void)
> > +{
> > + struct scmi_channel *curr, *_curr;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> > + {
> > + vunmap(curr->shmem);
> > + list_del(&curr->list);
> > + xfree(curr);
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > +}
> > +
> > +static __init bool scmi_probe(struct dt_device_node *scmi_node)
>
>
> Generic question to consider ...
> If probe fails for some reason (so we cannot use mediator in Xen) what
> should we do with SCMI nodes in domain's device-tree (leave as is, drop,
> etc)?
>
If probe fails - then the devices, which were configured to use scmi
clocks\resets\power-domains will not work properly. If Domain
configuration do not use SCMI - then it will not have the scmi node.
So from my standpoint, no additional check is needed before copying scmi
nodes to Domain device-tree.
>
> > +{
> > + struct dt_device_node *shmem_node;
> > + int ret, i;
> > + struct scmi_channel *channel, *agent_channel;
> > + int n_agents;
> > + scmi_msg_header_t hdr;
> > + struct rx_t {
> > + int32_t status;
> > + uint32_t attributes;
> > + } rx;
> > +
> > + uint32_t func_id;
> > +
> > + ASSERT(scmi_node != NULL);
> > +
> > + INIT_LIST_HEAD(&scmi_data.channel_list);
> > + spin_lock_init(&scmi_data.channel_list_lock);
> > +
> > + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> > + return false;
> > + }
> > +
> > + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
> > + if ( IS_ERR_OR_NULL(shmem_node) )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> > + PTR_ERR(shmem_node));
> > + return false;
> > + }
> > +
> > + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> > + &scmi_data.shmem_size);
> > + if ( IS_ERR_VALUE(ret) )
> > + return false;
> > +
> > + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> > + if ( IS_ERR(channel) )
> > + return false;
> > +
> > + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = check_scmi_status(rx.status);
> > + if ( ret )
> > + goto clean;
> > +
> > + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> > + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> > +
> > + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> > + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> > +
> > + for ( i = 1; i < n_agents; i++ )
> > + {
> > + uint32_t tx_agent_id = 0xFFFFFFFF;
> > + struct {
> > + int32_t status;
> > + uint32_t agent_id;
> > + char name[16];
> > + } da_rx;
> > +
> > + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> > + i * PAGE_SIZE);
> > + if ( IS_ERR(agent_channel) )
> > + {
> > + ret = PTR_ERR(agent_channel);
> > + goto clean;
> > + }
> > +
> > + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> > + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = check_scmi_status(da_rx.status);
> > + if ( ret )
> > + goto clean;
> > +
> > + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> > + da_rx.status, da_rx.agent_id, da_rx.name);
> > +
> > + agent_channel->agent_id = da_rx.agent_id;
> > + }
> > +
> > + scmi_data.initialized = true;
> > + return true;
> > +
> > +clean:
> > + free_channel_list();
> > + return ret == 0;
> > +}
> > +
> > +static int scmi_domain_init(struct domain *d)
> > +{
> > + struct scmi_channel *channel;
> > + int ret;
> > +
> > + if ( !scmi_data.initialized )
> > + return 0;
> > +
> > + channel = aquire_scmi_channel(d->domain_id);
> > + if ( IS_ERR_OR_NULL(channel) )
> > + return -ENOENT;
> > +
> > + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> > + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> > + channel->paddr);
>
>
> It seems this breaks build on Arm32:
>
> scmi_smc.c: In function ‘scmi_domain_init’:
> /home/otyshchenko/xen/xen/include/xen/config.h:53:24: error: format ‘%lx’
> expects argument of type ‘long unsigned int’, but argument 4 has type
> ‘uint64_t {aka long long unsigned int}’ [-Werror=format=]
> #define XENLOG_INFO "<2>"
> ^
> scmi_smc.c:569:12: note: in expansion of macro ‘XENLOG_INFO’
> printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id =
> %d"
> ^~~~~~~~~~~
> scmi_smc.c:570:25: note: format string is defined here
> "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> ~~^
> %llx
>
Thanks a lot. I will work through all buids in v2.
>
>
> > +
> > + if ( is_hardware_domain(d) )
> > + {
> > + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> > + scmi_data.shmem_size);
> > + if ( IS_ERR_VALUE(ret) )
> > + goto error;
> > +
> > + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> > + if ( IS_ERR_VALUE(ret) )
> > + {
> > + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> > + scmi_data.shmem_size);
> > + if ( rc )
> > + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> > +
> > + goto error;
> > + }
> > + }
> > +
> > + d->arch.sci = channel;
> > +
> > + return 0;
> > +error:
> > + relinquish_scmi_channel(channel);
> > +
> > + return ret;
> > +}
> > +
> > +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> > +{
> > + struct scmi_channel *channel, *agent_channel;
> > + scmi_msg_header_t hdr;
> > + scmi_perms_tx_t tx;
> > + struct rx_t {
> > + int32_t status;
> > + uint32_t attributes;
> > + } rx;
> > + int ret;
> > +
> > + if ( !scmi_data.initialized )
> > + return 0;
> > +
> > + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> > +
> > + agent_channel = get_channel_by_domain(d->domain_id);
> > + if ( IS_ERR_OR_NULL(agent_channel) )
> > + return PTR_ERR(agent_channel);
> > +
> > + channel = get_channel_by_id(HYP_CHANNEL);
> > + if ( IS_ERR_OR_NULL(channel) )
> > + return PTR_ERR(channel);
> > +
> > + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + tx.agent_id = agent_channel->agent_id;
> > + tx.device_id = scmi_devid;
> > + tx.flags = SCMI_ALLOW_ACCESS;
> > +
> > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + ret = check_scmi_status(rx.status);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + return 0;
> > +}
> > +
> > +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> > +{
> > + uint32_t scmi_devid;
> > +
> > + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> > + return 0;
> > +
> > + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> > + return 0;
> > +
> > + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> > +
> > + return scmi_add_device_by_devid(d, scmi_devid);
> > +}
> > +
> > +static int scmi_relinquish_resources(struct domain *d)
> > +{
> > + int ret;
> > + struct scmi_channel *channel, *agent_channel;
> > + scmi_msg_header_t hdr;
> > + struct reset_agent_tx {
> > + uint32_t agent_id;
> > + uint32_t flags;
> > + } tx;
> > + uint32_t rx;
> > +
> > + if ( !d->arch.sci )
> > + return 0;
> > +
> > + agent_channel = d->arch.sci;
> > +
> > + spin_lock(&agent_channel->lock);
> > + tx.agent_id = agent_channel->agent_id;
> > + spin_unlock(&agent_channel->lock);
> > +
> > + channel = get_channel_by_id(HYP_CHANNEL);
> > + if ( !channel )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> > + d->domain_id);
> > + return -EINVAL;
> > + }
> > +
> > + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + tx.flags = 0;
> > +
> > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> > + if ( ret )
> > + return ret;
> > +
> > + ret = check_scmi_status(rx);
> > +
> > + return ret;
> > +}
> > +
> > +static void scmi_domain_destroy(struct domain *d)
> > +{
> > + struct scmi_channel *channel;
> > +
> > + if ( !d->arch.sci )
> > + return;
> > +
> > + channel = d->arch.sci;
> > + spin_lock(&channel->lock);
> > +
> > + relinquish_scmi_channel(channel);
> > + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> > +
> > + d->arch.sci = NULL;
> > +
> > + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
> I didn't manage to find where corresponding map_memory_from_domain() is
> called for a non-hardware domain (it seems that scmi_domain_init() only
> directly handles hardware domain case).
> Or perhaps, it is called indirectly at do_domctl(): case
> XEN_DOMCTL_iomem_permission:?
>
You were right. We need to permit access only for the Hardware domain.
>
> I wonder, do we really need to call this here? Taking into the account that
> unmap_memory_from_domain() doesn't actually unmap anything, but only removes
> a range from the iomem_caps rangeset
> for the domain to be destroyed and all involved rangesets (including
> iomem_caps) will be removed soon at rangeset_domain_destroy() anyway. Or I
> missed something?
>
unmap_memory_from_domain is also used in scmi_domain_init if error
occured after successfull map_memory_to_domain call.
But this is a good point. I think it will be a good idea to make
unmap_memory_from_domain in scmi_domain_destroy only for hardware
domain. And make sure that for Software domains range is unmapped using
XEN_DOMCTL_iomem_permission call.
I'll make those changes in v2.
>
> > + spin_unlock(&channel->lock);
> > + return;
>
> empty return could be dropped, I think.
>
Agree. Will be fixed.
>
> > +}
> > +
> > +static bool scmi_handle_call(struct domain *d, void *args)
> > +{
> > + bool res = false;
> > + struct scmi_channel *agent_channel;
> > + struct arm_smccc_res resp;
> > + struct cpu_user_regs *regs = args;
> > +
> > + if ( !d->arch.sci )
> > + return false;
> > +
> > + agent_channel = d->arch.sci;
> > + spin_lock(&agent_channel->lock);
> > +
> > + if ( agent_channel->func_id != regs->x0 )
>
> This also breaks build on Arm32:
>
> scmi_smc.c: In function ‘scmi_handle_call’:
> scmi_smc.c:736:42: error: ‘struct cpu_user_regs’ has no member named ‘x0’;
> did you mean ‘r0’?
> if ( agent_channel->func_id != regs->x0 )
> ^~
> r0
> cc1: all warnings being treated as errors
>
Thanks, I will fix it in v2.
> ***
>
> BTW, I noticed that xen/arch/arm/traps.c contains the following construct,
> probably we might want something similar here?
>
> #ifdef CONFIG_ARM_64
> #define HYPERCALL_RESULT_REG(r) (r)->x0
> [snip]
> #else
> #define HYPERCALL_RESULT_REG(r) (r)->r0
> [snip]
> #endif
>
>
> This RFC patch series, so I think, there is no serious issues at the moment,
> this is rather to let you know for the future (when you drop RFC tag).
> I have to admit, I often forget to build-test on Arm32 also))
>
Thank you for understanding. But will try to cover all architectures.
>
> > + {
> > + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> > + goto unlock;
> > + }
> > +
> > + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> > + agent_channel->chan_id, &resp);
> > +
> > + set_user_reg(regs, 0, resp.a0);
> > + set_user_reg(regs, 1, resp.a1);
> > + set_user_reg(regs, 2, resp.a2);
> > + set_user_reg(regs, 3, resp.a3);
> > + res = true;
> > +unlock:
> > + spin_unlock(&agent_channel->lock);
> > +
> > + return res;
> > +}
> > +
> > +static int scmi_get_channel_paddr(void *scmi_ops,
> > + struct xen_arch_domainconfig *config)
> > +{
> > + struct scmi_channel *agent_channel = scmi_ops;
> > +
> > + if ( !agent_channel )
> > + return -EINVAL;
> > +
> > + config->sci_agent_paddr = agent_channel->paddr;
> > + return 0;
> > +}
> > +
> > +static const struct dt_device_match scmi_smc_match[] __initconst =
> > +{
> > + DT_MATCH_SCMI_SMC,
> > + { /* sentinel */ },
> > +};
> > +
> > +static const struct sci_mediator_ops scmi_ops =
> > +{
> > + .probe = scmi_probe,
> > + .domain_init = scmi_domain_init,
> > + .domain_destroy = scmi_domain_destroy,
> > + .add_dt_device = scmi_add_dt_device,
> > + .relinquish_resources = scmi_relinquish_resources,
> > + .handle_call = scmi_handle_call,
> > + .get_channel_info = scmi_get_channel_paddr
> > +};
> > +
> > +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> > + scmi_smc_match, &scmi_ops);
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > index 9180be5e86..a67237942d 100644
> > --- a/xen/include/public/arch-arm.h
> > +++ b/xen/include/public/arch-arm.h
> > @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> > #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> > #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> > +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
> > struct xen_arch_domainconfig {
> > /* IN/OUT */
>
> --
> Regards,
>
> Oleksandr Tyshchenko
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-17 13:23 ` Oleksii Moisieiev
@ 2021-12-17 13:37 ` Julien Grall
2021-12-17 13:58 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-17 13:37 UTC (permalink / raw)
To: Oleksii Moisieiev, Oleksandr
Cc: xen-devel, Stefano Stabellini, Volodymyr Babchuk, Bertrand Marquis
Hi,
On 17/12/2021 13:23, Oleksii Moisieiev wrote:
>>> +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
>>> +{
>>> + return iomem_permit_access(d, paddr_to_pfn(addr),
>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>> +}
>>> +
>>> +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
>>> + uint64_t len)
>>> +{
>>> + return iomem_deny_access(d, paddr_to_pfn(addr),
>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>> +}
>>
>> I wonder, why we need an extra level of indirection here. And if this is
>> really needed, I wonder why map(unmap)_memory* name was chosen, as there is
>> no memory mapping/unmapping really happens here.
>>
>
> I've added extra indirection to hide math like
> paddr_to_pfn(PAGE_ALIGN(addr + len -1)
> so you don't have to math in each call. unmap_memory_from_domain called
> from 2 places, so I moved both calls to separate function.
> Although, I agree that map/unmap is not perfect name. I consider
> renaming it to mem_permit_acces and mam_deny_access.
I haven't looked at the rest of the series. But this discussion caught
my eye. This code implies that the address is page-aligned but the
length not. Is that intended?
That said, if you give permission to the domain on a full page then it
means it may be able to access address it should not. Can you explain
why this is fine?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-17 7:16 ` Jan Beulich
@ 2021-12-17 13:40 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 13:40 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis, xen-devel
Hi Jan,
On Fri, Dec 17, 2021 at 08:16:05AM +0100, Jan Beulich wrote:
> On 17.12.2021 08:12, Jan Beulich wrote:
> > On 16.12.2021 18:36, Oleksii Moisieiev wrote:
> >> On Tue, Dec 14, 2021 at 10:41:30AM +0100, Jan Beulich wrote:
> >>> On 14.12.2021 10:34, Oleksii Moisieiev wrote:
> >>>> --- a/xen/include/public/domctl.h
> >>>> +++ b/xen/include/public/domctl.h
> >>>> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> >>>> #define XEN_DOMCTL_vmtrace_get_option 5
> >>>> #define XEN_DOMCTL_vmtrace_set_option 6
> >>>> };
> >>>> +
> >>>> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> >>>> +struct xen_domctl_sci_device_op {
> >>>> + uint32_t size; /* Length of the path */
> >>>> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> >>>> +};
> >>>
> >>> This being - aiui - Arm-only, please enclose it by respective #if,
> >>> just like we do for certain x86-only ops.
> >>>
> >>
> >> I agree. I will add #ifdefs in v2.
> >>
> >>> I'm further afraid the term "SCI" is ambiguous with ACPI's System
> >>> Control Interrupt, so there's some further tag needed in the names
> >>> used here.
> >>>
> >>
> >> Thank you for remark. I'm thinking about SC as System Control.
> >> What do you think?
> >
> > I guess "SC" could even more so stand for various things. Even the
> > spelled out "System Control" looks overly generic. If this isn't
> > Arm-specific (in which case adding "arm" into the name might at least
> > help the situation a little), then I guess some further disambiguation
> > is going to be wanted. Since I don't know any of the context of this,
> > I'm afraid you're in a far better position than me to come up with a
> > non-ambiguous name.
>
> Actually, looking at the title again - any reason not to add "mediator"
> into the name? While I have no idea whether there could be other
> mediators with an ambiguous acronym, this would at least address the
> ACPI related concern (I don't expect anything mediator-like to appear
> there, but then again I might be wrong).
>
> Jan
>
I wanted the name to be abbriveation. Also tee option in xen doesn't
have "mediator" in it's name either. As for the SC - the only 2 uses I
found are - Spreadsheet Calculator and some script name for
traffic-shaper.
But I think name still need to be discussed.
--
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-17 13:37 ` Julien Grall
@ 2021-12-17 13:58 ` Oleksii Moisieiev
2021-12-17 16:38 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-17 13:58 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi Julien,
On Fri, Dec 17, 2021 at 01:37:35PM +0000, Julien Grall wrote:
> Hi,
>
> On 17/12/2021 13:23, Oleksii Moisieiev wrote:
> > > > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > > > +{
> > > > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > +}
> > > > +
> > > > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > > > + uint64_t len)
> > > > +{
> > > > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > +}
> > >
> > > I wonder, why we need an extra level of indirection here. And if this is
> > > really needed, I wonder why map(unmap)_memory* name was chosen, as there is
> > > no memory mapping/unmapping really happens here.
> > >
> >
> > I've added extra indirection to hide math like
> > paddr_to_pfn(PAGE_ALIGN(addr + len -1)
> > so you don't have to math in each call. unmap_memory_from_domain called
> > from 2 places, so I moved both calls to separate function.
> > Although, I agree that map/unmap is not perfect name. I consider
> > renaming it to mem_permit_acces and mam_deny_access.
>
> I haven't looked at the rest of the series. But this discussion caught my
> eye. This code implies that the address is page-aligned but the length not.
> Is that intended?
>
> That said, if you give permission to the domain on a full page then it means
> it may be able to access address it should not. Can you explain why this is
> fine?
>
The idea was that xen receives some memory from the dt_node linux,scmi_mem,
then we split memory between the agents, so each agent get 1 page (we
allocate 0x10 pages right now).
But this is a good point, I think we should check length to be aligned
before making any permission changes.
I will add it in v2.
Thank you for remark.
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-17 13:58 ` Oleksii Moisieiev
@ 2021-12-17 16:38 ` Julien Grall
2021-12-20 15:41 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-17 16:38 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
On 17/12/2021 13:58, Oleksii Moisieiev wrote:
> Hi Julien,
Hi,
> On Fri, Dec 17, 2021 at 01:37:35PM +0000, Julien Grall wrote:
>> Hi,
>>
>> On 17/12/2021 13:23, Oleksii Moisieiev wrote:
>>>>> +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
>>>>> +{
>>>>> + return iomem_permit_access(d, paddr_to_pfn(addr),
>>>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>>>> +}
>>>>> +
>>>>> +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
>>>>> + uint64_t len)
>>>>> +{
>>>>> + return iomem_deny_access(d, paddr_to_pfn(addr),
>>>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>>>> +}
>>>>
>>>> I wonder, why we need an extra level of indirection here. And if this is
>>>> really needed, I wonder why map(unmap)_memory* name was chosen, as there is
>>>> no memory mapping/unmapping really happens here.
>>>>
>>>
>>> I've added extra indirection to hide math like
>>> paddr_to_pfn(PAGE_ALIGN(addr + len -1)
>>> so you don't have to math in each call. unmap_memory_from_domain called
>>> from 2 places, so I moved both calls to separate function.
>>> Although, I agree that map/unmap is not perfect name. I consider
>>> renaming it to mem_permit_acces and mam_deny_access.
>>
>> I haven't looked at the rest of the series. But this discussion caught my
>> eye. This code implies that the address is page-aligned but the length not.
>> Is that intended?
>>
>> That said, if you give permission to the domain on a full page then it means
>> it may be able to access address it should not. Can you explain why this is
>> fine?
>>
>
> The idea was that xen receives some memory from the dt_node linux,scmi_mem,
> then we split memory between the agents, so each agent get 1 page (we
> allocate 0x10 pages right now).
Thanks for the clarification. Does this imply the guest will be able to
write message directly to the firmware?
If so, this brings a few more questions:
1) What will the guest write in it? Can it contains addresses?
2) What are the expected memory attribute for the regions?
3) What's the threat model for the firmware? Will it verify every
request?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-14 9:34 ` [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver Oleksii Moisieiev
2021-12-17 11:35 ` Oleksandr
@ 2021-12-18 2:14 ` Stefano Stabellini
2021-12-20 18:12 ` Oleksii Moisieiev
1 sibling, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-18 2:14 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> This is the implementation of SCI interface, called SCMI-SMC driver,
> which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> This allows devices from the Domains to work with clocks, resets and
> power-domains without access to CPG.
>
> The following features are implemented:
> - request SCMI channels from ATF and pass channels to Domains;
> - set device permissions for Domains based on the Domain partial
> device-tree. Devices with permissions are able to work with clocks,
> resets and power-domains via SCMI;
> - redirect scmi messages from Domains to ATF.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> xen/arch/arm/Kconfig | 2 +
> xen/arch/arm/sci/Kconfig | 10 +
> xen/arch/arm/sci/Makefile | 1 +
> xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> xen/include/public/arch-arm.h | 1 +
> 5 files changed, 809 insertions(+)
> create mode 100644 xen/arch/arm/sci/Kconfig
> create mode 100644 xen/arch/arm/sci/scmi_smc.c
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 186e1db389..02d96c6cfc 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -114,6 +114,8 @@ config SCI
> support. It allows guests to control system resourcess via one of
> SCI mediators implemented in XEN.
>
> +source "arch/arm/sci/Kconfig"
> +
> endmenu
>
> menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> new file mode 100644
> index 0000000000..9563067ddc
> --- /dev/null
> +++ b/xen/arch/arm/sci/Kconfig
> @@ -0,0 +1,10 @@
> +config SCMI_SMC
> + bool "Enable SCMI-SMC mediator driver"
> + default n
> + depends on SCI
> + ---help---
> +
> + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> + This feature allows drivers from Domains to work with System
> + Controllers (such as power,resets,clock etc.). SCP is used as transport
> + for communication.
> diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> index 837dc7492b..67f2611872 100644
> --- a/xen/arch/arm/sci/Makefile
> +++ b/xen/arch/arm/sci/Makefile
> @@ -1 +1,2 @@
> obj-y += sci.o
> +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> new file mode 100644
> index 0000000000..2eb01ea82d
> --- /dev/null
> +++ b/xen/arch/arm/sci/scmi_smc.c
> @@ -0,0 +1,795 @@
> +/*
> + * xen/arch/arm/sci/scmi_smc.c
> + *
> + * SCMI mediator driver, using SCP as transport.
> + *
> + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> + * Copyright (C) 2021, EPAM Systems.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <asm/sci/sci.h>
> +#include <asm/smccc.h>
> +#include <asm/io.h>
> +#include <xen/bitops.h>
> +#include <xen/config.h>
> +#include <xen/sched.h>
> +#include <xen/device_tree.h>
> +#include <xen/iocap.h>
> +#include <xen/init.h>
> +#include <xen/err.h>
> +#include <xen/lib.h>
> +#include <xen/list.h>
> +#include <xen/mm.h>
> +#include <xen/string.h>
> +#include <xen/time.h>
> +#include <xen/vmap.h>
> +
> +#define SCMI_BASE_PROTOCOL 0x10
> +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> +#define SCMI_BASE_DISCOVER_AGENT 0x7
> +
> +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> +#define SCMI_SUCCESS 0
> +#define SCMI_NOT_SUPPORTED (-1)
> +#define SCMI_INVALID_PARAMETERS (-2)
> +#define SCMI_DENIED (-3)
> +#define SCMI_NOT_FOUND (-4)
> +#define SCMI_OUT_OF_RANGE (-5)
> +#define SCMI_BUSY (-6)
> +#define SCMI_COMMS_ERROR (-7)
> +#define SCMI_GENERIC_ERROR (-8)
> +#define SCMI_HARDWARE_ERROR (-9)
> +#define SCMI_PROTOCOL_ERROR (-10)
> +
> +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> +
> +#define SCMI_SMC_ID "arm,smc-id"
> +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
I could find the following SCMI binding in Linux, which describes
the arm,scmi-smc compatible and the arm,smc-id property:
Documentation/devicetree/bindings/firmware/arm,scmi.yaml
However, linux,scmi_mem is not described. Aren't you supposed to read
the "shmem" property instead? And the compatible string used for this
seems to be "arm,scmi-shmem".
> +#define SCMI_SHMEM "shmem"
> +
> +#define HYP_CHANNEL 0x0
> +
> +#define HDR_ID GENMASK(7,0)
> +#define HDR_TYPE GENMASK(9, 8)
> +#define HDR_PROTO GENMASK(17, 10)
> +
> +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> +
> +#define FIELD_GET(_mask, _reg)\
> + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> +#define FIELD_PREP(_mask, _val)\
> + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> +
> +typedef struct scmi_msg_header {
> + uint8_t id;
> + uint8_t type;
> + uint8_t protocol;
> +} scmi_msg_header_t;
> +
> +typedef struct scmi_perms_tx {
> + uint32_t agent_id;
> + uint32_t device_id;
> + uint32_t flags;
> +} scmi_perms_tx_t;
> +
> +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> +
> +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> +
> +struct scmi_shared_mem {
> + uint32_t reserved;
> + uint32_t channel_status;
> + uint32_t reserved1[2];
> + uint32_t flags;
> + uint32_t length;
> + uint32_t msg_header;
> + uint8_t msg_payload[];
> +};
> +
> +struct scmi_channel {
> + int chan_id;
> + int agent_id;
> + uint32_t func_id;
> + int domain_id;
> + uint64_t paddr;
> + struct scmi_shared_mem *shmem;
> + spinlock_t lock;
> + struct list_head list;
> +};
> +
> +struct scmi_data {
> + struct list_head channel_list;
> + spinlock_t channel_list_lock;
> + bool initialized;
> + u64 shmem_addr, shmem_size;
> +};
> +
> +static struct scmi_data scmi_data;
> +
> +/*
> + * pack_scmi_header() - packs and returns 32-bit header
> + *
> + * @hdr: pointer to header containing all the information on message id,
> + * protocol id and type id.
> + *
> + * Return: 32-bit packed message header to be sent to the platform.
> + */
> +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> +{
> + return FIELD_PREP(HDR_ID, hdr->id) |
> + FIELD_PREP(HDR_TYPE, hdr->type) |
> + FIELD_PREP(HDR_PROTO, hdr->protocol);
> +}
> +
> +/*
> + * unpack_scmi_header() - unpacks and records message and protocol id
> + *
> + * @msg_hdr: 32-bit packed message header sent from the platform
> + * @hdr: pointer to header to fetch message and protocol id.
> + */
> +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> +{
> + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> +}
> +
> +static inline int channel_is_free(struct scmi_channel *chan_info)
> +{
> + return ( chan_info->shmem->channel_status
> + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
Does this need a memory barrier? Or not, because the other end always
runs on the same CPU at a different execution level so the
channel_status would be always guaranteed to be read as updated?
> +}
> +
> +static int send_smc_message(struct scmi_channel *chan_info,
> + scmi_msg_header_t *hdr, void *data, int len)
> +{
> + struct arm_smccc_res resp;
> + int ret;
> +
> + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> + chan_info->shmem->channel_status, len);
> + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> + hdr->id, hdr->type, hdr->protocol);
> +
> + ret = channel_is_free(chan_info);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + chan_info->shmem->channel_status = 0x0;
> + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> + chan_info->shmem->flags = 0x0;
> + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> +
> + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> + chan_info->shmem);
> + if ( len > 0 && data )
> + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
Again, here we don't need a barrier because it is implicit in the SMC?
Don't we need to check that "len" fits in the shared memory?
> + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> + &resp);
> +
> + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> +
> + if ( resp.a0 )
> + return -EOPNOTSUPP;
Why is that?
> + return 0;
> +}
> +
> +static int check_scmi_status(int scmi_status)
> +{
> + if ( scmi_status == SCMI_SUCCESS )
> + return 0;
> +
> + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> +
> + switch ( scmi_status )
> + {
> + case SCMI_NOT_SUPPORTED:
> + return -EOPNOTSUPP;
> + case SCMI_INVALID_PARAMETERS:
> + return -EINVAL;
> + case SCMI_DENIED:
> + return -EACCES;
> + case SCMI_NOT_FOUND:
> + return -ENOENT;
> + case SCMI_OUT_OF_RANGE:
> + return -ERANGE;
> + case SCMI_BUSY:
> + return -EBUSY;
> + case SCMI_COMMS_ERROR:
> + return -ENOTCONN;
> + case SCMI_GENERIC_ERROR:
> + return -EIO;
> + case SCMI_HARDWARE_ERROR:
> + return -ENXIO;
> + case SCMI_PROTOCOL_ERROR:
> + return -EBADMSG;
> + }
> +
> + return -EINVAL;
> +}
> +
> +static int get_smc_response(struct scmi_channel *chan_info,
> + scmi_msg_header_t *hdr, void *data, int len)
> +{
> + int recv_len;
> + int ret;
> +
> + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> +
> + ret = channel_is_free(chan_info);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
I am not familiar with the spec (do you have a link?) but is it expected
that the channel is "free" when actually we want to read a message on
the channel?
> + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> +
> + if ( recv_len < 0 )
> + {
> + printk(XENLOG_ERR
> + "scmi: Wrong size of smc message. Data may be invalid\n");
> + return -EINVAL;
> + }
> +
> + if ( recv_len > len )
> + {
> + printk(XENLOG_ERR
> + "scmi: Not enough buffer for message %d, expecting %d\n",
> + recv_len, len);
> + return -EINVAL;
> + }
> +
> + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> +
> + if ( recv_len > 0 )
> + {
> + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> + }
> +
> + return 0;
> +}
> +
> +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> + void *rx_data, int rx_size)
> +{
> + int ret = 0;
> +
> + if ( !hdr )
> + return -EINVAL;
> +
> + spin_lock(&channel->lock);
> +
> + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> + if ( ret )
> + goto clean;
> +
> + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> +clean:
> + spin_unlock(&channel->lock);
> +
> + return ret;
> +}
> +
> +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
please use parenthesis around the inner if (also in other places)
> + if ( curr->chan_id == chan_id )
> + {
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
Use domid_t for domain ids.
Also, wouldn't it be better to implement it as:
static inline struct scmi_channel *get_channel_by_domain(struct domain *d) {
return d->arch.sci
}
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
> + if ( curr->domain_id == domain_id )
> + {
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> +{
> + struct scmi_channel *curr;
> + bool found = false;
> +
> + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry(curr, &scmi_data.channel_list, list)
> + if ( (curr->domain_id == DOMID_INVALID)
> + && (curr->chan_id != HYP_CHANNEL) )
If you use DOMID_XEN for HYP_CHANNEL, then this check becomes more
intuitive
> + {
> + curr->domain_id = domain_id;
> + found = true;
> + break;
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> + if ( found )
> + return curr;
> +
> + return NULL;
> +}
> +
> +static void relinquish_scmi_channel(struct scmi_channel *channel)
> +{
> + spin_lock(&scmi_data.channel_list_lock);
> + ASSERT(channel != NULL);
the ASSERT could be before the spin_lock
> + channel->domain_id = DOMID_INVALID;
> + spin_unlock(&scmi_data.channel_list_lock);
> +}
> +
> +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> + uint32_t func_id, uint64_t addr)
> +{
> + struct scmi_channel *channel;
> + mfn_t mfn;
> +
> + channel = get_channel_by_id(chan_id);
> + if ( channel )
> + return ERR_PTR(EEXIST);
> +
> + channel = xmalloc(struct scmi_channel);
> + if ( !channel )
> + return ERR_PTR(ENOMEM);
> +
> + channel->chan_id = chan_id;
> + channel->func_id = func_id;
> + channel->domain_id = DOMID_INVALID;
I take you are using DOMID_INVALID to mark a channel used by Xen itself?
If so, then DOMID_XEN would be more appropriate.
> + mfn = maddr_to_mfn(addr);
> + channel->shmem = vmap(&mfn, 1);
One thing to be careful is the mapping attributes, for a couple of
reasons. As you might be aware, the ARM architecture forbids mismatching
attributes for mapping memory in different places in the system. So the
attributes that we use here must be the same used by the firmware
(and/or the guest.)
The second reason to be careful is that in the bindings example
Documentation/devicetree/bindings/firmware/arm,scmi.yaml the shared
memory is "mmio-sram", which is special. It is not supposed to be normal
memory, but it is OK to map it cacheable. Still, it might be more
appropriate to use ioremap_cache.
> + if ( !channel->shmem )
> + {
> + xfree(channel);
> + return ERR_PTR(ENOMEM);
> + }
> +
> + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> + channel->paddr = addr;
> + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> + spin_lock_init(&channel->lock);
> + spin_lock(&scmi_data.channel_list_lock);
> + list_add(&channel->list, &scmi_data.channel_list);
> + spin_unlock(&scmi_data.channel_list_lock);
> + return channel;
> +}
> +
> +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> +{
> + return iomem_permit_access(d, paddr_to_pfn(addr),
> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> +}
> +
> +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> + uint64_t len)
> +{
> + return iomem_deny_access(d, paddr_to_pfn(addr),
> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> +}
> +
> +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> + uint64_t size)
> +{
> + struct dt_device_node *shmem_node;
> + __be32 *hw_reg;
> + const struct dt_property *pp;
> + uint32_t len;
> +
> + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
Here we are using "arm,scmi-shmem" while below we are checking for
"linux,scmi_mem". What's the difference?
Also, this function is looking for "arm,scmi-shmem" in dt_host and
replaces its value. For dom0less domUs we'll probably need a
make_scmi_node function to create the node from scratch like for
instance xen/arch/arm/domain_build.c:make_gic_domU_node.
I wonder if we had such a function whether it wouldn't be better to also
use it for dom0 (and blacklist the physical "arm,scmi-shmem" in
handle_node so that dom0 doesn't get the real shared memory information
by accident).
> +
> + if ( !shmem_node )
> + {
> + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> + return -EINVAL;
> + }
> +
> + pp = dt_find_property(shmem_node, "reg", &len);
> + if ( !pp )
> + {
> + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> + return -ENOENT;
> + }
> +
> + hw_reg = pp->value;
> + dt_set_range(&hw_reg, shmem_node, addr, size);
> +
> + return 0;
> +}
> +
> +static void free_channel_list(void)
> +{
> + struct scmi_channel *curr, *_curr;
> +
> + spin_lock(&scmi_data.channel_list_lock);
> + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> + {
> + vunmap(curr->shmem);
> + list_del(&curr->list);
> + xfree(curr);
> + }
> +
> + spin_unlock(&scmi_data.channel_list_lock);
> +}
> +
> +static __init bool scmi_probe(struct dt_device_node *scmi_node)
> +{
> + struct dt_device_node *shmem_node;
> + int ret, i;
> + struct scmi_channel *channel, *agent_channel;
> + int n_agents;
> + scmi_msg_header_t hdr;
> + struct rx_t {
> + int32_t status;
> + uint32_t attributes;
> + } rx;
Should rx be defined at the top together with scmi_perms_tx_t and
others?
> + uint32_t func_id;
> +
> + ASSERT(scmi_node != NULL);
> +
> + INIT_LIST_HEAD(&scmi_data.channel_list);
> + spin_lock_init(&scmi_data.channel_list_lock);
> +
> + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> + {
> + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> + return false;
> + }
> +
> + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
From the spec, it looks like you should be getting the shared memory
area from the phandle list "shmem".
> + if ( IS_ERR_OR_NULL(shmem_node) )
> + {
> + printk(XENLOG_ERR
> + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> + PTR_ERR(shmem_node));
> + return false;
> + }
> +
> + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> + &scmi_data.shmem_size);
> + if ( IS_ERR_VALUE(ret) )
> + return false;
> +
> + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> + if ( IS_ERR(channel) )
> + return false;
> +
> + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> + if ( ret )
> + goto clean;
> +
> + ret = check_scmi_status(rx.status);
> + if ( ret )
> + goto clean;
> +
> + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> +
> + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> +
> + for ( i = 1; i < n_agents; i++ )
> + {
Given that HYP_CHANNEL is actually zero, it looks like we could do
everything here in this loop but starting from i=0?
> + uint32_t tx_agent_id = 0xFFFFFFFF;
> + struct {
> + int32_t status;
> + uint32_t agent_id;
> + char name[16];
> + } da_rx;
> +
> + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> + i * PAGE_SIZE);
> + if ( IS_ERR(agent_channel) )
> + {
> + ret = PTR_ERR(agent_channel);
> + goto clean;
> + }
> +
> + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> + if ( ret )
> + goto clean;
> +
> + ret = check_scmi_status(da_rx.status);
> + if ( ret )
> + goto clean;
> +
> + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> + da_rx.status, da_rx.agent_id, da_rx.name);
> +
> + agent_channel->agent_id = da_rx.agent_id;
> + }
> +
> + scmi_data.initialized = true;
> + return true;
> +
> +clean:
> + free_channel_list();
> + return ret == 0;
> +}
> +
> +static int scmi_domain_init(struct domain *d)
> +{
> + struct scmi_channel *channel;
> + int ret;
> +
> + if ( !scmi_data.initialized )
> + return 0;
> +
> + channel = aquire_scmi_channel(d->domain_id);
> + if ( IS_ERR_OR_NULL(channel) )
> + return -ENOENT;
> +
> + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> + channel->paddr);
> +
> + if ( is_hardware_domain(d) )
> + {
> + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> + scmi_data.shmem_size);
> + if ( IS_ERR_VALUE(ret) )
> + goto error;
> +
> + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> + if ( IS_ERR_VALUE(ret) )
> + {
> + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> + scmi_data.shmem_size);
> + if ( rc )
> + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> +
> + goto error;
> + }
> + }
Is dom0 the only domain to get direct access to the shared memory
region? If so, I don't think it is a good idea to make Dom0 "special" in
this case.
Let me make an example: if we assign a device to a domU since boot, and
dom0 wants to change the frequency of a clock that affects the assigned
device (likely because it doesn't know it is assigned), then dom0
shouldn't be able to. We might have to perform checks in Xen to make
sure dom0 cannot stop the clock for the assigned device.
So I think it would be better if all domains are treated the same way in
the mediator unless really necessary.
On the other hand, if all domains get access to the shared memory
region, then I don't think this is likely the right place to create the
dom0 mapping. We probably want to do it in domain_build.c in a way that
can be reused for dom0less domUs.
In regards to shared memory: it looks like the only two functions to
access the real shared memory are send_smc_message and get_smc_response.
If that is the case, then we actually don't need to expose the real
shared memory to any of the domains.
We could simply:
- expose a regular normal memory region as dom0/domU channel memory
- on SMC trap, read from the "fake" shared memory and set the
corresponding real shared memory on the appropriate channel
- issue the SMC call
- on return from SMC, copy over data from the real shared memory to the
"fake" channel reagion
This is useful if we need to "filter" any of the SCMI commands and
options from the domains to the firmware, and also it is useful if the
channel memory is not page aligned. But if the permissions are
fine-grained enough and also the channel memory is page aligned (and
multiple of 4K in size) then we could map the memory.
> +
> + d->arch.sci = channel;
> +
> + return 0;
> +error:
> + relinquish_scmi_channel(channel);
> +
> + return ret;
> +}
> +
> +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> +{
> + struct scmi_channel *channel, *agent_channel;
> + scmi_msg_header_t hdr;
> + scmi_perms_tx_t tx;
> + struct rx_t {
> + int32_t status;
> + uint32_t attributes;
> + } rx;
> + int ret;
> +
> + if ( !scmi_data.initialized )
> + return 0;
> +
> + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> +
> + agent_channel = get_channel_by_domain(d->domain_id);
> + if ( IS_ERR_OR_NULL(agent_channel) )
> + return PTR_ERR(agent_channel);
> +
> + channel = get_channel_by_id(HYP_CHANNEL);
> + if ( IS_ERR_OR_NULL(channel) )
> + return PTR_ERR(channel);
> +
> + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + tx.agent_id = agent_channel->agent_id;
> + tx.device_id = scmi_devid;
> + tx.flags = SCMI_ALLOW_ACCESS;
> +
> + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + ret = check_scmi_status(rx.status);
> + if ( IS_ERR_VALUE(ret) )
> + return ret;
> +
> + return 0;
> +}
> +
> +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> +{
> + uint32_t scmi_devid;
> +
> + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> + return 0;
> +
> + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> + return 0;
scmi_devid is another property that is not documented in the binding.
> + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> +
> + return scmi_add_device_by_devid(d, scmi_devid);
> +}
> +
> +static int scmi_relinquish_resources(struct domain *d)
> +{
> + int ret;
> + struct scmi_channel *channel, *agent_channel;
> + scmi_msg_header_t hdr;
> + struct reset_agent_tx {
> + uint32_t agent_id;
> + uint32_t flags;
> + } tx;
> + uint32_t rx;
> +
> + if ( !d->arch.sci )
> + return 0;
> +
> + agent_channel = d->arch.sci;
> +
> + spin_lock(&agent_channel->lock);
> + tx.agent_id = agent_channel->agent_id;
> + spin_unlock(&agent_channel->lock);
> +
> + channel = get_channel_by_id(HYP_CHANNEL);
> + if ( !channel )
> + {
> + printk(XENLOG_ERR
> + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> + d->domain_id);
> + return -EINVAL;
> + }
> +
> + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> + hdr.type = 0;
> + hdr.protocol = SCMI_BASE_PROTOCOL;
> +
> + tx.flags = 0;
> +
> + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> + if ( ret )
> + return ret;
> +
> + ret = check_scmi_status(rx);
> +
> + return ret;
> +}
> +
> +static void scmi_domain_destroy(struct domain *d)
> +{
> + struct scmi_channel *channel;
> +
> + if ( !d->arch.sci )
> + return;
> +
> + channel = d->arch.sci;
> + spin_lock(&channel->lock);
> +
> + relinquish_scmi_channel(channel);
> + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> +
> + d->arch.sci = NULL;
> +
> + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
> + spin_unlock(&channel->lock);
> + return;
> +}
> +
> +static bool scmi_handle_call(struct domain *d, void *args)
> +{
> + bool res = false;
> + struct scmi_channel *agent_channel;
> + struct arm_smccc_res resp;
> + struct cpu_user_regs *regs = args;
> +
> + if ( !d->arch.sci )
> + return false;
> +
> + agent_channel = d->arch.sci;
> + spin_lock(&agent_channel->lock);
> +
> + if ( agent_channel->func_id != regs->x0 )
> + {
> + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> + goto unlock;
> + }
> +
> + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> + agent_channel->chan_id, &resp);
> +
> + set_user_reg(regs, 0, resp.a0);
> + set_user_reg(regs, 1, resp.a1);
> + set_user_reg(regs, 2, resp.a2);
> + set_user_reg(regs, 3, resp.a3);
> + res = true;
> +unlock:
> + spin_unlock(&agent_channel->lock);
> +
> + return res;
> +}
> +
> +static int scmi_get_channel_paddr(void *scmi_ops,
> + struct xen_arch_domainconfig *config)
> +{
> + struct scmi_channel *agent_channel = scmi_ops;
> +
> + if ( !agent_channel )
> + return -EINVAL;
> +
> + config->sci_agent_paddr = agent_channel->paddr;
> + return 0;
> +}
I am still not sure why it couldn't be done by scmi_domain_init.
> +static const struct dt_device_match scmi_smc_match[] __initconst =
> +{
> + DT_MATCH_SCMI_SMC,
> + { /* sentinel */ },
> +};
> +
> +static const struct sci_mediator_ops scmi_ops =
> +{
> + .probe = scmi_probe,
> + .domain_init = scmi_domain_init,
> + .domain_destroy = scmi_domain_destroy,
> + .add_dt_device = scmi_add_dt_device,
> + .relinquish_resources = scmi_relinquish_resources,
> + .handle_call = scmi_handle_call,
> + .get_channel_info = scmi_get_channel_paddr
> +};
> +
> +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> + scmi_smc_match, &scmi_ops);
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 9180be5e86..a67237942d 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
>
> #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
>
> struct xen_arch_domainconfig {
> /* IN/OUT */
> --
> 2.27.0
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-17 16:38 ` Julien Grall
@ 2021-12-20 15:41 ` Oleksii Moisieiev
2021-12-24 14:42 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-20 15:41 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi Julien,
On Fri, Dec 17, 2021 at 04:38:31PM +0000, Julien Grall wrote:
>
>
> On 17/12/2021 13:58, Oleksii Moisieiev wrote:
> > Hi Julien,
>
> Hi,
>
> > On Fri, Dec 17, 2021 at 01:37:35PM +0000, Julien Grall wrote:
> > > Hi,
> > >
> > > On 17/12/2021 13:23, Oleksii Moisieiev wrote:
> > > > > > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > > > > > +{
> > > > > > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > > > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > > > +}
> > > > > > +
> > > > > > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > > > > > + uint64_t len)
> > > > > > +{
> > > > > > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > > > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > > > +}
> > > > >
> > > > > I wonder, why we need an extra level of indirection here. And if this is
> > > > > really needed, I wonder why map(unmap)_memory* name was chosen, as there is
> > > > > no memory mapping/unmapping really happens here.
> > > > >
> > > >
> > > > I've added extra indirection to hide math like
> > > > paddr_to_pfn(PAGE_ALIGN(addr + len -1)
> > > > so you don't have to math in each call. unmap_memory_from_domain called
> > > > from 2 places, so I moved both calls to separate function.
> > > > Although, I agree that map/unmap is not perfect name. I consider
> > > > renaming it to mem_permit_acces and mam_deny_access.
> > >
> > > I haven't looked at the rest of the series. But this discussion caught my
> > > eye. This code implies that the address is page-aligned but the length not.
> > > Is that intended?
> > >
> > > That said, if you give permission to the domain on a full page then it means
> > > it may be able to access address it should not. Can you explain why this is
> > > fine?
> > >
> >
> > The idea was that xen receives some memory from the dt_node linux,scmi_mem,
> > then we split memory between the agents, so each agent get 1 page (we
> > allocate 0x10 pages right now).
>
> Thanks for the clarification. Does this imply the guest will be able to
> write message directly to the firmware?
We used DEN0056C Specification as base. Available on: https://developer.arm.com/documentation/den0056/latest.
SCMI transport is described in Section 5.1. We implemented Shared Memory transport.
Firmware has N pages of the shared memory, used to communicate with Agents.
It allocates N agents and assign a page for each agent, such as:
-------------------------------------
| Agent H | Agent 1 | Agent 2 | ... |
-------------------------------------
Agent H is the privilleged Hypervisor agent, which is used to do the base commands,
such as getting Agent list, set\unset permissions etc.
Hypervisor assign agent to the guest and maps page, related to the agent to the Guest.
So the Guest, which is Agent 1 will get an access to Agent 1 page.
Guest places SCMI message to Agent 1 memory, then sends SMC message.
Hypervisor process SMC request, add agent id to the message parameters and redirects it to the Firmware.
Based on the agent_id Firmware knows which page it should read.
Then after permission check ( if the resetId/clockID/powerID etc from message
is assigned to agent_id ) it does changes to the HW and places response to Agent
shared memory and marks channel as FREE ( by setting free bit in shared memory ).
Once channel is marked as free - Guest read response from the shared memory.
Non-virtualized systems will work as well. OS should send SMC directly to the Firmware.
>
> If so, this brings a few more questions:
> 1) What will the guest write in it? Can it contains addresses?
Guest can write scmi request to the shared memory, which include the following data:
1) protocol_id - which protocol is requested, such as clock, power, reset etc
2) message_id - action that should be done to HW, such as do_reset or get_clock
3) message data - which includes reset_id/clock_id/power_id etc. that should be changed.
Reset IDs and Clock IDs are assigned in Firmware. Guest receives ID, corresponding to the device from the device-tree.
dt_node as an example:
&avb {
scmi_devid = <0>;
clocks = <&scmi_clock 0>;
power-domains = <&scmi_power 0>;
resets = <&scmi_reset 0>;
};
> 2) What are the expected memory attribute for the regions?
xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> 3) What's the threat model for the firmware? Will it verify every request?
Firmware reads data from agent page, then makes check if clockid/resetid/powerid
etc is assigned to agent.
During building guest, Xen sends permission request to the firmware for each device,
which is passed-through to the guest. So for avb from previous example the
device_id 0 permission request will be sent. Based on the device_id firmware will
set permission for clockid 0, resetid 0 and powerid 0.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-18 2:14 ` Stefano Stabellini
@ 2021-12-20 18:12 ` Oleksii Moisieiev
2021-12-21 0:52 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-20 18:12 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > This is the implementation of SCI interface, called SCMI-SMC driver,
> > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > This allows devices from the Domains to work with clocks, resets and
> > power-domains without access to CPG.
> >
> > The following features are implemented:
> > - request SCMI channels from ATF and pass channels to Domains;
> > - set device permissions for Domains based on the Domain partial
> > device-tree. Devices with permissions are able to work with clocks,
> > resets and power-domains via SCMI;
> > - redirect scmi messages from Domains to ATF.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > xen/arch/arm/Kconfig | 2 +
> > xen/arch/arm/sci/Kconfig | 10 +
> > xen/arch/arm/sci/Makefile | 1 +
> > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > xen/include/public/arch-arm.h | 1 +
> > 5 files changed, 809 insertions(+)
> > create mode 100644 xen/arch/arm/sci/Kconfig
> > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index 186e1db389..02d96c6cfc 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -114,6 +114,8 @@ config SCI
> > support. It allows guests to control system resourcess via one of
> > SCI mediators implemented in XEN.
> >
> > +source "arch/arm/sci/Kconfig"
> > +
> > endmenu
> >
> > menu "ARM errata workaround via the alternative framework"
> > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > new file mode 100644
> > index 0000000000..9563067ddc
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/Kconfig
> > @@ -0,0 +1,10 @@
> > +config SCMI_SMC
> > + bool "Enable SCMI-SMC mediator driver"
> > + default n
> > + depends on SCI
> > + ---help---
> > +
> > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > + This feature allows drivers from Domains to work with System
> > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > + for communication.
> > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > index 837dc7492b..67f2611872 100644
> > --- a/xen/arch/arm/sci/Makefile
> > +++ b/xen/arch/arm/sci/Makefile
> > @@ -1 +1,2 @@
> > obj-y += sci.o
> > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > new file mode 100644
> > index 0000000000..2eb01ea82d
> > --- /dev/null
> > +++ b/xen/arch/arm/sci/scmi_smc.c
> > @@ -0,0 +1,795 @@
> > +/*
> > + * xen/arch/arm/sci/scmi_smc.c
> > + *
> > + * SCMI mediator driver, using SCP as transport.
> > + *
> > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > + * Copyright (C) 2021, EPAM Systems.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation; either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + */
> > +
> > +#include <asm/sci/sci.h>
> > +#include <asm/smccc.h>
> > +#include <asm/io.h>
> > +#include <xen/bitops.h>
> > +#include <xen/config.h>
> > +#include <xen/sched.h>
> > +#include <xen/device_tree.h>
> > +#include <xen/iocap.h>
> > +#include <xen/init.h>
> > +#include <xen/err.h>
> > +#include <xen/lib.h>
> > +#include <xen/list.h>
> > +#include <xen/mm.h>
> > +#include <xen/string.h>
> > +#include <xen/time.h>
> > +#include <xen/vmap.h>
> > +
> > +#define SCMI_BASE_PROTOCOL 0x10
> > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > +
> > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > +#define SCMI_SUCCESS 0
> > +#define SCMI_NOT_SUPPORTED (-1)
> > +#define SCMI_INVALID_PARAMETERS (-2)
> > +#define SCMI_DENIED (-3)
> > +#define SCMI_NOT_FOUND (-4)
> > +#define SCMI_OUT_OF_RANGE (-5)
> > +#define SCMI_BUSY (-6)
> > +#define SCMI_COMMS_ERROR (-7)
> > +#define SCMI_GENERIC_ERROR (-8)
> > +#define SCMI_HARDWARE_ERROR (-9)
> > +#define SCMI_PROTOCOL_ERROR (-10)
> > +
> > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > +
> > +#define SCMI_SMC_ID "arm,smc-id"
> > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
>
> I could find the following SCMI binding in Linux, which describes
> the arm,scmi-smc compatible and the arm,smc-id property:
>
> Documentation/devicetree/bindings/firmware/arm,scmi.yaml
>
> However, linux,scmi_mem is not described. Aren't you supposed to read
> the "shmem" property instead? And the compatible string used for this
> seems to be "arm,scmi-shmem".
>
We use linux,scmi_mem node to reserve memory, needed for all
channels:
reserved-memory {
/* reserved region for scmi channels*/
scmi_memory: linux,scmi_mem@53FF0000 {
no-map;
reg = <0x0 0x53FF0000 0x0 0x10000>;
};
};
arm,scmi-shmem node used in shmem property defines only 1 page needed to
the current scmi channel:
cpu_scp_shm: scp-shmem@0x53FF0000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF0000 0x0 0x1000>;
};
For each Domain reg points to unigue page from linux,scmi_mem region,
assigned to this agent.
>
> > +#define SCMI_SHMEM "shmem"
> > +
> > +#define HYP_CHANNEL 0x0
> > +
> > +#define HDR_ID GENMASK(7,0)
> > +#define HDR_TYPE GENMASK(9, 8)
> > +#define HDR_PROTO GENMASK(17, 10)
> > +
> > +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> > +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> > +
> > +#define FIELD_GET(_mask, _reg)\
> > + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> > +#define FIELD_PREP(_mask, _val)\
> > + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> > +
> > +typedef struct scmi_msg_header {
> > + uint8_t id;
> > + uint8_t type;
> > + uint8_t protocol;
> > +} scmi_msg_header_t;
> > +
> > +typedef struct scmi_perms_tx {
> > + uint32_t agent_id;
> > + uint32_t device_id;
> > + uint32_t flags;
> > +} scmi_perms_tx_t;
> > +
> > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> > +
> > +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> > +
> > +struct scmi_shared_mem {
> > + uint32_t reserved;
> > + uint32_t channel_status;
> > + uint32_t reserved1[2];
> > + uint32_t flags;
> > + uint32_t length;
> > + uint32_t msg_header;
> > + uint8_t msg_payload[];
> > +};
> > +
> > +struct scmi_channel {
> > + int chan_id;
> > + int agent_id;
> > + uint32_t func_id;
> > + int domain_id;
> > + uint64_t paddr;
> > + struct scmi_shared_mem *shmem;
> > + spinlock_t lock;
> > + struct list_head list;
> > +};
> > +
> > +struct scmi_data {
> > + struct list_head channel_list;
> > + spinlock_t channel_list_lock;
> > + bool initialized;
> > + u64 shmem_addr, shmem_size;
> > +};
> > +
> > +static struct scmi_data scmi_data;
> > +
> > +/*
> > + * pack_scmi_header() - packs and returns 32-bit header
> > + *
> > + * @hdr: pointer to header containing all the information on message id,
> > + * protocol id and type id.
> > + *
> > + * Return: 32-bit packed message header to be sent to the platform.
> > + */
> > +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> > +{
> > + return FIELD_PREP(HDR_ID, hdr->id) |
> > + FIELD_PREP(HDR_TYPE, hdr->type) |
> > + FIELD_PREP(HDR_PROTO, hdr->protocol);
> > +}
> > +
> > +/*
> > + * unpack_scmi_header() - unpacks and records message and protocol id
> > + *
> > + * @msg_hdr: 32-bit packed message header sent from the platform
> > + * @hdr: pointer to header to fetch message and protocol id.
> > + */
> > +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> > +{
> > + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> > + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> > + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> > +}
> > +
> > +static inline int channel_is_free(struct scmi_channel *chan_info)
> > +{
> > + return ( chan_info->shmem->channel_status
> > + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
>
> Does this need a memory barrier? Or not, because the other end always
> runs on the same CPU at a different execution level so the
> channel_status would be always guaranteed to be read as updated?
>
It don't because the other end runs on the same CPU. Other mediator
implemetaions, which uses different areas may need memory barrier.
>
> > +}
> > +
> > +static int send_smc_message(struct scmi_channel *chan_info,
> > + scmi_msg_header_t *hdr, void *data, int len)
> > +{
> > + struct arm_smccc_res resp;
> > + int ret;
> > +
> > + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> > + chan_info->shmem->channel_status, len);
> > + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> > + hdr->id, hdr->type, hdr->protocol);
> > +
> > + ret = channel_is_free(chan_info);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + chan_info->shmem->channel_status = 0x0;
> > + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> > + chan_info->shmem->flags = 0x0;
> > + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> > + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> > +
> > + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> > + chan_info->shmem);
> > + if ( len > 0 && data )
> > + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
>
> Again, here we don't need a barrier because it is implicit in the SMC?
>
As I mentioned before, the other end runs on the same CPU.
> Don't we need to check that "len" fits in the shared memory?
>
I think it's a good point. I'll add len check in v2.
>
> > + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> > + &resp);
> > +
> > + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> > +
> > + if ( resp.a0 )
> > + return -EOPNOTSUPP;
>
> Why is that?
>
This change was presented in kernel by Sudeep Holla in commit:
f7199cf489027ae38a9a82312d13025f7aefa0b8
However, link posted in the commit:
https://lore.kernel.org/r/20200417103232.6896-1-sudeep.holla@arm.com
Leads to slightly different patch:
+ if (res.a0 == SMCCC_RET_NOT_SUPPORTED)
+ return -EOPNOTSUPP;
+ else if (res.a0)
+ return -EINVAL;
+ return 0;
I don't know why it differs from the original commit, but I'll check and place
the correct implementation in v2.
>
> > + return 0;
> > +}
> > +
> > +static int check_scmi_status(int scmi_status)
> > +{
> > + if ( scmi_status == SCMI_SUCCESS )
> > + return 0;
> > +
> > + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> > +
> > + switch ( scmi_status )
> > + {
> > + case SCMI_NOT_SUPPORTED:
> > + return -EOPNOTSUPP;
> > + case SCMI_INVALID_PARAMETERS:
> > + return -EINVAL;
> > + case SCMI_DENIED:
> > + return -EACCES;
> > + case SCMI_NOT_FOUND:
> > + return -ENOENT;
> > + case SCMI_OUT_OF_RANGE:
> > + return -ERANGE;
> > + case SCMI_BUSY:
> > + return -EBUSY;
> > + case SCMI_COMMS_ERROR:
> > + return -ENOTCONN;
> > + case SCMI_GENERIC_ERROR:
> > + return -EIO;
> > + case SCMI_HARDWARE_ERROR:
> > + return -ENXIO;
> > + case SCMI_PROTOCOL_ERROR:
> > + return -EBADMSG;
> > + }
> > +
> > + return -EINVAL;
> > +}
> > +
> > +static int get_smc_response(struct scmi_channel *chan_info,
> > + scmi_msg_header_t *hdr, void *data, int len)
> > +{
> > + int recv_len;
> > + int ret;
> > +
> > + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> > +
> > + ret = channel_is_free(chan_info);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
>
> I am not familiar with the spec (do you have a link?) but is it expected
> that the channel is "free" when actually we want to read a message on
> the channel?
>
Here is the link https://developer.arm.com/documentation/den0056/latest
Figure 6 in Section 5.1.1.
Caller marks channel as busy, then callee process message and marks channel as free.
We are implementing polling based communication flow.
>
>
> > + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> > +
> > + if ( recv_len < 0 )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Wrong size of smc message. Data may be invalid\n");
> > + return -EINVAL;
> > + }
> > +
> > + if ( recv_len > len )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Not enough buffer for message %d, expecting %d\n",
> > + recv_len, len);
> > + return -EINVAL;
> > + }
> > +
> > + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> > +
> > + if ( recv_len > 0 )
> > + {
> > + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> > + void *rx_data, int rx_size)
> > +{
> > + int ret = 0;
> > +
> > + if ( !hdr )
> > + return -EINVAL;
> > +
> > + spin_lock(&channel->lock);
> > +
> > + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> > +clean:
> > + spin_unlock(&channel->lock);
> > +
> > + return ret;
> > +}
> > +
> > +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
>
> please use parenthesis around the inner if (also in other places)
>
Thank you for the remark. I will fix it in v2.
>
> > + if ( curr->chan_id == chan_id )
> > + {
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
>
> Use domid_t for domain ids.
Thanks, I will fix it in v2.
>
> Also, wouldn't it be better to implement it as:
>
> static inline struct scmi_channel *get_channel_by_domain(struct domain *d) {
> return d->arch.sci
> }
>
That's a good point. I will take a look on it and fix in v2.
>
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > + if ( curr->domain_id == domain_id )
> > + {
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> > +{
> > + struct scmi_channel *curr;
> > + bool found = false;
> > +
> > + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > + if ( (curr->domain_id == DOMID_INVALID)
> > + && (curr->chan_id != HYP_CHANNEL) )
>
> If you use DOMID_XEN for HYP_CHANNEL, then this check becomes more
> intuitive
>
We do not have direct relation between channel id and domain id.
One channel id can be reused by different domain_ids. So from my standpoint,
DOMID_XEN doesn't fit here.
>
> > + {
> > + curr->domain_id = domain_id;
> > + found = true;
> > + break;
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + if ( found )
> > + return curr;
> > +
> > + return NULL;
> > +}
> > +
> > +static void relinquish_scmi_channel(struct scmi_channel *channel)
> > +{
> > + spin_lock(&scmi_data.channel_list_lock);
> > + ASSERT(channel != NULL);
>
> the ASSERT could be before the spin_lock
>
Thank you. I will fix it in v2.
>
> > + channel->domain_id = DOMID_INVALID;
> > + spin_unlock(&scmi_data.channel_list_lock);
> > +}
> > +
> > +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> > + uint32_t func_id, uint64_t addr)
> > +{
> > + struct scmi_channel *channel;
> > + mfn_t mfn;
> > +
> > + channel = get_channel_by_id(chan_id);
> > + if ( channel )
> > + return ERR_PTR(EEXIST);
> > +
> > + channel = xmalloc(struct scmi_channel);
> > + if ( !channel )
> > + return ERR_PTR(ENOMEM);
> > +
> > + channel->chan_id = chan_id;
> > + channel->func_id = func_id;
> > + channel->domain_id = DOMID_INVALID;
>
> I take you are using DOMID_INVALID to mark a channel used by Xen itself?
> If so, then DOMID_XEN would be more appropriate.
>
I use DOMID_INVALID to mark channel as free.
>
> > + mfn = maddr_to_mfn(addr);
> > + channel->shmem = vmap(&mfn, 1);
>
> One thing to be careful is the mapping attributes, for a couple of
> reasons. As you might be aware, the ARM architecture forbids mismatching
> attributes for mapping memory in different places in the system. So the
> attributes that we use here must be the same used by the firmware
> (and/or the guest.)
>
> The second reason to be careful is that in the bindings example
> Documentation/devicetree/bindings/firmware/arm,scmi.yaml the shared
> memory is "mmio-sram", which is special. It is not supposed to be normal
> memory, but it is OK to map it cacheable. Still, it might be more
> appropriate to use ioremap_cache.
>
Originally, I used vmap here to have memcpy and it works fine in our
setup. But I will do some research and email you with the results.
>
> > + if ( !channel->shmem )
> > + {
> > + xfree(channel);
> > + return ERR_PTR(ENOMEM);
> > + }
> > +
> > + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> > + channel->paddr = addr;
> > + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> > + spin_lock_init(&channel->lock);
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_add(&channel->list, &scmi_data.channel_list);
> > + spin_unlock(&scmi_data.channel_list_lock);
> > + return channel;
> > +}
> > +
> > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > +{
> > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > +}
> > +
> > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > + uint64_t len)
> > +{
> > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > +}
> > +
> > +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> > + uint64_t size)
> > +{
> > + struct dt_device_node *shmem_node;
> > + __be32 *hw_reg;
> > + const struct dt_property *pp;
> > + uint32_t len;
> > +
> > + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
>
> Here we are using "arm,scmi-shmem" while below we are checking for
> "linux,scmi_mem". What's the difference?
linux,scmi_mem (I posted nodes examples above) describes memory region, allocated
for all channels, while arm,scmi-shmem points to the exact channel (page in
linux,scmi_mem region).
>
> Also, this function is looking for "arm,scmi-shmem" in dt_host and
> replaces its value. For dom0less domUs we'll probably need a
> make_scmi_node function to create the node from scratch like for
> instance xen/arch/arm/domain_build.c:make_gic_domU_node.
>
> I wonder if we had such a function whether it wouldn't be better to also
> use it for dom0 (and blacklist the physical "arm,scmi-shmem" in
> handle_node so that dom0 doesn't get the real shared memory information
> by accident).
>
Thank you for the remark. I will rework this in v2.
>
> > +
> > + if ( !shmem_node )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> > + return -EINVAL;
> > + }
> > +
> > + pp = dt_find_property(shmem_node, "reg", &len);
> > + if ( !pp )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> > + return -ENOENT;
> > + }
> > +
> > + hw_reg = pp->value;
> > + dt_set_range(&hw_reg, shmem_node, addr, size);
> > +
> > + return 0;
> > +}
> > +
> > +static void free_channel_list(void)
> > +{
> > + struct scmi_channel *curr, *_curr;
> > +
> > + spin_lock(&scmi_data.channel_list_lock);
> > + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> > + {
> > + vunmap(curr->shmem);
> > + list_del(&curr->list);
> > + xfree(curr);
> > + }
> > +
> > + spin_unlock(&scmi_data.channel_list_lock);
> > +}
> > +
> > +static __init bool scmi_probe(struct dt_device_node *scmi_node)
> > +{
> > + struct dt_device_node *shmem_node;
> > + int ret, i;
> > + struct scmi_channel *channel, *agent_channel;
> > + int n_agents;
> > + scmi_msg_header_t hdr;
> > + struct rx_t {
> > + int32_t status;
> > + uint32_t attributes;
> > + } rx;
>
> Should rx be defined at the top together with scmi_perms_tx_t and
> others?
>
I'd rather move scmi_perms_tx_t to scmi_add_device_by_devid because it's
used only in 1 place.
So we will have rx and tx in scmi_add_device_by_devid and rx ( which
differs from rx in scmi_add_device_by_devid ) in scmi_probe.
I think it will be more understandable and no need to make unique names.
What do you think about that?
>
> > + uint32_t func_id;
> > +
> > + ASSERT(scmi_node != NULL);
> > +
> > + INIT_LIST_HEAD(&scmi_data.channel_list);
> > + spin_lock_init(&scmi_data.channel_list_lock);
> > +
> > + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> > + {
> > + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> > + return false;
> > + }
> > +
> > + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
>
> From the spec, it looks like you should be getting the shared memory
> area from the phandle list "shmem".
>
We use SCMI_SHARED_MEMORY to get whole memory region (0x10 pages in my case),
we can use for the agents. As you can see below - Hypervisor received number of
agents from Firmware and split this region between agents.
>
> > + if ( IS_ERR_OR_NULL(shmem_node) )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> > + PTR_ERR(shmem_node));
> > + return false;
> > + }
> > +
> > + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> > + &scmi_data.shmem_size);
> > + if ( IS_ERR_VALUE(ret) )
> > + return false;
> > +
> > + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> > + if ( IS_ERR(channel) )
> > + return false;
> > +
> > + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = check_scmi_status(rx.status);
> > + if ( ret )
> > + goto clean;
> > +
> > + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> > + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> > +
> > + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> > + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> > +
> > + for ( i = 1; i < n_agents; i++ )
> > + {
>
> Given that HYP_CHANNEL is actually zero, it looks like we could do
> everything here in this loop but starting from i=0?
>
We allocate HYP_CHANNEL before loop because we need it to request number
of agents. And we don't need to send SCMI_BASE_DISCOVER_AGENT to
HYP_CHANNEL.
>
> > + uint32_t tx_agent_id = 0xFFFFFFFF;
> > + struct {
> > + int32_t status;
> > + uint32_t agent_id;
> > + char name[16];
> > + } da_rx;
> > +
> > + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> > + i * PAGE_SIZE);
> > + if ( IS_ERR(agent_channel) )
> > + {
> > + ret = PTR_ERR(agent_channel);
> > + goto clean;
> > + }
> > +
> > + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> > + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> > + if ( ret )
> > + goto clean;
> > +
> > + ret = check_scmi_status(da_rx.status);
> > + if ( ret )
> > + goto clean;
> > +
> > + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> > + da_rx.status, da_rx.agent_id, da_rx.name);
> > +
> > + agent_channel->agent_id = da_rx.agent_id;
> > + }
> > +
> > + scmi_data.initialized = true;
> > + return true;
> > +
> > +clean:
> > + free_channel_list();
> > + return ret == 0;
> > +}
> > +
> > +static int scmi_domain_init(struct domain *d)
> > +{
> > + struct scmi_channel *channel;
> > + int ret;
> > +
> > + if ( !scmi_data.initialized )
> > + return 0;
> > +
> > + channel = aquire_scmi_channel(d->domain_id);
> > + if ( IS_ERR_OR_NULL(channel) )
> > + return -ENOENT;
> > +
> > + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> > + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> > + channel->paddr);
> > +
> > + if ( is_hardware_domain(d) )
> > + {
> > + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> > + scmi_data.shmem_size);
> > + if ( IS_ERR_VALUE(ret) )
> > + goto error;
> > +
> > + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> > + if ( IS_ERR_VALUE(ret) )
> > + {
> > + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> > + scmi_data.shmem_size);
> > + if ( rc )
> > + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> > +
> > + goto error;
> > + }
> > + }
>
> Is dom0 the only domain to get direct access to the shared memory
> region? If so, I don't think it is a good idea to make Dom0 "special" in
> this case.
>
> Let me make an example: if we assign a device to a domU since boot, and
> dom0 wants to change the frequency of a clock that affects the assigned
> device (likely because it doesn't know it is assigned), then dom0
> shouldn't be able to. We might have to perform checks in Xen to make
> sure dom0 cannot stop the clock for the assigned device.
>
> So I think it would be better if all domains are treated the same way in
> the mediator unless really necessary.
>
> On the other hand, if all domains get access to the shared memory
> region, then I don't think this is likely the right place to create the
> dom0 mapping. We probably want to do it in domain_build.c in a way that
> can be reused for dom0less domUs.
>
The idea is that all domains have their own page in shared memory
region and unigue agent_id. Agent_id is used to set permissions for
clocks\resets\power-domains etc. So during creation of domain (domUs or
dom0) device-tree is processed using scmi_add_dt_device and
clocks\resets\power-domains which are related to dom0 devices will be
requested by using SCMI_BASE_SET_DEVICE_PERMISSIONS message.
All passed-through devices will be requested during DomU creation. Which
means dom0 do not have an access to clocks\resets\power-domains, which
are related to DomU.
>
> In regards to shared memory: it looks like the only two functions to
> access the real shared memory are send_smc_message and get_smc_response.
> If that is the case, then we actually don't need to expose the real
> shared memory to any of the domains.
>
> We could simply:
>
> - expose a regular normal memory region as dom0/domU channel memory
> - on SMC trap, read from the "fake" shared memory and set the
> corresponding real shared memory on the appropriate channel
> - issue the SMC call
> - on return from SMC, copy over data from the real shared memory to the
> "fake" channel reagion
Hypervisor redirects only SMC calls from guests and set agent_id to SMC
parameters as a7. The idea was to give page for each agent, so we don't
need to make additional read/write each time we receive SMC call.
All we povide from hypervisor is agent_id. Firmware is responsible for
reading memory from the correct address and place the response.
>
> This is useful if we need to "filter" any of the SCMI commands and
> options from the domains to the firmware, and also it is useful if the
> channel memory is not page aligned. But if the permissions are
> fine-grained enough and also the channel memory is page aligned (and
> multiple of 4K in size) then we could map the memory.
>
In current implementation we suppose that channel memory is page aligned.
I think that Firmware should be responsible for permissions handling and
"filtering", that's why permission calls were added to SCMI spec.
I tried to make mediator as thin as possible.
>
> > +
> > + d->arch.sci = channel;
> > +
> > + return 0;
> > +error:
> > + relinquish_scmi_channel(channel);
> > +
> > + return ret;
> > +}
> > +
> > +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> > +{
> > + struct scmi_channel *channel, *agent_channel;
> > + scmi_msg_header_t hdr;
> > + scmi_perms_tx_t tx;
> > + struct rx_t {
> > + int32_t status;
> > + uint32_t attributes;
> > + } rx;
> > + int ret;
> > +
> > + if ( !scmi_data.initialized )
> > + return 0;
> > +
> > + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> > +
> > + agent_channel = get_channel_by_domain(d->domain_id);
> > + if ( IS_ERR_OR_NULL(agent_channel) )
> > + return PTR_ERR(agent_channel);
> > +
> > + channel = get_channel_by_id(HYP_CHANNEL);
> > + if ( IS_ERR_OR_NULL(channel) )
> > + return PTR_ERR(channel);
> > +
> > + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + tx.agent_id = agent_channel->agent_id;
> > + tx.device_id = scmi_devid;
> > + tx.flags = SCMI_ALLOW_ACCESS;
> > +
> > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + ret = check_scmi_status(rx.status);
> > + if ( IS_ERR_VALUE(ret) )
> > + return ret;
> > +
> > + return 0;
> > +}
> > +
> > +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> > +{
> > + uint32_t scmi_devid;
> > +
> > + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> > + return 0;
> > +
> > + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> > + return 0;
>
> scmi_devid is another property that is not documented in the binding.
>
This property should be added to the device nodes, which are using scmi
to work with clocks\resets\power-domains etc. This id should match the
device_id, defined in Firmware. Hypervisor send this device_id to the Firmware
as parameter to the permission request. Firmware set permissions to
clocks\resets\power-domains, related to this device.
>
> > + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> > +
> > + return scmi_add_device_by_devid(d, scmi_devid);
> > +}
> > +
> > +static int scmi_relinquish_resources(struct domain *d)
> > +{
> > + int ret;
> > + struct scmi_channel *channel, *agent_channel;
> > + scmi_msg_header_t hdr;
> > + struct reset_agent_tx {
> > + uint32_t agent_id;
> > + uint32_t flags;
> > + } tx;
> > + uint32_t rx;
> > +
> > + if ( !d->arch.sci )
> > + return 0;
> > +
> > + agent_channel = d->arch.sci;
> > +
> > + spin_lock(&agent_channel->lock);
> > + tx.agent_id = agent_channel->agent_id;
> > + spin_unlock(&agent_channel->lock);
> > +
> > + channel = get_channel_by_id(HYP_CHANNEL);
> > + if ( !channel )
> > + {
> > + printk(XENLOG_ERR
> > + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> > + d->domain_id);
> > + return -EINVAL;
> > + }
> > +
> > + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> > + hdr.type = 0;
> > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > +
> > + tx.flags = 0;
> > +
> > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> > + if ( ret )
> > + return ret;
> > +
> > + ret = check_scmi_status(rx);
> > +
> > + return ret;
> > +}
> > +
> > +static void scmi_domain_destroy(struct domain *d)
> > +{
> > + struct scmi_channel *channel;
> > +
> > + if ( !d->arch.sci )
> > + return;
> > +
> > + channel = d->arch.sci;
> > + spin_lock(&channel->lock);
> > +
> > + relinquish_scmi_channel(channel);
> > + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> > +
> > + d->arch.sci = NULL;
> > +
> > + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
> > + spin_unlock(&channel->lock);
> > + return;
> > +}
> > +
> > +static bool scmi_handle_call(struct domain *d, void *args)
> > +{
> > + bool res = false;
> > + struct scmi_channel *agent_channel;
> > + struct arm_smccc_res resp;
> > + struct cpu_user_regs *regs = args;
> > +
> > + if ( !d->arch.sci )
> > + return false;
> > +
> > + agent_channel = d->arch.sci;
> > + spin_lock(&agent_channel->lock);
> > +
> > + if ( agent_channel->func_id != regs->x0 )
> > + {
> > + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> > + goto unlock;
> > + }
> > +
> > + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> > + agent_channel->chan_id, &resp);
> > +
> > + set_user_reg(regs, 0, resp.a0);
> > + set_user_reg(regs, 1, resp.a1);
> > + set_user_reg(regs, 2, resp.a2);
> > + set_user_reg(regs, 3, resp.a3);
> > + res = true;
> > +unlock:
> > + spin_unlock(&agent_channel->lock);
> > +
> > + return res;
> > +}
> > +
> > +static int scmi_get_channel_paddr(void *scmi_ops,
> > + struct xen_arch_domainconfig *config)
> > +{
> > + struct scmi_channel *agent_channel = scmi_ops;
> > +
> > + if ( !agent_channel )
> > + return -EINVAL;
> > +
> > + config->sci_agent_paddr = agent_channel->paddr;
> > + return 0;
> > +}
>
> I am still not sure why it couldn't be done by scmi_domain_init.
>
I can move this logic to scmi_domain_init, but in this case I have to add
struct xen_arch_domainconfig *config as input parameter to
scmi_domain_init and pass NULL from construct_dom0.
Do you think this approach would be better?
Also I think it's reasonable to pass xen_arch_domainconfig so different
implementations could set another data they would probably need.
>
> > +static const struct dt_device_match scmi_smc_match[] __initconst =
> > +{
> > + DT_MATCH_SCMI_SMC,
> > + { /* sentinel */ },
> > +};
> > +
> > +static const struct sci_mediator_ops scmi_ops =
> > +{
> > + .probe = scmi_probe,
> > + .domain_init = scmi_domain_init,
> > + .domain_destroy = scmi_domain_destroy,
> > + .add_dt_device = scmi_add_dt_device,
> > + .relinquish_resources = scmi_relinquish_resources,
> > + .handle_call = scmi_handle_call,
> > + .get_channel_info = scmi_get_channel_paddr
> > +};
> > +
> > +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> > + scmi_smc_match, &scmi_ops);
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > index 9180be5e86..a67237942d 100644
> > --- a/xen/include/public/arch-arm.h
> > +++ b/xen/include/public/arch-arm.h
> > @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> > #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> >
> > #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> > +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
> >
> > struct xen_arch_domainconfig {
> > /* IN/OUT */
> > --
> > 2.27.0
> >
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-20 18:12 ` Oleksii Moisieiev
@ 2021-12-21 0:52 ` Stefano Stabellini
2021-12-21 20:03 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-21 0:52 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> Hi Stefano,
>
> On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > This allows devices from the Domains to work with clocks, resets and
> > > power-domains without access to CPG.
> > >
> > > The following features are implemented:
> > > - request SCMI channels from ATF and pass channels to Domains;
> > > - set device permissions for Domains based on the Domain partial
> > > device-tree. Devices with permissions are able to work with clocks,
> > > resets and power-domains via SCMI;
> > > - redirect scmi messages from Domains to ATF.
> > >
> > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > ---
> > > xen/arch/arm/Kconfig | 2 +
> > > xen/arch/arm/sci/Kconfig | 10 +
> > > xen/arch/arm/sci/Makefile | 1 +
> > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > xen/include/public/arch-arm.h | 1 +
> > > 5 files changed, 809 insertions(+)
> > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > >
> > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > index 186e1db389..02d96c6cfc 100644
> > > --- a/xen/arch/arm/Kconfig
> > > +++ b/xen/arch/arm/Kconfig
> > > @@ -114,6 +114,8 @@ config SCI
> > > support. It allows guests to control system resourcess via one of
> > > SCI mediators implemented in XEN.
> > >
> > > +source "arch/arm/sci/Kconfig"
> > > +
> > > endmenu
> > >
> > > menu "ARM errata workaround via the alternative framework"
> > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > new file mode 100644
> > > index 0000000000..9563067ddc
> > > --- /dev/null
> > > +++ b/xen/arch/arm/sci/Kconfig
> > > @@ -0,0 +1,10 @@
> > > +config SCMI_SMC
> > > + bool "Enable SCMI-SMC mediator driver"
> > > + default n
> > > + depends on SCI
> > > + ---help---
> > > +
> > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > + This feature allows drivers from Domains to work with System
> > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > + for communication.
> > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > index 837dc7492b..67f2611872 100644
> > > --- a/xen/arch/arm/sci/Makefile
> > > +++ b/xen/arch/arm/sci/Makefile
> > > @@ -1 +1,2 @@
> > > obj-y += sci.o
> > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > new file mode 100644
> > > index 0000000000..2eb01ea82d
> > > --- /dev/null
> > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > @@ -0,0 +1,795 @@
> > > +/*
> > > + * xen/arch/arm/sci/scmi_smc.c
> > > + *
> > > + * SCMI mediator driver, using SCP as transport.
> > > + *
> > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > + * Copyright (C) 2021, EPAM Systems.
> > > + *
> > > + * This program is free software; you can redistribute it and/or modify
> > > + * it under the terms of the GNU General Public License as published by
> > > + * the Free Software Foundation; either version 2 of the License, or
> > > + * (at your option) any later version.
> > > + *
> > > + * This program is distributed in the hope that it will be useful,
> > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > + * GNU General Public License for more details.
> > > + */
> > > +
> > > +#include <asm/sci/sci.h>
> > > +#include <asm/smccc.h>
> > > +#include <asm/io.h>
> > > +#include <xen/bitops.h>
> > > +#include <xen/config.h>
> > > +#include <xen/sched.h>
> > > +#include <xen/device_tree.h>
> > > +#include <xen/iocap.h>
> > > +#include <xen/init.h>
> > > +#include <xen/err.h>
> > > +#include <xen/lib.h>
> > > +#include <xen/list.h>
> > > +#include <xen/mm.h>
> > > +#include <xen/string.h>
> > > +#include <xen/time.h>
> > > +#include <xen/vmap.h>
> > > +
> > > +#define SCMI_BASE_PROTOCOL 0x10
> > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > +
> > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > +#define SCMI_SUCCESS 0
> > > +#define SCMI_NOT_SUPPORTED (-1)
> > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > +#define SCMI_DENIED (-3)
> > > +#define SCMI_NOT_FOUND (-4)
> > > +#define SCMI_OUT_OF_RANGE (-5)
> > > +#define SCMI_BUSY (-6)
> > > +#define SCMI_COMMS_ERROR (-7)
> > > +#define SCMI_GENERIC_ERROR (-8)
> > > +#define SCMI_HARDWARE_ERROR (-9)
> > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > +
> > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > +
> > > +#define SCMI_SMC_ID "arm,smc-id"
> > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> >
> > I could find the following SCMI binding in Linux, which describes
> > the arm,scmi-smc compatible and the arm,smc-id property:
> >
> > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> >
> > However, linux,scmi_mem is not described. Aren't you supposed to read
> > the "shmem" property instead? And the compatible string used for this
> > seems to be "arm,scmi-shmem".
> >
>
> We use linux,scmi_mem node to reserve memory, needed for all
> channels:
>
> reserved-memory {
> /* reserved region for scmi channels*/
> scmi_memory: linux,scmi_mem@53FF0000 {
> no-map;
> reg = <0x0 0x53FF0000 0x0 0x10000>;
> };
> };
>
> arm,scmi-shmem node used in shmem property defines only 1 page needed to
> the current scmi channel:
>
> cpu_scp_shm: scp-shmem@0x53FF0000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF0000 0x0 0x1000>;
> };
>
> For each Domain reg points to unigue page from linux,scmi_mem region,
> assigned to this agent.
If we were to use "linux,scmi_mem" we would have to introduce it as a
compatible string, not as a node name, and it would need to be described
in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
But from your description I don't think it is necessary. We can just use
"arm,scmi-shmem" to describe all the required regions:
reserved-memory {
scp-shmem@0x53FF0000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF0000 0x0 0x1000>;
};
scp-shmem@0x53FF1000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF1000 0x0 0x1000>;
};
scp-shmem@0x53FF2000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF2000 0x0 0x1000>;
};
...
In other words, if all the individual channel pages are described as
"arm,scmi-shmem", why do we also need a single larger region as
"linux,scmi_mem"?
> > > +#define SCMI_SHMEM "shmem"
> > > +
> > > +#define HYP_CHANNEL 0x0
> > > +
> > > +#define HDR_ID GENMASK(7,0)
> > > +#define HDR_TYPE GENMASK(9, 8)
> > > +#define HDR_PROTO GENMASK(17, 10)
> > > +
> > > +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> > > +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> > > +
> > > +#define FIELD_GET(_mask, _reg)\
> > > + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> > > +#define FIELD_PREP(_mask, _val)\
> > > + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> > > +
> > > +typedef struct scmi_msg_header {
> > > + uint8_t id;
> > > + uint8_t type;
> > > + uint8_t protocol;
> > > +} scmi_msg_header_t;
> > > +
> > > +typedef struct scmi_perms_tx {
> > > + uint32_t agent_id;
> > > + uint32_t device_id;
> > > + uint32_t flags;
> > > +} scmi_perms_tx_t;
> > > +
> > > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> > > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> > > +
> > > +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> > > +
> > > +struct scmi_shared_mem {
> > > + uint32_t reserved;
> > > + uint32_t channel_status;
> > > + uint32_t reserved1[2];
> > > + uint32_t flags;
> > > + uint32_t length;
> > > + uint32_t msg_header;
> > > + uint8_t msg_payload[];
> > > +};
> > > +
> > > +struct scmi_channel {
> > > + int chan_id;
> > > + int agent_id;
> > > + uint32_t func_id;
> > > + int domain_id;
> > > + uint64_t paddr;
> > > + struct scmi_shared_mem *shmem;
> > > + spinlock_t lock;
> > > + struct list_head list;
> > > +};
> > > +
> > > +struct scmi_data {
> > > + struct list_head channel_list;
> > > + spinlock_t channel_list_lock;
> > > + bool initialized;
> > > + u64 shmem_addr, shmem_size;
> > > +};
> > > +
> > > +static struct scmi_data scmi_data;
> > > +
> > > +/*
> > > + * pack_scmi_header() - packs and returns 32-bit header
> > > + *
> > > + * @hdr: pointer to header containing all the information on message id,
> > > + * protocol id and type id.
> > > + *
> > > + * Return: 32-bit packed message header to be sent to the platform.
> > > + */
> > > +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> > > +{
> > > + return FIELD_PREP(HDR_ID, hdr->id) |
> > > + FIELD_PREP(HDR_TYPE, hdr->type) |
> > > + FIELD_PREP(HDR_PROTO, hdr->protocol);
> > > +}
> > > +
> > > +/*
> > > + * unpack_scmi_header() - unpacks and records message and protocol id
> > > + *
> > > + * @msg_hdr: 32-bit packed message header sent from the platform
> > > + * @hdr: pointer to header to fetch message and protocol id.
> > > + */
> > > +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> > > +{
> > > + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> > > + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> > > + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> > > +}
> > > +
> > > +static inline int channel_is_free(struct scmi_channel *chan_info)
> > > +{
> > > + return ( chan_info->shmem->channel_status
> > > + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
> >
> > Does this need a memory barrier? Or not, because the other end always
> > runs on the same CPU at a different execution level so the
> > channel_status would be always guaranteed to be read as updated?
> >
>
> It don't because the other end runs on the same CPU. Other mediator
> implemetaions, which uses different areas may need memory barrier.
>
> >
> > > +}
> > > +
> > > +static int send_smc_message(struct scmi_channel *chan_info,
> > > + scmi_msg_header_t *hdr, void *data, int len)
> > > +{
> > > + struct arm_smccc_res resp;
> > > + int ret;
> > > +
> > > + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> > > + chan_info->shmem->channel_status, len);
> > > + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> > > + hdr->id, hdr->type, hdr->protocol);
> > > +
> > > + ret = channel_is_free(chan_info);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + return ret;
> > > +
> > > + chan_info->shmem->channel_status = 0x0;
> > > + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> > > + chan_info->shmem->flags = 0x0;
> > > + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> > > + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> > > +
> > > + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> > > + chan_info->shmem);
> > > + if ( len > 0 && data )
> > > + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
> >
> > Again, here we don't need a barrier because it is implicit in the SMC?
> >
>
> As I mentioned before, the other end runs on the same CPU.
>
> > Don't we need to check that "len" fits in the shared memory?
> >
>
> I think it's a good point. I'll add len check in v2.
>
> >
> > > + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> > > + &resp);
> > > +
> > > + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> > > +
> > > + if ( resp.a0 )
> > > + return -EOPNOTSUPP;
> >
> > Why is that?
> >
>
> This change was presented in kernel by Sudeep Holla in commit:
> f7199cf489027ae38a9a82312d13025f7aefa0b8
>
> However, link posted in the commit:
> https://lore.kernel.org/r/20200417103232.6896-1-sudeep.holla@arm.com
>
> Leads to slightly different patch:
>
> + if (res.a0 == SMCCC_RET_NOT_SUPPORTED)
> + return -EOPNOTSUPP;
> + else if (res.a0)
> + return -EINVAL;
> + return 0;
>
> I don't know why it differs from the original commit, but I'll check and place
> the correct implementation in v2.
>
> >
> > > + return 0;
> > > +}
> > > +
> > > +static int check_scmi_status(int scmi_status)
> > > +{
> > > + if ( scmi_status == SCMI_SUCCESS )
> > > + return 0;
> > > +
> > > + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> > > +
> > > + switch ( scmi_status )
> > > + {
> > > + case SCMI_NOT_SUPPORTED:
> > > + return -EOPNOTSUPP;
> > > + case SCMI_INVALID_PARAMETERS:
> > > + return -EINVAL;
> > > + case SCMI_DENIED:
> > > + return -EACCES;
> > > + case SCMI_NOT_FOUND:
> > > + return -ENOENT;
> > > + case SCMI_OUT_OF_RANGE:
> > > + return -ERANGE;
> > > + case SCMI_BUSY:
> > > + return -EBUSY;
> > > + case SCMI_COMMS_ERROR:
> > > + return -ENOTCONN;
> > > + case SCMI_GENERIC_ERROR:
> > > + return -EIO;
> > > + case SCMI_HARDWARE_ERROR:
> > > + return -ENXIO;
> > > + case SCMI_PROTOCOL_ERROR:
> > > + return -EBADMSG;
> > > + }
> > > +
> > > + return -EINVAL;
> > > +}
> > > +
> > > +static int get_smc_response(struct scmi_channel *chan_info,
> > > + scmi_msg_header_t *hdr, void *data, int len)
> > > +{
> > > + int recv_len;
> > > + int ret;
> > > +
> > > + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> > > +
> > > + ret = channel_is_free(chan_info);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + return ret;
> >
> > I am not familiar with the spec (do you have a link?) but is it expected
> > that the channel is "free" when actually we want to read a message on
> > the channel?
> >
>
> Here is the link https://developer.arm.com/documentation/den0056/latest
> Figure 6 in Section 5.1.1.
> Caller marks channel as busy, then callee process message and marks channel as free.
> We are implementing polling based communication flow.
OK
> > > + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> > > +
> > > + if ( recv_len < 0 )
> > > + {
> > > + printk(XENLOG_ERR
> > > + "scmi: Wrong size of smc message. Data may be invalid\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if ( recv_len > len )
> > > + {
> > > + printk(XENLOG_ERR
> > > + "scmi: Not enough buffer for message %d, expecting %d\n",
> > > + recv_len, len);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> > > +
> > > + if ( recv_len > 0 )
> > > + {
> > > + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> > > + void *rx_data, int rx_size)
> > > +{
> > > + int ret = 0;
> > > +
> > > + if ( !hdr )
> > > + return -EINVAL;
> > > +
> > > + spin_lock(&channel->lock);
> > > +
> > > + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> > > + if ( ret )
> > > + goto clean;
> > > +
> > > + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> > > +clean:
> > > + spin_unlock(&channel->lock);
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> > > +{
> > > + struct scmi_channel *curr;
> > > + bool found = false;
> > > +
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> >
> > please use parenthesis around the inner if (also in other places)
> >
>
> Thank you for the remark. I will fix it in v2.
>
> >
> > > + if ( curr->chan_id == chan_id )
> > > + {
> > > + found = true;
> > > + break;
> > > + }
> > > +
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > + if ( found )
> > > + return curr;
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
> >
> > Use domid_t for domain ids.
>
> Thanks, I will fix it in v2.
> >
> > Also, wouldn't it be better to implement it as:
> >
> > static inline struct scmi_channel *get_channel_by_domain(struct domain *d) {
> > return d->arch.sci
> > }
> >
> That's a good point. I will take a look on it and fix in v2.
> >
> > > +{
> > > + struct scmi_channel *curr;
> > > + bool found = false;
> > > +
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > > + if ( curr->domain_id == domain_id )
> > > + {
> > > + found = true;
> > > + break;
> > > + }
> > > +
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > + if ( found )
> > > + return curr;
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> > > +{
> > > + struct scmi_channel *curr;
> > > + bool found = false;
> > > +
> > > + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> > > +
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > > + if ( (curr->domain_id == DOMID_INVALID)
> > > + && (curr->chan_id != HYP_CHANNEL) )
> >
> > If you use DOMID_XEN for HYP_CHANNEL, then this check becomes more
> > intuitive
> >
>
> We do not have direct relation between channel id and domain id.
> One channel id can be reused by different domain_ids. So from my standpoint,
> DOMID_XEN doesn't fit here.
Below you wrote that you used DOMID_INVALID to mark a channel as free.
That is fine, but then DOMID_INVALID shouldn't be used for HYP_CHANNEL
because HYP_CHANNEL is not "free". In this function the check for
curr->chan_id != HYP_CHANNEL should be unnecessary and I think the code
would look as follows:
list_for_each_entry(curr, &scmi_data.channel_list, list)
{
if ( (curr->domain_id == DOMID_INVALID) )
{
> > > + {
> > > + curr->domain_id = domain_id;
> > > + found = true;
> > > + break;
> > > + }
> > > +
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > + if ( found )
> > > + return curr;
> > > +
> > > + return NULL;
> > > +}
> > > +
> > > +static void relinquish_scmi_channel(struct scmi_channel *channel)
> > > +{
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + ASSERT(channel != NULL);
> >
> > the ASSERT could be before the spin_lock
> >
>
> Thank you. I will fix it in v2.
>
> >
> > > + channel->domain_id = DOMID_INVALID;
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > +}
> > > +
> > > +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> > > + uint32_t func_id, uint64_t addr)
> > > +{
> > > + struct scmi_channel *channel;
> > > + mfn_t mfn;
> > > +
> > > + channel = get_channel_by_id(chan_id);
> > > + if ( channel )
> > > + return ERR_PTR(EEXIST);
> > > +
> > > + channel = xmalloc(struct scmi_channel);
> > > + if ( !channel )
> > > + return ERR_PTR(ENOMEM);
> > > +
> > > + channel->chan_id = chan_id;
> > > + channel->func_id = func_id;
> > > + channel->domain_id = DOMID_INVALID;
> >
> > I take you are using DOMID_INVALID to mark a channel used by Xen itself?
> > If so, then DOMID_XEN would be more appropriate.
> >
>
> I use DOMID_INVALID to mark channel as free.
>
> >
> > > + mfn = maddr_to_mfn(addr);
> > > + channel->shmem = vmap(&mfn, 1);
> >
> > One thing to be careful is the mapping attributes, for a couple of
> > reasons. As you might be aware, the ARM architecture forbids mismatching
> > attributes for mapping memory in different places in the system. So the
> > attributes that we use here must be the same used by the firmware
> > (and/or the guest.)
> >
> > The second reason to be careful is that in the bindings example
> > Documentation/devicetree/bindings/firmware/arm,scmi.yaml the shared
> > memory is "mmio-sram", which is special. It is not supposed to be normal
> > memory, but it is OK to map it cacheable. Still, it might be more
> > appropriate to use ioremap_cache.
> >
>
> Originally, I used vmap here to have memcpy and it works fine in our
> setup. But I will do some research and email you with the results.
>
> >
> > > + if ( !channel->shmem )
> > > + {
> > > + xfree(channel);
> > > + return ERR_PTR(ENOMEM);
> > > + }
> > > +
> > > + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> > > + channel->paddr = addr;
> > > + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> > > + spin_lock_init(&channel->lock);
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + list_add(&channel->list, &scmi_data.channel_list);
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > + return channel;
> > > +}
> > > +
> > > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > > +{
> > > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > +}
> > > +
> > > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > > + uint64_t len)
> > > +{
> > > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > +}
> > > +
> > > +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> > > + uint64_t size)
> > > +{
> > > + struct dt_device_node *shmem_node;
> > > + __be32 *hw_reg;
> > > + const struct dt_property *pp;
> > > + uint32_t len;
> > > +
> > > + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
> >
> > Here we are using "arm,scmi-shmem" while below we are checking for
> > "linux,scmi_mem". What's the difference?
>
> linux,scmi_mem (I posted nodes examples above) describes memory region, allocated
> for all channels, while arm,scmi-shmem points to the exact channel (page in
> linux,scmi_mem region).
>
>
> > Also, this function is looking for "arm,scmi-shmem" in dt_host and
> > replaces its value. For dom0less domUs we'll probably need a
> > make_scmi_node function to create the node from scratch like for
> > instance xen/arch/arm/domain_build.c:make_gic_domU_node.
> >
> > I wonder if we had such a function whether it wouldn't be better to also
> > use it for dom0 (and blacklist the physical "arm,scmi-shmem" in
> > handle_node so that dom0 doesn't get the real shared memory information
> > by accident).
> >
>
> Thank you for the remark. I will rework this in v2.
>
> >
> > > +
> > > + if ( !shmem_node )
> > > + {
> > > + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + pp = dt_find_property(shmem_node, "reg", &len);
> > > + if ( !pp )
> > > + {
> > > + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> > > + return -ENOENT;
> > > + }
> > > +
> > > + hw_reg = pp->value;
> > > + dt_set_range(&hw_reg, shmem_node, addr, size);
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void free_channel_list(void)
> > > +{
> > > + struct scmi_channel *curr, *_curr;
> > > +
> > > + spin_lock(&scmi_data.channel_list_lock);
> > > + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> > > + {
> > > + vunmap(curr->shmem);
> > > + list_del(&curr->list);
> > > + xfree(curr);
> > > + }
> > > +
> > > + spin_unlock(&scmi_data.channel_list_lock);
> > > +}
> > > +
> > > +static __init bool scmi_probe(struct dt_device_node *scmi_node)
> > > +{
> > > + struct dt_device_node *shmem_node;
> > > + int ret, i;
> > > + struct scmi_channel *channel, *agent_channel;
> > > + int n_agents;
> > > + scmi_msg_header_t hdr;
> > > + struct rx_t {
> > > + int32_t status;
> > > + uint32_t attributes;
> > > + } rx;
> >
> > Should rx be defined at the top together with scmi_perms_tx_t and
> > others?
> >
>
> I'd rather move scmi_perms_tx_t to scmi_add_device_by_devid because it's
> used only in 1 place.
> So we will have rx and tx in scmi_add_device_by_devid and rx ( which
> differs from rx in scmi_add_device_by_devid ) in scmi_probe.
> I think it will be more understandable and no need to make unique names.
> What do you think about that?
I think that's OK
> > > + uint32_t func_id;
> > > +
> > > + ASSERT(scmi_node != NULL);
> > > +
> > > + INIT_LIST_HEAD(&scmi_data.channel_list);
> > > + spin_lock_init(&scmi_data.channel_list_lock);
> > > +
> > > + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> > > + {
> > > + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> > > + return false;
> > > + }
> > > +
> > > + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
> >
> > From the spec, it looks like you should be getting the shared memory
> > area from the phandle list "shmem".
> >
>
> We use SCMI_SHARED_MEMORY to get whole memory region (0x10 pages in my case),
> we can use for the agents. As you can see below - Hypervisor received number of
> agents from Firmware and split this region between agents.
In general we can't use properties that are not part of the device tree
spec, either https://www.devicetree.org/specifications/ or
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings
"linux,scmi_mem" is currently absent. Are you aware of any upstreaming
activities to get "linux,scmi_mem" upstream under
Documentation/devicetree/bindings in Linux?
If "linux,scmi_mem" is going upstream in Linux, then we could use it.
Otherwise, first "linux,scmi_mem" needs to be added somewhere under
Documentation/devicetree/bindings (probably
Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
work on the Xen code that makes use of it.
Does it make sense?
> > > + if ( IS_ERR_OR_NULL(shmem_node) )
> > > + {
> > > + printk(XENLOG_ERR
> > > + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> > > + PTR_ERR(shmem_node));
> > > + return false;
> > > + }
> > > +
> > > + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> > > + &scmi_data.shmem_size);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + return false;
> > > +
> > > + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> > > + if ( IS_ERR(channel) )
> > > + return false;
> > > +
> > > + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> > > + hdr.type = 0;
> > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > +
> > > + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> > > + if ( ret )
> > > + goto clean;
> > > +
> > > + ret = check_scmi_status(rx.status);
> > > + if ( ret )
> > > + goto clean;
> > > +
> > > + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> > > + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> > > +
> > > + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> > > + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> > > +
> > > + for ( i = 1; i < n_agents; i++ )
> > > + {
> >
> > Given that HYP_CHANNEL is actually zero, it looks like we could do
> > everything here in this loop but starting from i=0?
> >
>
> We allocate HYP_CHANNEL before loop because we need it to request number
> of agents. And we don't need to send SCMI_BASE_DISCOVER_AGENT to
> HYP_CHANNEL.
OK
> > > + uint32_t tx_agent_id = 0xFFFFFFFF;
> > > + struct {
> > > + int32_t status;
> > > + uint32_t agent_id;
> > > + char name[16];
> > > + } da_rx;
> > > +
> > > + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> > > + i * PAGE_SIZE);
> > > + if ( IS_ERR(agent_channel) )
> > > + {
> > > + ret = PTR_ERR(agent_channel);
> > > + goto clean;
> > > + }
> > > +
> > > + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> > > + hdr.type = 0;
> > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > +
> > > + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> > > + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> > > + if ( ret )
> > > + goto clean;
> > > +
> > > + ret = check_scmi_status(da_rx.status);
> > > + if ( ret )
> > > + goto clean;
> > > +
> > > + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> > > + da_rx.status, da_rx.agent_id, da_rx.name);
> > > +
> > > + agent_channel->agent_id = da_rx.agent_id;
> > > + }
> > > +
> > > + scmi_data.initialized = true;
> > > + return true;
> > > +
> > > +clean:
> > > + free_channel_list();
> > > + return ret == 0;
> > > +}
> > > +
> > > +static int scmi_domain_init(struct domain *d)
> > > +{
> > > + struct scmi_channel *channel;
> > > + int ret;
> > > +
> > > + if ( !scmi_data.initialized )
> > > + return 0;
> > > +
> > > + channel = aquire_scmi_channel(d->domain_id);
> > > + if ( IS_ERR_OR_NULL(channel) )
> > > + return -ENOENT;
> > > +
> > > + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> > > + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> > > + channel->paddr);
> > > +
> > > + if ( is_hardware_domain(d) )
> > > + {
> > > + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> > > + scmi_data.shmem_size);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + goto error;
> > > +
> > > + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + {
> > > + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> > > + scmi_data.shmem_size);
> > > + if ( rc )
> > > + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> > > +
> > > + goto error;
> > > + }
> > > + }
> >
> > Is dom0 the only domain to get direct access to the shared memory
> > region? If so, I don't think it is a good idea to make Dom0 "special" in
> > this case.
> >
> > Let me make an example: if we assign a device to a domU since boot, and
> > dom0 wants to change the frequency of a clock that affects the assigned
> > device (likely because it doesn't know it is assigned), then dom0
> > shouldn't be able to. We might have to perform checks in Xen to make
> > sure dom0 cannot stop the clock for the assigned device.
> >
> > So I think it would be better if all domains are treated the same way in
> > the mediator unless really necessary.
> >
> > On the other hand, if all domains get access to the shared memory
> > region, then I don't think this is likely the right place to create the
> > dom0 mapping. We probably want to do it in domain_build.c in a way that
> > can be reused for dom0less domUs.
> >
>
> The idea is that all domains have their own page in shared memory
> region and unigue agent_id. Agent_id is used to set permissions for
> clocks\resets\power-domains etc. So during creation of domain (domUs or
> dom0) device-tree is processed using scmi_add_dt_device and
> clocks\resets\power-domains which are related to dom0 devices will be
> requested by using SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> All passed-through devices will be requested during DomU creation. Which
> means dom0 do not have an access to clocks\resets\power-domains, which
> are related to DomU.
OK, excellent
> > In regards to shared memory: it looks like the only two functions to
> > access the real shared memory are send_smc_message and get_smc_response.
> > If that is the case, then we actually don't need to expose the real
> > shared memory to any of the domains.
> >
> > We could simply:
> >
> > - expose a regular normal memory region as dom0/domU channel memory
> > - on SMC trap, read from the "fake" shared memory and set the
> > corresponding real shared memory on the appropriate channel
> > - issue the SMC call
> > - on return from SMC, copy over data from the real shared memory to the
> > "fake" channel reagion
>
> Hypervisor redirects only SMC calls from guests and set agent_id to SMC
> parameters as a7. The idea was to give page for each agent, so we don't
> need to make additional read/write each time we receive SMC call.
> All we povide from hypervisor is agent_id. Firmware is responsible for
> reading memory from the correct address and place the response.
>
> >
> > This is useful if we need to "filter" any of the SCMI commands and
> > options from the domains to the firmware, and also it is useful if the
> > channel memory is not page aligned. But if the permissions are
> > fine-grained enough and also the channel memory is page aligned (and
> > multiple of 4K in size) then we could map the memory.
> >
>
> In current implementation we suppose that channel memory is page aligned.
> I think that Firmware should be responsible for permissions handling and
> "filtering", that's why permission calls were added to SCMI spec.
> I tried to make mediator as thin as possible.
OK. Can we check that the channel memory is page aligned at init time
and throw an error if it is not the case?
> > > +
> > > + d->arch.sci = channel;
> > > +
> > > + return 0;
> > > +error:
> > > + relinquish_scmi_channel(channel);
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> > > +{
> > > + struct scmi_channel *channel, *agent_channel;
> > > + scmi_msg_header_t hdr;
> > > + scmi_perms_tx_t tx;
> > > + struct rx_t {
> > > + int32_t status;
> > > + uint32_t attributes;
> > > + } rx;
> > > + int ret;
> > > +
> > > + if ( !scmi_data.initialized )
> > > + return 0;
> > > +
> > > + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> > > +
> > > + agent_channel = get_channel_by_domain(d->domain_id);
> > > + if ( IS_ERR_OR_NULL(agent_channel) )
> > > + return PTR_ERR(agent_channel);
> > > +
> > > + channel = get_channel_by_id(HYP_CHANNEL);
> > > + if ( IS_ERR_OR_NULL(channel) )
> > > + return PTR_ERR(channel);
> > > +
> > > + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> > > + hdr.type = 0;
> > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > +
> > > + tx.agent_id = agent_channel->agent_id;
> > > + tx.device_id = scmi_devid;
> > > + tx.flags = SCMI_ALLOW_ACCESS;
> > > +
> > > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> > > + if ( IS_ERR_VALUE(ret) )
> > > + return ret;
> > > +
> > > + ret = check_scmi_status(rx.status);
> > > + if ( IS_ERR_VALUE(ret) )
> > > + return ret;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> > > +{
> > > + uint32_t scmi_devid;
> > > +
> > > + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> > > + return 0;
> > > +
> > > + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> > > + return 0;
> >
> > scmi_devid is another property that is not documented in the binding.
> >
>
> This property should be added to the device nodes, which are using scmi
> to work with clocks\resets\power-domains etc. This id should match the
> device_id, defined in Firmware. Hypervisor send this device_id to the Firmware
> as parameter to the permission request. Firmware set permissions to
> clocks\resets\power-domains, related to this device.
OK, I see. Unfortunately, scmi_devid is also not described under
Documentation/devicetree/bindings/.
This property seems to be actually required for the system to work
correctly.
Unless somebody else is already working on this, please send a patch to
the Linux kernel mailing list CCing the SCMI maintainers and Rob Herring
to introduce scmi_devid as a new property.
> > > + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> > > +
> > > + return scmi_add_device_by_devid(d, scmi_devid);
> > > +}
> > > +
> > > +static int scmi_relinquish_resources(struct domain *d)
> > > +{
> > > + int ret;
> > > + struct scmi_channel *channel, *agent_channel;
> > > + scmi_msg_header_t hdr;
> > > + struct reset_agent_tx {
> > > + uint32_t agent_id;
> > > + uint32_t flags;
> > > + } tx;
> > > + uint32_t rx;
> > > +
> > > + if ( !d->arch.sci )
> > > + return 0;
> > > +
> > > + agent_channel = d->arch.sci;
> > > +
> > > + spin_lock(&agent_channel->lock);
> > > + tx.agent_id = agent_channel->agent_id;
> > > + spin_unlock(&agent_channel->lock);
> > > +
> > > + channel = get_channel_by_id(HYP_CHANNEL);
> > > + if ( !channel )
> > > + {
> > > + printk(XENLOG_ERR
> > > + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> > > + d->domain_id);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> > > + hdr.type = 0;
> > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > +
> > > + tx.flags = 0;
> > > +
> > > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> > > + if ( ret )
> > > + return ret;
> > > +
> > > + ret = check_scmi_status(rx);
> > > +
> > > + return ret;
> > > +}
> > > +
> > > +static void scmi_domain_destroy(struct domain *d)
> > > +{
> > > + struct scmi_channel *channel;
> > > +
> > > + if ( !d->arch.sci )
> > > + return;
> > > +
> > > + channel = d->arch.sci;
> > > + spin_lock(&channel->lock);
> > > +
> > > + relinquish_scmi_channel(channel);
> > > + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> > > +
> > > + d->arch.sci = NULL;
> > > +
> > > + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
> > > + spin_unlock(&channel->lock);
> > > + return;
> > > +}
> > > +
> > > +static bool scmi_handle_call(struct domain *d, void *args)
> > > +{
> > > + bool res = false;
> > > + struct scmi_channel *agent_channel;
> > > + struct arm_smccc_res resp;
> > > + struct cpu_user_regs *regs = args;
> > > +
> > > + if ( !d->arch.sci )
> > > + return false;
> > > +
> > > + agent_channel = d->arch.sci;
> > > + spin_lock(&agent_channel->lock);
> > > +
> > > + if ( agent_channel->func_id != regs->x0 )
> > > + {
> > > + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> > > + goto unlock;
> > > + }
> > > +
> > > + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> > > + agent_channel->chan_id, &resp);
> > > +
> > > + set_user_reg(regs, 0, resp.a0);
> > > + set_user_reg(regs, 1, resp.a1);
> > > + set_user_reg(regs, 2, resp.a2);
> > > + set_user_reg(regs, 3, resp.a3);
> > > + res = true;
> > > +unlock:
> > > + spin_unlock(&agent_channel->lock);
> > > +
> > > + return res;
> > > +}
> > > +
> > > +static int scmi_get_channel_paddr(void *scmi_ops,
> > > + struct xen_arch_domainconfig *config)
> > > +{
> > > + struct scmi_channel *agent_channel = scmi_ops;
> > > +
> > > + if ( !agent_channel )
> > > + return -EINVAL;
> > > +
> > > + config->sci_agent_paddr = agent_channel->paddr;
> > > + return 0;
> > > +}
> >
> > I am still not sure why it couldn't be done by scmi_domain_init.
> >
>
> I can move this logic to scmi_domain_init, but in this case I have to add
> struct xen_arch_domainconfig *config as input parameter to
> scmi_domain_init and pass NULL from construct_dom0.
> Do you think this approach would be better?
I think it is OK to pass struct xen_arch_domainconfig *config as input
parameter to scmi_domain_init.
For dom0, why is sci_agent_paddr not supposed to be set?
> Also I think it's reasonable to pass xen_arch_domainconfig so different
> implementations could set another data they would probably need.
>
> >
> > > +static const struct dt_device_match scmi_smc_match[] __initconst =
> > > +{
> > > + DT_MATCH_SCMI_SMC,
> > > + { /* sentinel */ },
> > > +};
> > > +
> > > +static const struct sci_mediator_ops scmi_ops =
> > > +{
> > > + .probe = scmi_probe,
> > > + .domain_init = scmi_domain_init,
> > > + .domain_destroy = scmi_domain_destroy,
> > > + .add_dt_device = scmi_add_dt_device,
> > > + .relinquish_resources = scmi_relinquish_resources,
> > > + .handle_call = scmi_handle_call,
> > > + .get_channel_info = scmi_get_channel_paddr
> > > +};
> > > +
> > > +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> > > + scmi_smc_match, &scmi_ops);
> > > +
> > > +/*
> > > + * Local variables:
> > > + * mode: C
> > > + * c-file-style: "BSD"
> > > + * c-basic-offset: 4
> > > + * indent-tabs-mode: nil
> > > + * End:
> > > + */
> > > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > > index 9180be5e86..a67237942d 100644
> > > --- a/xen/include/public/arch-arm.h
> > > +++ b/xen/include/public/arch-arm.h
> > > @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> > > #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> > >
> > > #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> > > +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
> > >
> > > struct xen_arch_domainconfig {
> > > /* IN/OUT */
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
2021-12-15 21:51 ` Oleksandr
@ 2021-12-21 0:54 ` Stefano Stabellini
2021-12-22 10:24 ` Oleksii Moisieiev
2021-12-21 13:27 ` Anthony PERARD
2 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-21 0:54 UTC (permalink / raw)
To: Oleksii Moisieiev; +Cc: xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> This enumeration sets SCI type for the domain. Currently there is
> two possible options: either 'none' or 'scmi_smc'.
>
> 'none' is the default value and it disables SCI support at all.
>
> 'scmi_smc' enables access to the Firmware from the domains using SCMI
> protocol and SMC as transport.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> tools/include/libxl.h | 5 +++++
> tools/libs/light/libxl_types.idl | 6 ++++++
> tools/xl/xl_parse.c | 9 +++++++++
> 4 files changed, 42 insertions(+)
>
> diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> index b98d161398..92d0593875 100644
> --- a/docs/man/xl.cfg.5.pod.in
> +++ b/docs/man/xl.cfg.5.pod.in
> @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
>
> =back
>
> +=item B<sci="STRING">
> +
> +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> +allows domain to manage various functions that are provided by HW platform.
> +
> +=over 4
> +
> +=item B<none>
> +
> +Don't allow guest to use SCI if present on the platform. This is the default
> +value.
> +
> +=item B<scmi_smc>
> +
> +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> +Inferface - allows domain to manage various functions that are provided by HW
> +platform, such as clocks, resets and power-domains. Xen will mediate access to
> +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> +SMC is used as transport.
Would it make sense to actually enable SCMI-SMC support for the guest by
default if the guest has any devices directly assigned?
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
2021-12-14 9:41 ` Jan Beulich
2021-12-16 0:04 ` Oleksandr
@ 2021-12-21 1:37 ` Stefano Stabellini
2021-12-22 13:41 ` Oleksii Moisieiev
2 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-21 1:37 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Anthony PERARD,
Volodymyr Babchuk, Bertrand Marquis
On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> Integration of the SCMI mediator with xen libs:
> - add hypercalls to aquire SCI channel and set device permissions
> for DomUs;
> - add SCMI_SMC nodes to DomUs device-tree based on partial device-tree;
> - SCI requests redirection from DomUs to Firmware.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
> tools/include/xenctrl.h | 3 +
> tools/include/xenguest.h | 2 +
> tools/libs/ctrl/xc_domain.c | 23 ++++++
> tools/libs/guest/xg_dom_arm.c | 5 +-
> tools/libs/light/libxl_arm.c | 122 +++++++++++++++++++++++++++---
> tools/libs/light/libxl_create.c | 54 ++++++++++++-
> tools/libs/light/libxl_dom.c | 1 +
> tools/libs/light/libxl_internal.h | 4 +
> xen/arch/arm/domctl.c | 15 ++++
> xen/include/public/domctl.h | 9 +++
> 10 files changed, 223 insertions(+), 15 deletions(-)
>
> diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
> index 07b96e6671..cdd14f465f 100644
> --- a/tools/include/xenctrl.h
> +++ b/tools/include/xenctrl.h
> @@ -1238,6 +1238,9 @@ int xc_domain_getvnuma(xc_interface *xch,
> int xc_domain_soft_reset(xc_interface *xch,
> uint32_t domid);
>
> +int xc_domain_add_sci_device(xc_interface *xch,
> + uint32_t domid, char *path);
> +
> #if defined(__i386__) || defined(__x86_64__)
> /*
> * PC BIOS standard E820 types and structure.
> diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
> index 61d0a82f48..35c611ac73 100644
> --- a/tools/include/xenguest.h
> +++ b/tools/include/xenguest.h
> @@ -242,6 +242,8 @@ struct xc_dom_image {
>
> /* Number of vCPUs */
> unsigned int max_vcpus;
> +
> + xen_pfn_t sci_shmem_gfn;
We should be able to avoid introducing sci_shmem_gfn (more below)
> };
>
> /* --- arch specific hooks ----------------------------------------- */
> diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
> index b155d6afd2..07bb390e17 100644
> --- a/tools/libs/ctrl/xc_domain.c
> +++ b/tools/libs/ctrl/xc_domain.c
> @@ -2206,6 +2206,29 @@ int xc_domain_soft_reset(xc_interface *xch,
> domctl.domain = domid;
> return do_domctl(xch, &domctl);
> }
> +
> +int xc_domain_add_sci_device(xc_interface *xch,
> + uint32_t domid, char *path)
> +{
> + size_t size = strlen(path);
> + int rc;
> + DECLARE_DOMCTL;
> + DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
> +
> + if ( xc_hypercall_bounce_pre(xch, path) )
> + return -1;
> +
> + domctl.cmd = XEN_DOMCTL_add_sci_device;
> + domctl.domain = domid;
> + domctl.u.sci_device_op.size = size;
> + set_xen_guest_handle(domctl.u.sci_device_op.path, path);
> + rc = do_domctl(xch, &domctl);
> +
> + xc_hypercall_bounce_post(xch, path);
> +
> + return rc;
> +}
I'd skip this xc_ function and hypercall (more below)
> /*
> * Local variables:
> * mode: C
> diff --git a/tools/libs/guest/xg_dom_arm.c b/tools/libs/guest/xg_dom_arm.c
> index 5e3b76355e..368a670c46 100644
> --- a/tools/libs/guest/xg_dom_arm.c
> +++ b/tools/libs/guest/xg_dom_arm.c
> @@ -25,11 +25,12 @@
>
> #include "xg_private.h"
>
> -#define NR_MAGIC_PAGES 4
> +#define NR_MAGIC_PAGES 5
> #define CONSOLE_PFN_OFFSET 0
> #define XENSTORE_PFN_OFFSET 1
> #define MEMACCESS_PFN_OFFSET 2
> #define VUART_PFN_OFFSET 3
> +#define SCI_SHMEM_OFFSET 4
>
> #define LPAE_SHIFT 9
>
> @@ -69,11 +70,13 @@ static int alloc_magic_pages(struct xc_dom_image *dom)
> dom->console_pfn = base + CONSOLE_PFN_OFFSET;
> dom->xenstore_pfn = base + XENSTORE_PFN_OFFSET;
> dom->vuart_gfn = base + VUART_PFN_OFFSET;
> + dom->sci_shmem_gfn = base + SCI_SHMEM_OFFSET;
>
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->console_pfn);
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->xenstore_pfn);
> xc_clear_domain_page(dom->xch, dom->guest_domid, base + MEMACCESS_PFN_OFFSET);
> xc_clear_domain_page(dom->xch, dom->guest_domid, dom->vuart_gfn);
> + xc_clear_domain_page(dom->xch, dom->guest_domid, dom->sci_shmem_gfn);
Given that sci_shmem_gfn doesn't actually require any allocations, just
a remapping of an existing and already specified physical page, then I
don't think we need to add a new page to alloc_magic_pages for it.
We can just #define a static address for the SCMI page in the domU
addres space and use it for the mapping. No need to allocate a new
page.
> xc_hvm_param_set(dom->xch, dom->guest_domid, HVM_PARAM_CONSOLE_PFN,
> dom->console_pfn);
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index eef1de0939..73ef591822 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -8,6 +8,8 @@
> #include <assert.h>
> #include <xen/device_tree_defs.h>
>
> +#define SCMI_NODE_PATH "/firmware/scmi"
> +
> static const char *gicv_to_string(libxl_gic_version gic_version)
> {
> switch (gic_version) {
> @@ -101,6 +103,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
> return ERROR_FAIL;
> }
>
> + switch (d_config->b_info.sci) {
> + case LIBXL_SCI_TYPE_NONE:
> + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_NONE;
> + break;
> + case LIBXL_SCI_TYPE_SCMI_SMC:
> + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_SCMI_SMC;
> + break;
> + default:
> + LOG(ERROR, "Unknown SCI type %d",
> + d_config->b_info.sci);
> + return ERROR_FAIL;
> + }
> +
> return 0;
> }
>
> @@ -122,6 +137,7 @@ int libxl__arch_domain_save_config(libxl__gc *gc,
> }
>
> state->clock_frequency = config->arch.clock_frequency;
> + state->sci_agent_paddr = config->arch.sci_agent_paddr;
>
> return 0;
> }
> @@ -502,9 +518,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> int res;
> LOG(DEBUG, "Creating OP-TEE node in dtb");
>
> - res = fdt_begin_node(fdt, "firmware");
> - if (res) return res;
> -
> res = fdt_begin_node(fdt, "optee");
> if (res) return res;
>
> @@ -517,9 +530,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> res = fdt_end_node(fdt);
> if (res) return res;
>
> - res = fdt_end_node(fdt);
> - if (res) return res;
> -
> return 0;
> }
>
> @@ -902,10 +912,9 @@ static int copy_node(libxl__gc *gc, void *fdt, void *pfdt,
> return 0;
> }
>
> -static int copy_node_by_path(libxl__gc *gc, const char *path,
> - void *fdt, void *pfdt)
> +static int get_path_nodeoff(const char *path, void *pfdt)
> {
> - int nodeoff, r;
> + int nodeoff;
> const char *name = strrchr(path, '/');
>
> if (!name)
> @@ -925,12 +934,101 @@ static int copy_node_by_path(libxl__gc *gc, const char *path,
> if (strcmp(fdt_get_name(pfdt, nodeoff, NULL), name))
> return -FDT_ERR_NOTFOUND;
>
> + return nodeoff;
> +}
> +
> +static int copy_node_by_path(libxl__gc *gc, const char *path,
> + void *fdt, void *pfdt)
> +{
> + int nodeoff, r;
> +
> + nodeoff = get_path_nodeoff(path, pfdt);
> + if (nodeoff < 0)
> + return nodeoff;
> +
> r = copy_node(gc, fdt, pfdt, nodeoff, 0);
> if (r) return r;
>
> return 0;
> }
>
> +static int get_node_phandle(const char *path, void *pfdt, uint32_t *phandle)
> +{
> + int nodeoff;
> + const char *name = strrchr(path, '/');
> +
> + if (!name)
> + return -FDT_ERR_INTERNAL;
> +
> + name++;
> + nodeoff = fdt_path_offset(pfdt, path);
> + if (nodeoff < 0)
> + return nodeoff;
> +
> + *phandle = fdt_get_phandle(pfdt, nodeoff);
> + return 0;
> +}
> +
> +static int make_scmi_shmem_node(libxl__gc *gc, void *fdt, void *pfdt,
> + struct xc_dom_image *dom)
> +{
> + int res;
> + char buf[64];
> + uint32_t phandle = 0;
> +
> + res = get_node_phandle("/scp-shmem", pfdt, &phandle);
> + if (res) return res;
I hope we'll be able to avoid requiring the user to write a partial
device tree with SCMI info in order to use it.
But if we have to go down this route, then we need to add a description
of scp-shmem under docs/misc/arm/
Also, in general, we cannot mandate or require specific paths in device
tree and instead we should find nodes based on the compatible string.
(There are exceptions like /reserved-memory and /firmware but they are
only a couple.)
> + snprintf(buf, sizeof(buf), "scp-shmem@%lx",
> + dom->sci_shmem_gfn << XC_PAGE_SHIFT);
> + res = fdt_begin_node(fdt, buf);
> + if (res) return res;
> +
> + res = fdt_property_compat(gc, fdt, 1, "arm,scmi-shmem");
> + if (res) return res;
> +
> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> + GUEST_ROOT_SIZE_CELLS, 1,
> + dom->sci_shmem_gfn << XC_PAGE_SHIFT, XC_PAGE_SHIFT);
> + if (res) return res;
> +
> + LOG(DEBUG, "scmi: setting phandle = %u\n", phandle);
> + res = fdt_property_cell(fdt, "phandle", phandle);
> + if (res) return res;
> +
> + res = fdt_end_node(fdt);
> + if (res) return res;
> +
> + return 0;
> +}
> +
> +static int make_firmware_node(libxl__gc *gc, void *fdt, void *pfdt, int tee,
> + int sci)
> +{
> + int res;
> +
> + if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci != LIBXL_SCI_TYPE_NONE))
> + return 0;
Shouldn't this be:
if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci == LIBXL_SCI_TYPE_NONE))
> + res = fdt_begin_node(fdt, "firmware");
> + if (res) return res;
> +
> + if (tee == LIBXL_TEE_TYPE_OPTEE) {
> + res = make_optee_node(gc, fdt);
> + if (res) return res;
> + }
> +
> + if (sci == LIBXL_SCI_TYPE_SCMI_SMC) {
> + res = copy_node_by_path(gc, SCMI_NODE_PATH, fdt, pfdt);
> + if (res) return res;
> + }
Do we really need to copy the node from the partial device tree? That
makes it a lot harder to use for the user. Ideally a user would only
need to specify sci = "scmi_smc" and everything else should be done
automatically.
Can we automatically generate the SCMI device tree node instead? It
looks like we should have all the information to be able to do it. If
not, what is missing?
> + res = fdt_end_node(fdt);
> + if (res) return res;
> +
> + return 0;
> +}
> +
> /*
> * The partial device tree is not copied entirely. Only the relevant bits are
> * copied to the guest device tree:
> @@ -1088,8 +1186,10 @@ next_resize:
> if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART)
> FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) );
>
> - if (info->tee == LIBXL_TEE_TYPE_OPTEE)
> - FDT( make_optee_node(gc, fdt) );
> + FDT( make_firmware_node(gc, fdt, pfdt, info->tee, info->sci) );
> +
> + if (info->sci == LIBXL_SCI_TYPE_SCMI_SMC)
> + FDT( make_scmi_shmem_node(gc, fdt, pfdt, dom) );
>
> if (d_config->num_pcidevs)
> FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
> index dcd09d32ba..c7372fd344 100644
> --- a/tools/libs/light/libxl_create.c
> +++ b/tools/libs/light/libxl_create.c
> @@ -596,6 +596,37 @@ out:
> return ret;
> }
>
> +static int map_sci_page(libxl__gc *gc, uint32_t domid, uint64_t paddr,
> + uint64_t guest_addr)
> +{
> + int ret;
> + uint64_t _paddr_pfn = paddr >> XC_PAGE_SHIFT;
> + uint64_t _guest_pfn = guest_addr >> XC_PAGE_SHIFT;
> +
> + LOG(DEBUG, "iomem %"PRIx64, _paddr_pfn);
> +
> + ret = xc_domain_iomem_permission(CTX->xch, domid, _paddr_pfn, 1, 1);
> + if (ret < 0) {
> + LOG(ERROR,
> + "failed give domain access to iomem page %"PRIx64,
> + _paddr_pfn);
> + return ret;
> + }
> +
> + ret = xc_domain_memory_mapping(CTX->xch, domid,
> + _guest_pfn, _paddr_pfn,
> + 1, 1);
> + if (ret < 0) {
> + LOG(ERROR,
> + "failed to map to domain iomem page %"PRIx64
> + " to guest address %"PRIx64,
> + _paddr_pfn, _guest_pfn);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> libxl__domain_build_state *state,
> uint32_t *domid, bool soft_reset)
> @@ -762,6 +793,16 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> goto out;
> }
>
> + if (state->sci_agent_paddr != 0) {
Shouldn't we also check for sci_type == XEN_DOMCTL_CONFIG_SCI_NONE ?
If the user specifies sci_type == XEN_DOMCTL_CONFIG_SCI_SCMI_SMC, we
shouldn't automatically map any SCMI channels to the guest, right?
> + ret = map_sci_page(gc, *domid, state->sci_agent_paddr,
> + state->sci_shmem_gfn << XC_PAGE_SHIFT);
> + if (ret < 0) {
> + LOGED(ERROR, *domid, "map SCI page fail");
> + rc = ERROR_FAIL;
> + goto out;
> + }
> + }
> +
> dom_path = libxl__xs_get_dompath(gc, *domid);
> if (!dom_path) {
> rc = ERROR_FAIL;
> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> {
> AO_GC;
> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> - int i, rc = 0;
> + int i, rc = 0, rc_sci = 0;
>
> for (i = 0; i < d_config->num_dtdevs; i++) {
> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>
> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> - if (rc < 0) {
> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> +
> + if ((rc < 0) && (rc_sci < 0)) {
> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> + "xc_domain_add_sci_device failed: %d",
> + rc, rc_sci);
> goto out;
> }
> +
> + if (rc)
> + rc = rc_sci;
I would make this simpler actually. Do we need to pass dtdev->path
twice, once for xc_assign_dt_device and a second time for
xc_domain_add_sci_device?
I would skip adding xc_domain_add_sci_device altogether. A general SCMI
enable/disable for the domain is necessary, but then we can just get the
directly assigned devices from xc_assign_dt_device. There is no need to
specify the list twice. If a device is not manageable via SCMI we can
detect it automatically because it is going to be missing scmi_devid on
device tree.
> }
>
> out:
> diff --git a/tools/libs/light/libxl_dom.c b/tools/libs/light/libxl_dom.c
> index fe9f760f71..b1d288a8b9 100644
> --- a/tools/libs/light/libxl_dom.c
> +++ b/tools/libs/light/libxl_dom.c
> @@ -710,6 +710,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
> state->console_mfn = dom->console_pfn;
> state->store_mfn = dom->xenstore_pfn;
> state->vuart_gfn = dom->vuart_gfn;
> + state->sci_shmem_gfn = dom->sci_shmem_gfn;
> } else {
> state->console_mfn = xc_dom_p2m(dom, dom->console_pfn);
> state->store_mfn = xc_dom_p2m(dom, dom->xenstore_pfn);
> diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
> index 0b4671318c..f9f9cc633a 100644
> --- a/tools/libs/light/libxl_internal.h
> +++ b/tools/libs/light/libxl_internal.h
> @@ -1407,6 +1407,10 @@ typedef struct {
> /* Whether this domain is being migrated/restored, or booting fresh. Only
> * applicable to the primary domain, not support domains (e.g. stub QEMU). */
> bool restore;
> +
> + /* sci channel paddr to be set to device-tree node */
> + uint64_t sci_agent_paddr;
> + xen_pfn_t sci_shmem_gfn;
> } libxl__domain_build_state;
>
> _hidden void libxl__domain_build_state_init(libxl__domain_build_state *s);
> diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
> index 6245af6d0b..ba200407da 100644
> --- a/xen/arch/arm/domctl.c
> +++ b/xen/arch/arm/domctl.c
> @@ -4,6 +4,7 @@
> * Copyright (c) 2012, Citrix Systems
> */
>
> +#include <asm/sci/sci.h>
> #include <xen/errno.h>
> #include <xen/guest_access.h>
> #include <xen/hypercall.h>
> @@ -175,6 +176,20 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
>
> return rc;
> }
> + case XEN_DOMCTL_add_sci_device:
> + {
> + int rc;
> + struct dt_device_node *dev;
> +
> + rc = dt_find_node_by_gpath(domctl->u.sci_device_op.path,
> + domctl->u.sci_device_op.size,
> + &dev);
> + if ( rc )
> + return rc;
> +
> + return sci_add_dt_device(d, dev);
> + }
I would skip it and instead I would add a call to sci_add_dt_device in
the implementation of XEN_DOMCTL_assign_device.
> default:
> {
> int rc;
> diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> index b85e6170b0..671c72c3e8 100644
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> #define XEN_DOMCTL_vmtrace_get_option 5
> #define XEN_DOMCTL_vmtrace_set_option 6
> };
> +
> +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> +struct xen_domctl_sci_device_op {
> + uint32_t size; /* Length of the path */
> + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> +};
> +
> typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
> DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
>
> @@ -1265,6 +1272,7 @@ struct xen_domctl {
> #define XEN_DOMCTL_get_cpu_policy 82
> #define XEN_DOMCTL_set_cpu_policy 83
> #define XEN_DOMCTL_vmtrace_op 84
> +#define XEN_DOMCTL_add_sci_device 85
> #define XEN_DOMCTL_gdbsx_guestmemio 1000
> #define XEN_DOMCTL_gdbsx_pausevcpu 1001
> #define XEN_DOMCTL_gdbsx_unpausevcpu 1002
> @@ -1326,6 +1334,7 @@ struct xen_domctl {
> struct xen_domctl_psr_alloc psr_alloc;
> struct xen_domctl_vuart_op vuart_op;
> struct xen_domctl_vmtrace_op vmtrace_op;
> + struct xen_domctl_sci_device_op sci_device_op;
> uint8_t pad[128];
> } u;
> };
> --
> 2.27.0
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
2021-12-15 21:51 ` Oleksandr
2021-12-21 0:54 ` Stefano Stabellini
@ 2021-12-21 13:27 ` Anthony PERARD
2021-12-22 12:20 ` Oleksii Moisieiev
2 siblings, 1 reply; 95+ messages in thread
From: Anthony PERARD @ 2021-12-21 13:27 UTC (permalink / raw)
To: Oleksii Moisieiev; +Cc: xen-devel, Wei Liu, Juergen Gross
On Tue, Dec 14, 2021 at 09:34:28AM +0000, Oleksii Moisieiev wrote:
> This enumeration sets SCI type for the domain. Currently there is
> two possible options: either 'none' or 'scmi_smc'.
>
> 'none' is the default value and it disables SCI support at all.
>
> 'scmi_smc' enables access to the Firmware from the domains using SCMI
> protocol and SMC as transport.
>
> Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> ---
Thanks for the patch, it looks good too me.
But it is kind of weird that the manual describes something that isn't
implemented yet. Could you maybe add in the patch description that the
feature isn't implemented yet or that the feature is implemented in
follow-up patches?
Also, about the golang binding thingy, could you add a note after a line
of a three dash "---" that let know the committer to regenerate
everything that needs re-generating due to change in the *.idl file,
just in case?
Thanks,
--
Anthony PERARD
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-17 12:15 ` Oleksii Moisieiev
@ 2021-12-21 14:45 ` Anthony PERARD
2021-12-21 21:39 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Anthony PERARD @ 2021-12-21 14:45 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Wei Liu, Juergen Gross, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Volodymyr Babchuk, Bertrand Marquis
On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
> > On 14.12.21 11:34, Oleksii Moisieiev wrote:
> > > @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> > > {
> > > AO_GC;
> > > libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> > > - int i, rc = 0;
> > > + int i, rc = 0, rc_sci = 0;
> > > for (i = 0; i < d_config->num_dtdevs; i++) {
> > > const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
> > > LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> > > rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> > > - if (rc < 0) {
> > > - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> > > + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> > > +
> > > + if ((rc < 0) && (rc_sci < 0)) {
> > > + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> > > + "xc_domain_add_sci_device failed: %d",
> > > + rc, rc_sci);
> > > goto out;
> > > }
> > > +
> > > + if (rc)
> > > + rc = rc_sci;
> >
> >
> > If I get this code right, it sounds like the dom.cfg's dtdev property is
> > reused to describe sci devices as well, but the libxl__add_dtdevs() does not
> > (and can not) differentiate them. So it has no option but to send two
> > domctls for each device in dtdevs[] hoping to at least one domctl to
> > succeeded. Or I really missed something?
> >
> > It feels to me that:
> > - either new dom.cfg's property could be introduced (scidev?) to describe
> > sci devices together with new parsing logic/management code, so you will end
> > up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
> > so no mixing things.
> > - or indeed dtdev logic could be *completely* reused including extending
> > XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
> > is used to describe devices for the passthrough (to configure an IOMMU for
> > the device), in that case you wouldn't need an extra
> > XEN_DOMCTL_add_sci_device introduced by current patch.
> >
> > Personally I would use the first option as I am not sure that second option
> > is conceptually correct && possible. I would leave this for the maintainers
> > to clarify.
> >
>
> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
> seems not to be conceptually correct. Introducing new dom.cfg property
> seems to be the only way to avoid extra Domctl calls.
> I will handle this in v2 if there will be no complains from Maintainers.
I don't know if there will be a complains in v2 or not :-), but using
something different from dtdev sound good.
If I understand correctly, dtdev seems to be about passing through an
existing device to a guest, and this new sci device seems to be about having Xen
"emulating" an sci device which the guest will use. So introducing
something new (scidev or other name) sounds good.
Thanks,
--
Anthony PERARD
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-21 0:52 ` Stefano Stabellini
@ 2021-12-21 20:03 ` Oleksii Moisieiev
2021-12-21 21:22 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-21 20:03 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > Hi Stefano,
> >
> > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > This allows devices from the Domains to work with clocks, resets and
> > > > power-domains without access to CPG.
> > > >
> > > > The following features are implemented:
> > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > - set device permissions for Domains based on the Domain partial
> > > > device-tree. Devices with permissions are able to work with clocks,
> > > > resets and power-domains via SCMI;
> > > > - redirect scmi messages from Domains to ATF.
> > > >
> > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > ---
> > > > xen/arch/arm/Kconfig | 2 +
> > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > xen/arch/arm/sci/Makefile | 1 +
> > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > xen/include/public/arch-arm.h | 1 +
> > > > 5 files changed, 809 insertions(+)
> > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > >
> > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > index 186e1db389..02d96c6cfc 100644
> > > > --- a/xen/arch/arm/Kconfig
> > > > +++ b/xen/arch/arm/Kconfig
> > > > @@ -114,6 +114,8 @@ config SCI
> > > > support. It allows guests to control system resourcess via one of
> > > > SCI mediators implemented in XEN.
> > > >
> > > > +source "arch/arm/sci/Kconfig"
> > > > +
> > > > endmenu
> > > >
> > > > menu "ARM errata workaround via the alternative framework"
> > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > new file mode 100644
> > > > index 0000000000..9563067ddc
> > > > --- /dev/null
> > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > @@ -0,0 +1,10 @@
> > > > +config SCMI_SMC
> > > > + bool "Enable SCMI-SMC mediator driver"
> > > > + default n
> > > > + depends on SCI
> > > > + ---help---
> > > > +
> > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > + This feature allows drivers from Domains to work with System
> > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > + for communication.
> > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > index 837dc7492b..67f2611872 100644
> > > > --- a/xen/arch/arm/sci/Makefile
> > > > +++ b/xen/arch/arm/sci/Makefile
> > > > @@ -1 +1,2 @@
> > > > obj-y += sci.o
> > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > new file mode 100644
> > > > index 0000000000..2eb01ea82d
> > > > --- /dev/null
> > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > @@ -0,0 +1,795 @@
> > > > +/*
> > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > + *
> > > > + * SCMI mediator driver, using SCP as transport.
> > > > + *
> > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > + * Copyright (C) 2021, EPAM Systems.
> > > > + *
> > > > + * This program is free software; you can redistribute it and/or modify
> > > > + * it under the terms of the GNU General Public License as published by
> > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > + * (at your option) any later version.
> > > > + *
> > > > + * This program is distributed in the hope that it will be useful,
> > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > + * GNU General Public License for more details.
> > > > + */
> > > > +
> > > > +#include <asm/sci/sci.h>
> > > > +#include <asm/smccc.h>
> > > > +#include <asm/io.h>
> > > > +#include <xen/bitops.h>
> > > > +#include <xen/config.h>
> > > > +#include <xen/sched.h>
> > > > +#include <xen/device_tree.h>
> > > > +#include <xen/iocap.h>
> > > > +#include <xen/init.h>
> > > > +#include <xen/err.h>
> > > > +#include <xen/lib.h>
> > > > +#include <xen/list.h>
> > > > +#include <xen/mm.h>
> > > > +#include <xen/string.h>
> > > > +#include <xen/time.h>
> > > > +#include <xen/vmap.h>
> > > > +
> > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > +
> > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > +#define SCMI_SUCCESS 0
> > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > +#define SCMI_DENIED (-3)
> > > > +#define SCMI_NOT_FOUND (-4)
> > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > +#define SCMI_BUSY (-6)
> > > > +#define SCMI_COMMS_ERROR (-7)
> > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > +
> > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > +
> > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > >
> > > I could find the following SCMI binding in Linux, which describes
> > > the arm,scmi-smc compatible and the arm,smc-id property:
> > >
> > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > >
> > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > the "shmem" property instead? And the compatible string used for this
> > > seems to be "arm,scmi-shmem".
> > >
> >
> > We use linux,scmi_mem node to reserve memory, needed for all
> > channels:
> >
> > reserved-memory {
> > /* reserved region for scmi channels*/
> > scmi_memory: linux,scmi_mem@53FF0000 {
> > no-map;
> > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > };
> > };
> >
> > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > the current scmi channel:
> >
> > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > };
> >
> > For each Domain reg points to unigue page from linux,scmi_mem region,
> > assigned to this agent.
>
> If we were to use "linux,scmi_mem" we would have to introduce it as a
> compatible string, not as a node name, and it would need to be described
> in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
>
> But from your description I don't think it is necessary. We can just use
> "arm,scmi-shmem" to describe all the required regions:
>
> reserved-memory {
> scp-shmem@0x53FF0000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF0000 0x0 0x1000>;
> };
> scp-shmem@0x53FF1000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF1000 0x0 0x1000>;
> };
> scp-shmem@0x53FF2000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF2000 0x0 0x1000>;
> };
> ...
>
> In other words, if all the individual channel pages are described as
> "arm,scmi-shmem", why do we also need a single larger region as
> "linux,scmi_mem"?
>
That was my first implementation. But I've met a problem with
scmi driver in kernel. I don't remember the exact place, but I remember
there were some if, checking if memory weren't reserved.
That's why I ended up splitting nodes reserved memory region and actual
shmem page.
For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
which has no compatible string and provides no-map property.
linux,scmi_shmem node is needed to prevent xen from allocating this
space for the domain.
Very interesting question about should I introduce linux,scmi_mem node
and scmi_devid property to the
Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
Those node and property are needed only for Xen and useless for
non-virtualized systems. I can add this node and property description to
arm,scmi.yaml, but leave a note that this is Xen specific params.
What do you think about it?
>
> > > > +#define SCMI_SHMEM "shmem"
> > > > +
> > > > +#define HYP_CHANNEL 0x0
> > > > +
> > > > +#define HDR_ID GENMASK(7,0)
> > > > +#define HDR_TYPE GENMASK(9, 8)
> > > > +#define HDR_PROTO GENMASK(17, 10)
> > > > +
> > > > +/* SCMI protocol, refer to section 4.2.2.2 (DEN0056C) */
> > > > +#define MSG_N_AGENTS_MASK GENMASK(15, 8)
> > > > +
> > > > +#define FIELD_GET(_mask, _reg)\
> > > > + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> > > > +#define FIELD_PREP(_mask, _val)\
> > > > + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
> > > > +
> > > > +typedef struct scmi_msg_header {
> > > > + uint8_t id;
> > > > + uint8_t type;
> > > > + uint8_t protocol;
> > > > +} scmi_msg_header_t;
> > > > +
> > > > +typedef struct scmi_perms_tx {
> > > > + uint32_t agent_id;
> > > > + uint32_t device_id;
> > > > + uint32_t flags;
> > > > +} scmi_perms_tx_t;
> > > > +
> > > > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE BIT(0, UL)
> > > > +#define SCMI_SHMEM_CHAN_STAT_CHANNEL_ERROR BIT(1, UL)
> > > > +
> > > > +#define SCMI_ALLOW_ACCESS BIT(0, UL)
> > > > +
> > > > +struct scmi_shared_mem {
> > > > + uint32_t reserved;
> > > > + uint32_t channel_status;
> > > > + uint32_t reserved1[2];
> > > > + uint32_t flags;
> > > > + uint32_t length;
> > > > + uint32_t msg_header;
> > > > + uint8_t msg_payload[];
> > > > +};
> > > > +
> > > > +struct scmi_channel {
> > > > + int chan_id;
> > > > + int agent_id;
> > > > + uint32_t func_id;
> > > > + int domain_id;
> > > > + uint64_t paddr;
> > > > + struct scmi_shared_mem *shmem;
> > > > + spinlock_t lock;
> > > > + struct list_head list;
> > > > +};
> > > > +
> > > > +struct scmi_data {
> > > > + struct list_head channel_list;
> > > > + spinlock_t channel_list_lock;
> > > > + bool initialized;
> > > > + u64 shmem_addr, shmem_size;
> > > > +};
> > > > +
> > > > +static struct scmi_data scmi_data;
> > > > +
> > > > +/*
> > > > + * pack_scmi_header() - packs and returns 32-bit header
> > > > + *
> > > > + * @hdr: pointer to header containing all the information on message id,
> > > > + * protocol id and type id.
> > > > + *
> > > > + * Return: 32-bit packed message header to be sent to the platform.
> > > > + */
> > > > +static inline uint32_t pack_scmi_header(scmi_msg_header_t *hdr)
> > > > +{
> > > > + return FIELD_PREP(HDR_ID, hdr->id) |
> > > > + FIELD_PREP(HDR_TYPE, hdr->type) |
> > > > + FIELD_PREP(HDR_PROTO, hdr->protocol);
> > > > +}
> > > > +
> > > > +/*
> > > > + * unpack_scmi_header() - unpacks and records message and protocol id
> > > > + *
> > > > + * @msg_hdr: 32-bit packed message header sent from the platform
> > > > + * @hdr: pointer to header to fetch message and protocol id.
> > > > + */
> > > > +static inline void unpack_scmi_header(uint32_t msg_hdr, scmi_msg_header_t *hdr)
> > > > +{
> > > > + hdr->id = FIELD_GET(HDR_ID, msg_hdr);
> > > > + hdr->type = FIELD_GET(HDR_TYPE, msg_hdr);
> > > > + hdr->protocol = FIELD_GET(HDR_PROTO, msg_hdr);
> > > > +}
> > > > +
> > > > +static inline int channel_is_free(struct scmi_channel *chan_info)
> > > > +{
> > > > + return ( chan_info->shmem->channel_status
> > > > + & SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE ) ? 0 : -EBUSY;
> > >
> > > Does this need a memory barrier? Or not, because the other end always
> > > runs on the same CPU at a different execution level so the
> > > channel_status would be always guaranteed to be read as updated?
> > >
> >
> > It don't because the other end runs on the same CPU. Other mediator
> > implemetaions, which uses different areas may need memory barrier.
> >
> > >
> > > > +}
> > > > +
> > > > +static int send_smc_message(struct scmi_channel *chan_info,
> > > > + scmi_msg_header_t *hdr, void *data, int len)
> > > > +{
> > > > + struct arm_smccc_res resp;
> > > > + int ret;
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: status =%d len=%d\n",
> > > > + chan_info->shmem->channel_status, len);
> > > > + printk(XENLOG_DEBUG "scmi: header id = %d type = %d, proto = %d\n",
> > > > + hdr->id, hdr->type, hdr->protocol);
> > > > +
> > > > + ret = channel_is_free(chan_info);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + return ret;
> > > > +
> > > > + chan_info->shmem->channel_status = 0x0;
> > > > + /* Writing 0x0 right now, but SCMI_SHMEM_FLAG_INTR_ENABLED can be set */
> > > > + chan_info->shmem->flags = 0x0;
> > > > + chan_info->shmem->length = sizeof(chan_info->shmem->msg_header) + len;
> > > > + chan_info->shmem->msg_header = pack_scmi_header(hdr);
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: Writing to shmem address %p\n",
> > > > + chan_info->shmem);
> > > > + if ( len > 0 && data )
> > > > + memcpy((void *)(chan_info->shmem->msg_payload), data, len);
> > >
> > > Again, here we don't need a barrier because it is implicit in the SMC?
> > >
> >
> > As I mentioned before, the other end runs on the same CPU.
> >
> > > Don't we need to check that "len" fits in the shared memory?
> > >
> >
> > I think it's a good point. I'll add len check in v2.
> >
> > >
> > > > + arm_smccc_smc(chan_info->func_id, 0, 0, 0, 0, 0, 0, chan_info->chan_id,
> > > > + &resp);
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: scmccc_smc response %d\n", (int)(resp.a0));
> > > > +
> > > > + if ( resp.a0 )
> > > > + return -EOPNOTSUPP;
> > >
> > > Why is that?
> > >
> >
> > This change was presented in kernel by Sudeep Holla in commit:
> > f7199cf489027ae38a9a82312d13025f7aefa0b8
> >
> > However, link posted in the commit:
> > https://urldefense.com/v3/__https://lore.kernel.org/r/20200417103232.6896-1-sudeep.holla@arm.com__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7Bnhb7jXoB3$ [lore[.]kernel[.]org]
> >
> > Leads to slightly different patch:
> >
> > + if (res.a0 == SMCCC_RET_NOT_SUPPORTED)
> > + return -EOPNOTSUPP;
> > + else if (res.a0)
> > + return -EINVAL;
> > + return 0;
> >
> > I don't know why it differs from the original commit, but I'll check and place
> > the correct implementation in v2.
> >
> > >
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int check_scmi_status(int scmi_status)
> > > > +{
> > > > + if ( scmi_status == SCMI_SUCCESS )
> > > > + return 0;
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: Error received: %d\n", scmi_status);
> > > > +
> > > > + switch ( scmi_status )
> > > > + {
> > > > + case SCMI_NOT_SUPPORTED:
> > > > + return -EOPNOTSUPP;
> > > > + case SCMI_INVALID_PARAMETERS:
> > > > + return -EINVAL;
> > > > + case SCMI_DENIED:
> > > > + return -EACCES;
> > > > + case SCMI_NOT_FOUND:
> > > > + return -ENOENT;
> > > > + case SCMI_OUT_OF_RANGE:
> > > > + return -ERANGE;
> > > > + case SCMI_BUSY:
> > > > + return -EBUSY;
> > > > + case SCMI_COMMS_ERROR:
> > > > + return -ENOTCONN;
> > > > + case SCMI_GENERIC_ERROR:
> > > > + return -EIO;
> > > > + case SCMI_HARDWARE_ERROR:
> > > > + return -ENXIO;
> > > > + case SCMI_PROTOCOL_ERROR:
> > > > + return -EBADMSG;
> > > > + }
> > > > +
> > > > + return -EINVAL;
> > > > +}
> > > > +
> > > > +static int get_smc_response(struct scmi_channel *chan_info,
> > > > + scmi_msg_header_t *hdr, void *data, int len)
> > > > +{
> > > > + int recv_len;
> > > > + int ret;
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: get smc responce msgid %d\n", hdr->id);
> > > > +
> > > > + ret = channel_is_free(chan_info);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + return ret;
> > >
> > > I am not familiar with the spec (do you have a link?) but is it expected
> > > that the channel is "free" when actually we want to read a message on
> > > the channel?
> > >
> >
> > Here is the link https://urldefense.com/v3/__https://developer.arm.com/documentation/den0056/latest__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhRd1Yyp8$ [developer[.]arm[.]com]
> > Figure 6 in Section 5.1.1.
> > Caller marks channel as busy, then callee process message and marks channel as free.
> > We are implementing polling based communication flow.
>
> OK
>
>
> > > > + recv_len = chan_info->shmem->length - sizeof(chan_info->shmem->msg_header);
> > > > +
> > > > + if ( recv_len < 0 )
> > > > + {
> > > > + printk(XENLOG_ERR
> > > > + "scmi: Wrong size of smc message. Data may be invalid\n");
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + if ( recv_len > len )
> > > > + {
> > > > + printk(XENLOG_ERR
> > > > + "scmi: Not enough buffer for message %d, expecting %d\n",
> > > > + recv_len, len);
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + unpack_scmi_header(chan_info->shmem->msg_header, hdr);
> > > > +
> > > > + if ( recv_len > 0 )
> > > > + {
> > > > + memcpy(data, chan_info->shmem->msg_payload, recv_len);
> > > > + }
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int do_smc_xfer(struct scmi_channel *channel, scmi_msg_header_t *hdr, void *tx_data, int tx_size,
> > > > + void *rx_data, int rx_size)
> > > > +{
> > > > + int ret = 0;
> > > > +
> > > > + if ( !hdr )
> > > > + return -EINVAL;
> > > > +
> > > > + spin_lock(&channel->lock);
> > > > +
> > > > + ret = send_smc_message(channel, hdr, tx_data, tx_size);
> > > > + if ( ret )
> > > > + goto clean;
> > > > +
> > > > + ret = get_smc_response(channel, hdr, rx_data, rx_size);
> > > > +clean:
> > > > + spin_unlock(&channel->lock);
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static struct scmi_channel *get_channel_by_id(uint8_t chan_id)
> > > > +{
> > > > + struct scmi_channel *curr;
> > > > + bool found = false;
> > > > +
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > >
> > > please use parenthesis around the inner if (also in other places)
> > >
> >
> > Thank you for the remark. I will fix it in v2.
> >
> > >
> > > > + if ( curr->chan_id == chan_id )
> > > > + {
> > > > + found = true;
> > > > + break;
> > > > + }
> > > > +
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > + if ( found )
> > > > + return curr;
> > > > +
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static struct scmi_channel *get_channel_by_domain(uint8_t domain_id)
> > >
> > > Use domid_t for domain ids.
> >
> > Thanks, I will fix it in v2.
> > >
> > > Also, wouldn't it be better to implement it as:
> > >
> > > static inline struct scmi_channel *get_channel_by_domain(struct domain *d) {
> > > return d->arch.sci
> > > }
> > >
> > That's a good point. I will take a look on it and fix in v2.
> > >
> > > > +{
> > > > + struct scmi_channel *curr;
> > > > + bool found = false;
> > > > +
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > > > + if ( curr->domain_id == domain_id )
> > > > + {
> > > > + found = true;
> > > > + break;
> > > > + }
> > > > +
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > + if ( found )
> > > > + return curr;
> > > > +
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static struct scmi_channel *aquire_scmi_channel(int domain_id)
> > > > +{
> > > > + struct scmi_channel *curr;
> > > > + bool found = false;
> > > > +
> > > > + ASSERT(domain_id != DOMID_INVALID && domain_id >= 0);
> > > > +
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + list_for_each_entry(curr, &scmi_data.channel_list, list)
> > > > + if ( (curr->domain_id == DOMID_INVALID)
> > > > + && (curr->chan_id != HYP_CHANNEL) )
> > >
> > > If you use DOMID_XEN for HYP_CHANNEL, then this check becomes more
> > > intuitive
> > >
> >
> > We do not have direct relation between channel id and domain id.
> > One channel id can be reused by different domain_ids. So from my standpoint,
> > DOMID_XEN doesn't fit here.
>
> Below you wrote that you used DOMID_INVALID to mark a channel as free.
> That is fine, but then DOMID_INVALID shouldn't be used for HYP_CHANNEL
> because HYP_CHANNEL is not "free". In this function the check for
> curr->chan_id != HYP_CHANNEL should be unnecessary and I think the code
> would look as follows:
>
> list_for_each_entry(curr, &scmi_data.channel_list, list)
> {
> if ( (curr->domain_id == DOMID_INVALID) )
> {
>
Oh, now I understood your point. Thank you for that.
I will use DOMID_XEN for HYP_CHANNEL and remove curr->chan_id !=
HYP_CHANNEL. This will be fixed in v2.
>
> > > > + {
> > > > + curr->domain_id = domain_id;
> > > > + found = true;
> > > > + break;
> > > > + }
> > > > +
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > + if ( found )
> > > > + return curr;
> > > > +
> > > > + return NULL;
> > > > +}
> > > > +
> > > > +static void relinquish_scmi_channel(struct scmi_channel *channel)
> > > > +{
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + ASSERT(channel != NULL);
> > >
> > > the ASSERT could be before the spin_lock
> > >
> >
> > Thank you. I will fix it in v2.
> >
> > >
> > > > + channel->domain_id = DOMID_INVALID;
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > +}
> > > > +
> > > > +static struct scmi_channel *smc_create_channel(uint8_t chan_id,
> > > > + uint32_t func_id, uint64_t addr)
> > > > +{
> > > > + struct scmi_channel *channel;
> > > > + mfn_t mfn;
> > > > +
> > > > + channel = get_channel_by_id(chan_id);
> > > > + if ( channel )
> > > > + return ERR_PTR(EEXIST);
> > > > +
> > > > + channel = xmalloc(struct scmi_channel);
> > > > + if ( !channel )
> > > > + return ERR_PTR(ENOMEM);
> > > > +
> > > > + channel->chan_id = chan_id;
> > > > + channel->func_id = func_id;
> > > > + channel->domain_id = DOMID_INVALID;
> > >
> > > I take you are using DOMID_INVALID to mark a channel used by Xen itself?
> > > If so, then DOMID_XEN would be more appropriate.
> > >
> >
> > I use DOMID_INVALID to mark channel as free.
> >
> > >
> > > > + mfn = maddr_to_mfn(addr);
> > > > + channel->shmem = vmap(&mfn, 1);
> > >
> > > One thing to be careful is the mapping attributes, for a couple of
> > > reasons. As you might be aware, the ARM architecture forbids mismatching
> > > attributes for mapping memory in different places in the system. So the
> > > attributes that we use here must be the same used by the firmware
> > > (and/or the guest.)
> > >
> > > The second reason to be careful is that in the bindings example
> > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml the shared
> > > memory is "mmio-sram", which is special. It is not supposed to be normal
> > > memory, but it is OK to map it cacheable. Still, it might be more
> > > appropriate to use ioremap_cache.
> > >
> >
> > Originally, I used vmap here to have memcpy and it works fine in our
> > setup. But I will do some research and email you with the results.
> >
> > >
> > > > + if ( !channel->shmem )
> > > > + {
> > > > + xfree(channel);
> > > > + return ERR_PTR(ENOMEM);
> > > > + }
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: Got shmem after vmap %p\n", channel->shmem);
> > > > + channel->paddr = addr;
> > > > + channel->shmem->channel_status = SCMI_SHMEM_CHAN_STAT_CHANNEL_FREE;
> > > > + spin_lock_init(&channel->lock);
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + list_add(&channel->list, &scmi_data.channel_list);
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > + return channel;
> > > > +}
> > > > +
> > > > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > > > +{
> > > > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > +}
> > > > +
> > > > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > > > + uint64_t len)
> > > > +{
> > > > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > +}
> > > > +
> > > > +static int dt_update_domain_range(struct domain *d, uint64_t addr,
> > > > + uint64_t size)
> > > > +{
> > > > + struct dt_device_node *shmem_node;
> > > > + __be32 *hw_reg;
> > > > + const struct dt_property *pp;
> > > > + uint32_t len;
> > > > +
> > > > + shmem_node = dt_find_compatible_node(NULL, NULL, "arm,scmi-shmem");
> > >
> > > Here we are using "arm,scmi-shmem" while below we are checking for
> > > "linux,scmi_mem". What's the difference?
> >
> > linux,scmi_mem (I posted nodes examples above) describes memory region, allocated
> > for all channels, while arm,scmi-shmem points to the exact channel (page in
> > linux,scmi_mem region).
> >
> >
> > > Also, this function is looking for "arm,scmi-shmem" in dt_host and
> > > replaces its value. For dom0less domUs we'll probably need a
> > > make_scmi_node function to create the node from scratch like for
> > > instance xen/arch/arm/domain_build.c:make_gic_domU_node.
> > >
> > > I wonder if we had such a function whether it wouldn't be better to also
> > > use it for dom0 (and blacklist the physical "arm,scmi-shmem" in
> > > handle_node so that dom0 doesn't get the real shared memory information
> > > by accident).
> > >
> >
> > Thank you for the remark. I will rework this in v2.
> >
> > >
> > > > +
> > > > + if ( !shmem_node )
> > > > + {
> > > > + printk(XENLOG_ERR "scmi: Unable to find %s node in DT\n", SCMI_SHMEM);
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + pp = dt_find_property(shmem_node, "reg", &len);
> > > > + if ( !pp )
> > > > + {
> > > > + printk(XENLOG_ERR "scmi: Unable to find regs entry in shmem node\n");
> > > > + return -ENOENT;
> > > > + }
> > > > +
> > > > + hw_reg = pp->value;
> > > > + dt_set_range(&hw_reg, shmem_node, addr, size);
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static void free_channel_list(void)
> > > > +{
> > > > + struct scmi_channel *curr, *_curr;
> > > > +
> > > > + spin_lock(&scmi_data.channel_list_lock);
> > > > + list_for_each_entry_safe (curr, _curr, &scmi_data.channel_list, list)
> > > > + {
> > > > + vunmap(curr->shmem);
> > > > + list_del(&curr->list);
> > > > + xfree(curr);
> > > > + }
> > > > +
> > > > + spin_unlock(&scmi_data.channel_list_lock);
> > > > +}
> > > > +
> > > > +static __init bool scmi_probe(struct dt_device_node *scmi_node)
> > > > +{
> > > > + struct dt_device_node *shmem_node;
> > > > + int ret, i;
> > > > + struct scmi_channel *channel, *agent_channel;
> > > > + int n_agents;
> > > > + scmi_msg_header_t hdr;
> > > > + struct rx_t {
> > > > + int32_t status;
> > > > + uint32_t attributes;
> > > > + } rx;
> > >
> > > Should rx be defined at the top together with scmi_perms_tx_t and
> > > others?
> > >
> >
> > I'd rather move scmi_perms_tx_t to scmi_add_device_by_devid because it's
> > used only in 1 place.
> > So we will have rx and tx in scmi_add_device_by_devid and rx ( which
> > differs from rx in scmi_add_device_by_devid ) in scmi_probe.
> > I think it will be more understandable and no need to make unique names.
> > What do you think about that?
>
> I think that's OK
>
>
> > > > + uint32_t func_id;
> > > > +
> > > > + ASSERT(scmi_node != NULL);
> > > > +
> > > > + INIT_LIST_HEAD(&scmi_data.channel_list);
> > > > + spin_lock_init(&scmi_data.channel_list_lock);
> > > > +
> > > > + if ( !dt_property_read_u32(scmi_node, SCMI_SMC_ID, &func_id) )
> > > > + {
> > > > + printk(XENLOG_ERR "scmi: Unable to read smc-id from DT\n");
> > > > + return false;
> > > > + }
> > > > +
> > > > + shmem_node = dt_find_node_by_name(NULL, SCMI_SHARED_MEMORY);
> > >
> > > From the spec, it looks like you should be getting the shared memory
> > > area from the phandle list "shmem".
> > >
> >
> > We use SCMI_SHARED_MEMORY to get whole memory region (0x10 pages in my case),
> > we can use for the agents. As you can see below - Hypervisor received number of
> > agents from Firmware and split this region between agents.
>
> In general we can't use properties that are not part of the device tree
> spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
>
> "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> activities to get "linux,scmi_mem" upstream under
> Documentation/devicetree/bindings in Linux?
>
> If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> Documentation/devicetree/bindings (probably
> Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> work on the Xen code that makes use of it.
>
> Does it make sense?
>
Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
>
> > > > + if ( IS_ERR_OR_NULL(shmem_node) )
> > > > + {
> > > > + printk(XENLOG_ERR
> > > > + "scmi: Device tree error, can't parse shmem phandle %ld\n",
> > > > + PTR_ERR(shmem_node));
> > > > + return false;
> > > > + }
> > > > +
> > > > + ret = dt_device_get_address(shmem_node, 0, &scmi_data.shmem_addr,
> > > > + &scmi_data.shmem_size);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + return false;
> > > > +
> > > > + channel = smc_create_channel(HYP_CHANNEL, func_id, scmi_data.shmem_addr);
> > > > + if ( IS_ERR(channel) )
> > > > + return false;
> > > > +
> > > > + hdr.id = SCMI_BASE_PROTOCOL_ATTIBUTES;
> > > > + hdr.type = 0;
> > > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > > +
> > > > + ret = do_smc_xfer(channel, &hdr, NULL, 0, &rx, sizeof(rx));
> > > > + if ( ret )
> > > > + goto clean;
> > > > +
> > > > + ret = check_scmi_status(rx.status);
> > > > + if ( ret )
> > > > + goto clean;
> > > > +
> > > > + n_agents = FIELD_GET(MSG_N_AGENTS_MASK, rx.attributes);
> > > > + printk(XENLOG_DEBUG "scmi: Got agent count %d\n", n_agents);
> > > > +
> > > > + n_agents = (n_agents > scmi_data.shmem_size / PAGE_SIZE) ?
> > > > + scmi_data.shmem_size / PAGE_SIZE : n_agents;
> > > > +
> > > > + for ( i = 1; i < n_agents; i++ )
> > > > + {
> > >
> > > Given that HYP_CHANNEL is actually zero, it looks like we could do
> > > everything here in this loop but starting from i=0?
> > >
> >
> > We allocate HYP_CHANNEL before loop because we need it to request number
> > of agents. And we don't need to send SCMI_BASE_DISCOVER_AGENT to
> > HYP_CHANNEL.
>
> OK
>
>
> > > > + uint32_t tx_agent_id = 0xFFFFFFFF;
> > > > + struct {
> > > > + int32_t status;
> > > > + uint32_t agent_id;
> > > > + char name[16];
> > > > + } da_rx;
> > > > +
> > > > + agent_channel = smc_create_channel(i, func_id, scmi_data.shmem_addr +
> > > > + i * PAGE_SIZE);
> > > > + if ( IS_ERR(agent_channel) )
> > > > + {
> > > > + ret = PTR_ERR(agent_channel);
> > > > + goto clean;
> > > > + }
> > > > +
> > > > + hdr.id = SCMI_BASE_DISCOVER_AGENT;
> > > > + hdr.type = 0;
> > > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > > +
> > > > + ret = do_smc_xfer(agent_channel, &hdr, &tx_agent_id,
> > > > + sizeof(tx_agent_id), &da_rx, sizeof(da_rx));
> > > > + if ( ret )
> > > > + goto clean;
> > > > +
> > > > + ret = check_scmi_status(da_rx.status);
> > > > + if ( ret )
> > > > + goto clean;
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: status=0x%x id=0x%x name=%s\n",
> > > > + da_rx.status, da_rx.agent_id, da_rx.name);
> > > > +
> > > > + agent_channel->agent_id = da_rx.agent_id;
> > > > + }
> > > > +
> > > > + scmi_data.initialized = true;
> > > > + return true;
> > > > +
> > > > +clean:
> > > > + free_channel_list();
> > > > + return ret == 0;
> > > > +}
> > > > +
> > > > +static int scmi_domain_init(struct domain *d)
> > > > +{
> > > > + struct scmi_channel *channel;
> > > > + int ret;
> > > > +
> > > > + if ( !scmi_data.initialized )
> > > > + return 0;
> > > > +
> > > > + channel = aquire_scmi_channel(d->domain_id);
> > > > + if ( IS_ERR_OR_NULL(channel) )
> > > > + return -ENOENT;
> > > > +
> > > > + printk(XENLOG_INFO "scmi: Aquire SCMI channel id = 0x%x , domain_id = %d"
> > > > + "paddr = 0x%lx\n", channel->chan_id, channel->domain_id,
> > > > + channel->paddr);
> > > > +
> > > > + if ( is_hardware_domain(d) )
> > > > + {
> > > > + ret = map_memory_to_domain(d, scmi_data.shmem_addr,
> > > > + scmi_data.shmem_size);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + goto error;
> > > > +
> > > > + ret = dt_update_domain_range(d, channel->paddr, PAGE_SIZE);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + {
> > > > + int rc = unmap_memory_from_domain(d, scmi_data.shmem_addr,
> > > > + scmi_data.shmem_size);
> > > > + if ( rc )
> > > > + printk(XENLOG_ERR "Unable to unmap_memory_from_domain\n");
> > > > +
> > > > + goto error;
> > > > + }
> > > > + }
> > >
> > > Is dom0 the only domain to get direct access to the shared memory
> > > region? If so, I don't think it is a good idea to make Dom0 "special" in
> > > this case.
> > >
> > > Let me make an example: if we assign a device to a domU since boot, and
> > > dom0 wants to change the frequency of a clock that affects the assigned
> > > device (likely because it doesn't know it is assigned), then dom0
> > > shouldn't be able to. We might have to perform checks in Xen to make
> > > sure dom0 cannot stop the clock for the assigned device.
> > >
> > > So I think it would be better if all domains are treated the same way in
> > > the mediator unless really necessary.
> > >
> > > On the other hand, if all domains get access to the shared memory
> > > region, then I don't think this is likely the right place to create the
> > > dom0 mapping. We probably want to do it in domain_build.c in a way that
> > > can be reused for dom0less domUs.
> > >
> >
> > The idea is that all domains have their own page in shared memory
> > region and unigue agent_id. Agent_id is used to set permissions for
> > clocks\resets\power-domains etc. So during creation of domain (domUs or
> > dom0) device-tree is processed using scmi_add_dt_device and
> > clocks\resets\power-domains which are related to dom0 devices will be
> > requested by using SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > All passed-through devices will be requested during DomU creation. Which
> > means dom0 do not have an access to clocks\resets\power-domains, which
> > are related to DomU.
>
> OK, excellent
>
>
> > > In regards to shared memory: it looks like the only two functions to
> > > access the real shared memory are send_smc_message and get_smc_response.
> > > If that is the case, then we actually don't need to expose the real
> > > shared memory to any of the domains.
> > >
> > > We could simply:
> > >
> > > - expose a regular normal memory region as dom0/domU channel memory
> > > - on SMC trap, read from the "fake" shared memory and set the
> > > corresponding real shared memory on the appropriate channel
> > > - issue the SMC call
> > > - on return from SMC, copy over data from the real shared memory to the
> > > "fake" channel reagion
> >
> > Hypervisor redirects only SMC calls from guests and set agent_id to SMC
> > parameters as a7. The idea was to give page for each agent, so we don't
> > need to make additional read/write each time we receive SMC call.
> > All we povide from hypervisor is agent_id. Firmware is responsible for
> > reading memory from the correct address and place the response.
> >
> > >
> > > This is useful if we need to "filter" any of the SCMI commands and
> > > options from the domains to the firmware, and also it is useful if the
> > > channel memory is not page aligned. But if the permissions are
> > > fine-grained enough and also the channel memory is page aligned (and
> > > multiple of 4K in size) then we could map the memory.
> > >
> >
> > In current implementation we suppose that channel memory is page aligned.
> > I think that Firmware should be responsible for permissions handling and
> > "filtering", that's why permission calls were added to SCMI spec.
> > I tried to make mediator as thin as possible.
>
> OK. Can we check that the channel memory is page aligned at init time
> and throw an error if it is not the case?
>
Yes. I will add this check in v2.
>
> > > > +
> > > > + d->arch.sci = channel;
> > > > +
> > > > + return 0;
> > > > +error:
> > > > + relinquish_scmi_channel(channel);
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static int scmi_add_device_by_devid(struct domain *d, uint32_t scmi_devid)
> > > > +{
> > > > + struct scmi_channel *channel, *agent_channel;
> > > > + scmi_msg_header_t hdr;
> > > > + scmi_perms_tx_t tx;
> > > > + struct rx_t {
> > > > + int32_t status;
> > > > + uint32_t attributes;
> > > > + } rx;
> > > > + int ret;
> > > > +
> > > > + if ( !scmi_data.initialized )
> > > > + return 0;
> > > > +
> > > > + printk(XENLOG_DEBUG "scmi: scmi_devid = %d\n", scmi_devid);
> > > > +
> > > > + agent_channel = get_channel_by_domain(d->domain_id);
> > > > + if ( IS_ERR_OR_NULL(agent_channel) )
> > > > + return PTR_ERR(agent_channel);
> > > > +
> > > > + channel = get_channel_by_id(HYP_CHANNEL);
> > > > + if ( IS_ERR_OR_NULL(channel) )
> > > > + return PTR_ERR(channel);
> > > > +
> > > > + hdr.id = SCMI_BASE_SET_DEVICE_PERMISSIONS;
> > > > + hdr.type = 0;
> > > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > > +
> > > > + tx.agent_id = agent_channel->agent_id;
> > > > + tx.device_id = scmi_devid;
> > > > + tx.flags = SCMI_ALLOW_ACCESS;
> > > > +
> > > > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(&rx));
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + return ret;
> > > > +
> > > > + ret = check_scmi_status(rx.status);
> > > > + if ( IS_ERR_VALUE(ret) )
> > > > + return ret;
> > > > +
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static int scmi_add_dt_device(struct domain *d, struct dt_device_node *dev)
> > > > +{
> > > > + uint32_t scmi_devid;
> > > > +
> > > > + if ( (!scmi_data.initialized) || (!d->arch.sci) )
> > > > + return 0;
> > > > +
> > > > + if ( !dt_property_read_u32(dev, "scmi_devid", &scmi_devid) )
> > > > + return 0;
> > >
> > > scmi_devid is another property that is not documented in the binding.
> > >
> >
> > This property should be added to the device nodes, which are using scmi
> > to work with clocks\resets\power-domains etc. This id should match the
> > device_id, defined in Firmware. Hypervisor send this device_id to the Firmware
> > as parameter to the permission request. Firmware set permissions to
> > clocks\resets\power-domains, related to this device.
>
> OK, I see. Unfortunately, scmi_devid is also not described under
> Documentation/devicetree/bindings/.
>
> This property seems to be actually required for the system to work
> correctly.
>
> Unless somebody else is already working on this, please send a patch to
> the Linux kernel mailing list CCing the SCMI maintainers and Rob Herring
> to introduce scmi_devid as a new property.
>
I will prepare and send patch introducing scmi_devid and linux,scmi_mem.
>
> > > > + printk(XENLOG_INFO "scmi: dt_node = %s\n", dt_node_full_name(dev));
> > > > +
> > > > + return scmi_add_device_by_devid(d, scmi_devid);
> > > > +}
> > > > +
> > > > +static int scmi_relinquish_resources(struct domain *d)
> > > > +{
> > > > + int ret;
> > > > + struct scmi_channel *channel, *agent_channel;
> > > > + scmi_msg_header_t hdr;
> > > > + struct reset_agent_tx {
> > > > + uint32_t agent_id;
> > > > + uint32_t flags;
> > > > + } tx;
> > > > + uint32_t rx;
> > > > +
> > > > + if ( !d->arch.sci )
> > > > + return 0;
> > > > +
> > > > + agent_channel = d->arch.sci;
> > > > +
> > > > + spin_lock(&agent_channel->lock);
> > > > + tx.agent_id = agent_channel->agent_id;
> > > > + spin_unlock(&agent_channel->lock);
> > > > +
> > > > + channel = get_channel_by_id(HYP_CHANNEL);
> > > > + if ( !channel )
> > > > + {
> > > > + printk(XENLOG_ERR
> > > > + "scmi: Unable to get Hypervisor scmi channel for domain %d\n",
> > > > + d->domain_id);
> > > > + return -EINVAL;
> > > > + }
> > > > +
> > > > + hdr.id = SCMI_BASE_RESET_AGENT_CONFIGURATION;
> > > > + hdr.type = 0;
> > > > + hdr.protocol = SCMI_BASE_PROTOCOL;
> > > > +
> > > > + tx.flags = 0;
> > > > +
> > > > + ret = do_smc_xfer(channel, &hdr, &tx, sizeof(tx), &rx, sizeof(rx));
> > > > + if ( ret )
> > > > + return ret;
> > > > +
> > > > + ret = check_scmi_status(rx);
> > > > +
> > > > + return ret;
> > > > +}
> > > > +
> > > > +static void scmi_domain_destroy(struct domain *d)
> > > > +{
> > > > + struct scmi_channel *channel;
> > > > +
> > > > + if ( !d->arch.sci )
> > > > + return;
> > > > +
> > > > + channel = d->arch.sci;
> > > > + spin_lock(&channel->lock);
> > > > +
> > > > + relinquish_scmi_channel(channel);
> > > > + printk(XENLOG_DEBUG "scmi: Free domain %d\n", d->domain_id);
> > > > +
> > > > + d->arch.sci = NULL;
> > > > +
> > > > + unmap_memory_from_domain(d, channel->paddr, PAGE_SIZE);
> > > > + spin_unlock(&channel->lock);
> > > > + return;
> > > > +}
> > > > +
> > > > +static bool scmi_handle_call(struct domain *d, void *args)
> > > > +{
> > > > + bool res = false;
> > > > + struct scmi_channel *agent_channel;
> > > > + struct arm_smccc_res resp;
> > > > + struct cpu_user_regs *regs = args;
> > > > +
> > > > + if ( !d->arch.sci )
> > > > + return false;
> > > > +
> > > > + agent_channel = d->arch.sci;
> > > > + spin_lock(&agent_channel->lock);
> > > > +
> > > > + if ( agent_channel->func_id != regs->x0 )
> > > > + {
> > > > + printk(XENLOG_ERR "scmi: func_id mismatch, exiting\n");
> > > > + goto unlock;
> > > > + }
> > > > +
> > > > + arm_smccc_smc(agent_channel->func_id, 0, 0, 0, 0, 0, 0,
> > > > + agent_channel->chan_id, &resp);
> > > > +
> > > > + set_user_reg(regs, 0, resp.a0);
> > > > + set_user_reg(regs, 1, resp.a1);
> > > > + set_user_reg(regs, 2, resp.a2);
> > > > + set_user_reg(regs, 3, resp.a3);
> > > > + res = true;
> > > > +unlock:
> > > > + spin_unlock(&agent_channel->lock);
> > > > +
> > > > + return res;
> > > > +}
> > > > +
> > > > +static int scmi_get_channel_paddr(void *scmi_ops,
> > > > + struct xen_arch_domainconfig *config)
> > > > +{
> > > > + struct scmi_channel *agent_channel = scmi_ops;
> > > > +
> > > > + if ( !agent_channel )
> > > > + return -EINVAL;
> > > > +
> > > > + config->sci_agent_paddr = agent_channel->paddr;
> > > > + return 0;
> > > > +}
> > >
> > > I am still not sure why it couldn't be done by scmi_domain_init.
> > >
> >
> > I can move this logic to scmi_domain_init, but in this case I have to add
> > struct xen_arch_domainconfig *config as input parameter to
> > scmi_domain_init and pass NULL from construct_dom0.
> > Do you think this approach would be better?
>
> I think it is OK to pass struct xen_arch_domainconfig *config as input
> parameter to scmi_domain_init.
Ok, I will fix it in v2.
>
> For dom0, why is sci_agent_paddr not supposed to be set?
>
paddr is the channel address. We need this address to update reg address
of arm,scmi-shmem node in guest device-tree, so guest scmi driver will
use shmem related to it.
>
>
> > Also I think it's reasonable to pass xen_arch_domainconfig so different
> > implementations could set another data they would probably need.
> >
> > >
> > > > +static const struct dt_device_match scmi_smc_match[] __initconst =
> > > > +{
> > > > + DT_MATCH_SCMI_SMC,
> > > > + { /* sentinel */ },
> > > > +};
> > > > +
> > > > +static const struct sci_mediator_ops scmi_ops =
> > > > +{
> > > > + .probe = scmi_probe,
> > > > + .domain_init = scmi_domain_init,
> > > > + .domain_destroy = scmi_domain_destroy,
> > > > + .add_dt_device = scmi_add_dt_device,
> > > > + .relinquish_resources = scmi_relinquish_resources,
> > > > + .handle_call = scmi_handle_call,
> > > > + .get_channel_info = scmi_get_channel_paddr
> > > > +};
> > > > +
> > > > +REGISTER_SCI_MEDIATOR(scmi_smc, "SCMI-SMC", XEN_DOMCTL_CONFIG_SCI_SCMI_SMC,
> > > > + scmi_smc_match, &scmi_ops);
> > > > +
> > > > +/*
> > > > + * Local variables:
> > > > + * mode: C
> > > > + * c-file-style: "BSD"
> > > > + * c-basic-offset: 4
> > > > + * indent-tabs-mode: nil
> > > > + * End:
> > > > + */
> > > > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > > > index 9180be5e86..a67237942d 100644
> > > > --- a/xen/include/public/arch-arm.h
> > > > +++ b/xen/include/public/arch-arm.h
> > > > @@ -315,6 +315,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> > > > #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> > > >
> > > > #define XEN_DOMCTL_CONFIG_SCI_NONE 0
> > > > +#define XEN_DOMCTL_CONFIG_SCI_SCMI_SMC 1
> > > >
> > > > struct xen_arch_domainconfig {
> > > > /* IN/OUT */
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-21 20:03 ` Oleksii Moisieiev
@ 2021-12-21 21:22 ` Stefano Stabellini
2021-12-22 11:04 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-21 21:22 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> Hi Stefano,
>
> On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > Hi Stefano,
> > >
> > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > power-domains without access to CPG.
> > > > >
> > > > > The following features are implemented:
> > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > - set device permissions for Domains based on the Domain partial
> > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > resets and power-domains via SCMI;
> > > > > - redirect scmi messages from Domains to ATF.
> > > > >
> > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > ---
> > > > > xen/arch/arm/Kconfig | 2 +
> > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > xen/include/public/arch-arm.h | 1 +
> > > > > 5 files changed, 809 insertions(+)
> > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > >
> > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > index 186e1db389..02d96c6cfc 100644
> > > > > --- a/xen/arch/arm/Kconfig
> > > > > +++ b/xen/arch/arm/Kconfig
> > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > support. It allows guests to control system resourcess via one of
> > > > > SCI mediators implemented in XEN.
> > > > >
> > > > > +source "arch/arm/sci/Kconfig"
> > > > > +
> > > > > endmenu
> > > > >
> > > > > menu "ARM errata workaround via the alternative framework"
> > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > new file mode 100644
> > > > > index 0000000000..9563067ddc
> > > > > --- /dev/null
> > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > @@ -0,0 +1,10 @@
> > > > > +config SCMI_SMC
> > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > + default n
> > > > > + depends on SCI
> > > > > + ---help---
> > > > > +
> > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > + This feature allows drivers from Domains to work with System
> > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > + for communication.
> > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > index 837dc7492b..67f2611872 100644
> > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > @@ -1 +1,2 @@
> > > > > obj-y += sci.o
> > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > new file mode 100644
> > > > > index 0000000000..2eb01ea82d
> > > > > --- /dev/null
> > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > @@ -0,0 +1,795 @@
> > > > > +/*
> > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > + *
> > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > + *
> > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > + *
> > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > + * it under the terms of the GNU General Public License as published by
> > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > + * (at your option) any later version.
> > > > > + *
> > > > > + * This program is distributed in the hope that it will be useful,
> > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > + * GNU General Public License for more details.
> > > > > + */
> > > > > +
> > > > > +#include <asm/sci/sci.h>
> > > > > +#include <asm/smccc.h>
> > > > > +#include <asm/io.h>
> > > > > +#include <xen/bitops.h>
> > > > > +#include <xen/config.h>
> > > > > +#include <xen/sched.h>
> > > > > +#include <xen/device_tree.h>
> > > > > +#include <xen/iocap.h>
> > > > > +#include <xen/init.h>
> > > > > +#include <xen/err.h>
> > > > > +#include <xen/lib.h>
> > > > > +#include <xen/list.h>
> > > > > +#include <xen/mm.h>
> > > > > +#include <xen/string.h>
> > > > > +#include <xen/time.h>
> > > > > +#include <xen/vmap.h>
> > > > > +
> > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > +
> > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > +#define SCMI_SUCCESS 0
> > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > +#define SCMI_DENIED (-3)
> > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > +#define SCMI_BUSY (-6)
> > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > +
> > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > +
> > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > >
> > > > I could find the following SCMI binding in Linux, which describes
> > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > >
> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > >
> > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > the "shmem" property instead? And the compatible string used for this
> > > > seems to be "arm,scmi-shmem".
> > > >
> > >
> > > We use linux,scmi_mem node to reserve memory, needed for all
> > > channels:
> > >
> > > reserved-memory {
> > > /* reserved region for scmi channels*/
> > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > no-map;
> > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > };
> > > };
> > >
> > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > the current scmi channel:
> > >
> > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > };
> > >
> > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > assigned to this agent.
> >
> > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > compatible string, not as a node name, and it would need to be described
> > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> >
> > But from your description I don't think it is necessary. We can just use
> > "arm,scmi-shmem" to describe all the required regions:
> >
> > reserved-memory {
> > scp-shmem@0x53FF0000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > };
> > scp-shmem@0x53FF1000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > };
> > scp-shmem@0x53FF2000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > };
> > ...
> >
> > In other words, if all the individual channel pages are described as
> > "arm,scmi-shmem", why do we also need a single larger region as
> > "linux,scmi_mem"?
> >
>
> That was my first implementation. But I've met a problem with
> scmi driver in kernel. I don't remember the exact place, but I remember
> there were some if, checking if memory weren't reserved.
> That's why I ended up splitting nodes reserved memory region and actual
> shmem page.
> For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> which has no compatible string and provides no-map property.
> linux,scmi_shmem node is needed to prevent xen from allocating this
> space for the domain.
>
> Very interesting question about should I introduce linux,scmi_mem node
> and scmi_devid property to the
> Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> Those node and property are needed only for Xen and useless for
> non-virtualized systems. I can add this node and property description to
> arm,scmi.yaml, but leave a note that this is Xen specific params.
> What do you think about it?
Reply below
[...]
> > In general we can't use properties that are not part of the device tree
> > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> >
> > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > activities to get "linux,scmi_mem" upstream under
> > Documentation/devicetree/bindings in Linux?
> >
> > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > Documentation/devicetree/bindings (probably
> > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > work on the Xen code that makes use of it.
> >
> > Does it make sense?
> >
>
> Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
Xen specific. In general, it would be best not to introduce Xen specific
properties into generic bindings. It is a problem both from a
specification perspective (because it has hard to handle Xen specific
cases in fully generic bindings, especially as those bindings are
maintained as part of the Linux kernel) and from a user perspective
(because now the user has to deal with a Xen-specific dtb, or has to
modify the host dtb to add Xen-specific information by hand.)
Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
looks like a generic property that should be needed for the Linux SCMI
driver too. Why the Linux driver doesn't need it?
In regards to linux,scmi_mem, I think it would be best to do without it
and fix the Linux SCMI driver if we need to do so. Xen should be able to
parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
way, I don't think we should need linux,scmi_mem.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-21 14:45 ` Anthony PERARD
@ 2021-12-21 21:39 ` Stefano Stabellini
2021-12-22 9:24 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-21 21:39 UTC (permalink / raw)
To: Anthony PERARD
Cc: Oleksii Moisieiev, Oleksandr, xen-devel, Wei Liu, Juergen Gross,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 4289 bytes --]
On Tue, 21 Dec 2021, Anthony PERARD wrote:
> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
> > > On 14.12.21 11:34, Oleksii Moisieiev wrote:
> > > > @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> > > > {
> > > > AO_GC;
> > > > libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> > > > - int i, rc = 0;
> > > > + int i, rc = 0, rc_sci = 0;
> > > > for (i = 0; i < d_config->num_dtdevs; i++) {
> > > > const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
> > > > LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> > > > rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> > > > - if (rc < 0) {
> > > > - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> > > > + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> > > > +
> > > > + if ((rc < 0) && (rc_sci < 0)) {
> > > > + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> > > > + "xc_domain_add_sci_device failed: %d",
> > > > + rc, rc_sci);
> > > > goto out;
> > > > }
> > > > +
> > > > + if (rc)
> > > > + rc = rc_sci;
> > >
> > >
> > > If I get this code right, it sounds like the dom.cfg's dtdev property is
> > > reused to describe sci devices as well, but the libxl__add_dtdevs() does not
> > > (and can not) differentiate them. So it has no option but to send two
> > > domctls for each device in dtdevs[] hoping to at least one domctl to
> > > succeeded. Or I really missed something?
> > >
> > > It feels to me that:
> > > - either new dom.cfg's property could be introduced (scidev?) to describe
> > > sci devices together with new parsing logic/management code, so you will end
> > > up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
> > > so no mixing things.
> > > - or indeed dtdev logic could be *completely* reused including extending
> > > XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
> > > is used to describe devices for the passthrough (to configure an IOMMU for
> > > the device), in that case you wouldn't need an extra
> > > XEN_DOMCTL_add_sci_device introduced by current patch.
I realize I did my review before reading Oleksandr's comments. I fully
agree with his feedback. Having seen how difficult is for users to setup
a domU configuration correctly today, I would advise to try to reuse the
existing dtdev property instead of adding yet one new property to make
the life of our users easier.
> > > Personally I would use the first option as I am not sure that second option
> > > is conceptually correct && possible. I would leave this for the maintainers
> > > to clarify.
> > >
> >
> > Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
> > seems not to be conceptually correct. Introducing new dom.cfg property
> > seems to be the only way to avoid extra Domctl calls.
> > I will handle this in v2 if there will be no complains from Maintainers.
>
> I don't know if there will be a complains in v2 or not :-), but using
> something different from dtdev sound good.
>
> If I understand correctly, dtdev seems to be about passing through an
> existing device to a guest, and this new sci device seems to be about having Xen
> "emulating" an sci device which the guest will use. So introducing
> something new (scidev or other name) sounds good.
Users already have to provide 4 properties (dtdev, iomem, irqs,
device_tree) to setup device assignment. I think that making it 5
properties would not be a step forward :-)
To me dtdev and XEN_DOMCTL_assign_device are appropriate because they
signify device assignment of one or more devices. We are not adding any
additional "meaning" to them. It is just that we'll automatically detect
and generate any SCMI configurations based on the list of assigned
devices. Because if SCMI is enabled and a device is assigned to the
guest, then I think we want to add the device to the SCMI list of
devices automatically.
If we really want to introduce a new list of devices, please make it
optional so that most times the user can skip it unless really required.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-21 21:39 ` Stefano Stabellini
@ 2021-12-22 9:24 ` Julien Grall
2021-12-22 11:17 ` Volodymyr Babchuk
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-22 9:24 UTC (permalink / raw)
To: Stefano Stabellini, Anthony PERARD
Cc: Oleksii Moisieiev, Oleksandr, xen-devel, Wei Liu, Juergen Gross,
Andrew Cooper, George Dunlap, Jan Beulich, Volodymyr Babchuk,
Bertrand Marquis
Hi Stefano,
On 21/12/2021 22:39, Stefano Stabellini wrote:
> On Tue, 21 Dec 2021, Anthony PERARD wrote:
>> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
>>>> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>>>>> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
>>>>> {
>>>>> AO_GC;
>>>>> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
>>>>> - int i, rc = 0;
>>>>> + int i, rc = 0, rc_sci = 0;
>>>>> for (i = 0; i < d_config->num_dtdevs; i++) {
>>>>> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>>>>> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
>>>>> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
>>>>> - if (rc < 0) {
>>>>> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
>>>>> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
>>>>> +
>>>>> + if ((rc < 0) && (rc_sci < 0)) {
>>>>> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
>>>>> + "xc_domain_add_sci_device failed: %d",
>>>>> + rc, rc_sci);
>>>>> goto out;
>>>>> }
>>>>> +
>>>>> + if (rc)
>>>>> + rc = rc_sci;
>>>>
>>>>
>>>> If I get this code right, it sounds like the dom.cfg's dtdev property is
>>>> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
>>>> (and can not) differentiate them. So it has no option but to send two
>>>> domctls for each device in dtdevs[] hoping to at least one domctl to
>>>> succeeded. Or I really missed something?
>>>>
>>>> It feels to me that:
>>>> - either new dom.cfg's property could be introduced (scidev?) to describe
>>>> sci devices together with new parsing logic/management code, so you will end
>>>> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
>>>> so no mixing things.
>>>> - or indeed dtdev logic could be *completely* reused including extending
>>>> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
>>>> is used to describe devices for the passthrough (to configure an IOMMU for
>>>> the device), in that case you wouldn't need an extra
>>>> XEN_DOMCTL_add_sci_device introduced by current patch.
>
> I realize I did my review before reading Oleksandr's comments. I fully
> agree with his feedback. Having seen how difficult is for users to setup
> a domU configuration correctly today, I would advise to try to reuse the
> existing dtdev property instead of adding yet one new property to make
> the life of our users easier.
>
>
>
>>>> Personally I would use the first option as I am not sure that second option
>>>> is conceptually correct && possible. I would leave this for the maintainers
>>>> to clarify.
>>>>
>>>
>>> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
>>> seems not to be conceptually correct. Introducing new dom.cfg property
>>> seems to be the only way to avoid extra Domctl calls.
>>> I will handle this in v2 if there will be no complains from Maintainers.
>>
>> I don't know if there will be a complains in v2 or not :-), but using
>> something different from dtdev sound good.
>>
>> If I understand correctly, dtdev seems to be about passing through an
>> existing device to a guest, and this new sci device seems to be about having Xen
>> "emulating" an sci device which the guest will use. So introducing
>> something new (scidev or other name) sounds good.
>
> Users already have to provide 4 properties (dtdev, iomem, irqs,
> device_tree) to setup device assignment. I think that making it 5
> properties would not be a step forward :-)
IIRC, in the past, we discussed to fetch the information directly from
the partial device-tree. Maybe this discussion needs to be revived?
Although, this is a separate topic from this series.
>
> To me dtdev and XEN_DOMCTL_assign_device are appropriate because they
> signify device assignment of one or more devices. We are not adding any
> additional "meaning" to them. It is just that we'll automatically detect
> and generate any SCMI configurations based on the list of assigned
> devices. Because if SCMI is enabled and a device is assigned to the
> guest, then I think we want to add the device to the SCMI list of
> devices automatically.
I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is a
pitfall with that approach.
At the moment, they are only used for device protected by the IOMMU. If
the device is not protected by the IOMMU then it will return an error.
Now, with your approach we may have a device that is not protected by
the IOMMU but require to a SCMI configuration.
I don't think it would be sensible to just return "succeed" here because
technically you are asking to assign a non-protected device. But at the
same time, it would prevent a user to assign a non-DMA capable device.
So how do you suggest to approach this?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-21 0:54 ` Stefano Stabellini
@ 2021-12-22 10:24 ` Oleksii Moisieiev
2021-12-23 2:23 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-22 10:24 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
On Mon, Dec 20, 2021 at 04:54:25PM -0800, Stefano Stabellini wrote:
> On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > This enumeration sets SCI type for the domain. Currently there is
> > two possible options: either 'none' or 'scmi_smc'.
> >
> > 'none' is the default value and it disables SCI support at all.
> >
> > 'scmi_smc' enables access to the Firmware from the domains using SCMI
> > protocol and SMC as transport.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> > tools/include/libxl.h | 5 +++++
> > tools/libs/light/libxl_types.idl | 6 ++++++
> > tools/xl/xl_parse.c | 9 +++++++++
> > 4 files changed, 42 insertions(+)
> >
> > diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> > index b98d161398..92d0593875 100644
> > --- a/docs/man/xl.cfg.5.pod.in
> > +++ b/docs/man/xl.cfg.5.pod.in
> > @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
> >
> > =back
> >
> > +=item B<sci="STRING">
> > +
> > +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> > +allows domain to manage various functions that are provided by HW platform.
> > +
> > +=over 4
> > +
> > +=item B<none>
> > +
> > +Don't allow guest to use SCI if present on the platform. This is the default
> > +value.
> > +
> > +=item B<scmi_smc>
> > +
> > +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> > +Inferface - allows domain to manage various functions that are provided by HW
> > +platform, such as clocks, resets and power-domains. Xen will mediate access to
> > +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> > +SMC is used as transport.
>
> Would it make sense to actually enable SCMI-SMC support for the guest by
> default if the guest has any devices directly assigned?
Hi Stefano,
Previously we discussed with Oleksandr about introducing new dom.cfg
parameter, such as scidev[] lists all sci related nodes, which will help to
avoid extra domctl calls.
But there is still a question how mediator types should be set for
different domains? I know currently system supports only one mediator
type, but different implementation should be also possible.
For example, if we have to start dom0 and domd using scmi_mailbox
mediator and domU using scmi_smc, because our system supports only 2
mailboxes.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-21 21:22 ` Stefano Stabellini
@ 2021-12-22 11:04 ` Oleksii Moisieiev
2021-12-23 2:23 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-22 11:04 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > Hi Stefano,
> >
> > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > Hi Stefano,
> > > >
> > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > power-domains without access to CPG.
> > > > > >
> > > > > > The following features are implemented:
> > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > resets and power-domains via SCMI;
> > > > > > - redirect scmi messages from Domains to ATF.
> > > > > >
> > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > ---
> > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > 5 files changed, 809 insertions(+)
> > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > >
> > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > support. It allows guests to control system resourcess via one of
> > > > > > SCI mediators implemented in XEN.
> > > > > >
> > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > +
> > > > > > endmenu
> > > > > >
> > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > new file mode 100644
> > > > > > index 0000000000..9563067ddc
> > > > > > --- /dev/null
> > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > @@ -0,0 +1,10 @@
> > > > > > +config SCMI_SMC
> > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > + default n
> > > > > > + depends on SCI
> > > > > > + ---help---
> > > > > > +
> > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > + This feature allows drivers from Domains to work with System
> > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > + for communication.
> > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > index 837dc7492b..67f2611872 100644
> > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > @@ -1 +1,2 @@
> > > > > > obj-y += sci.o
> > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > new file mode 100644
> > > > > > index 0000000000..2eb01ea82d
> > > > > > --- /dev/null
> > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > @@ -0,0 +1,795 @@
> > > > > > +/*
> > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > + *
> > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > + *
> > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > + *
> > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > + * (at your option) any later version.
> > > > > > + *
> > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > + * GNU General Public License for more details.
> > > > > > + */
> > > > > > +
> > > > > > +#include <asm/sci/sci.h>
> > > > > > +#include <asm/smccc.h>
> > > > > > +#include <asm/io.h>
> > > > > > +#include <xen/bitops.h>
> > > > > > +#include <xen/config.h>
> > > > > > +#include <xen/sched.h>
> > > > > > +#include <xen/device_tree.h>
> > > > > > +#include <xen/iocap.h>
> > > > > > +#include <xen/init.h>
> > > > > > +#include <xen/err.h>
> > > > > > +#include <xen/lib.h>
> > > > > > +#include <xen/list.h>
> > > > > > +#include <xen/mm.h>
> > > > > > +#include <xen/string.h>
> > > > > > +#include <xen/time.h>
> > > > > > +#include <xen/vmap.h>
> > > > > > +
> > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > +
> > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > +#define SCMI_SUCCESS 0
> > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > +#define SCMI_DENIED (-3)
> > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > +#define SCMI_BUSY (-6)
> > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > +
> > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > +
> > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > >
> > > > > I could find the following SCMI binding in Linux, which describes
> > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > >
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > >
> > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > the "shmem" property instead? And the compatible string used for this
> > > > > seems to be "arm,scmi-shmem".
> > > > >
> > > >
> > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > channels:
> > > >
> > > > reserved-memory {
> > > > /* reserved region for scmi channels*/
> > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > no-map;
> > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > };
> > > > };
> > > >
> > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > the current scmi channel:
> > > >
> > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > };
> > > >
> > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > assigned to this agent.
> > >
> > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > compatible string, not as a node name, and it would need to be described
> > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > >
> > > But from your description I don't think it is necessary. We can just use
> > > "arm,scmi-shmem" to describe all the required regions:
> > >
> > > reserved-memory {
> > > scp-shmem@0x53FF0000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > };
> > > scp-shmem@0x53FF1000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > };
> > > scp-shmem@0x53FF2000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > };
> > > ...
> > >
> > > In other words, if all the individual channel pages are described as
> > > "arm,scmi-shmem", why do we also need a single larger region as
> > > "linux,scmi_mem"?
> > >
> >
> > That was my first implementation. But I've met a problem with
> > scmi driver in kernel. I don't remember the exact place, but I remember
> > there were some if, checking if memory weren't reserved.
> > That's why I ended up splitting nodes reserved memory region and actual
> > shmem page.
> > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > which has no compatible string and provides no-map property.
> > linux,scmi_shmem node is needed to prevent xen from allocating this
> > space for the domain.
> >
> > Very interesting question about should I introduce linux,scmi_mem node
> > and scmi_devid property to the
> > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > Those node and property are needed only for Xen and useless for
> > non-virtualized systems. I can add this node and property description to
> > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > What do you think about it?
>
> Reply below
>
> [...]
>
>
> > > In general we can't use properties that are not part of the device tree
> > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > >
> > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > activities to get "linux,scmi_mem" upstream under
> > > Documentation/devicetree/bindings in Linux?
> > >
> > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > Documentation/devicetree/bindings (probably
> > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > work on the Xen code that makes use of it.
> > >
> > > Does it make sense?
> > >
> >
> > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
>
> I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> Xen specific. In general, it would be best not to introduce Xen specific
> properties into generic bindings. It is a problem both from a
> specification perspective (because it has hard to handle Xen specific
> cases in fully generic bindings, especially as those bindings are
> maintained as part of the Linux kernel) and from a user perspective
> (because now the user has to deal with a Xen-specific dtb, or has to
> modify the host dtb to add Xen-specific information by hand.)
>
>
> Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> looks like a generic property that should be needed for the Linux SCMI
> driver too. Why the Linux driver doesn't need it?
>
scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
On non-virtualized systems - there is no need of this call, because OS
is the only one entity, running on the system.
I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
list of device_ids to dom.cfg, such as:
sci_devs = [ 0, 1, 15, 35 ];
Using this approach, we can remove scmi_devid from the device tree and
just pass a list of scmi_devids to XEN using additional hypercall.
We can probably make hypercall taking devid list as input parameter.
This will take only 1 hypercall to setup sci permissions.
>
> In regards to linux,scmi_mem, I think it would be best to do without it
> and fix the Linux SCMI driver if we need to do so. Xen should be able to
> parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> way, I don't think we should need linux,scmi_mem.
This requires further investigation. I will try to make implementation
without linux,scmi_mem, using only arm,scmi-shmem nodes and share
reuslts with you.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-22 9:24 ` Julien Grall
@ 2021-12-22 11:17 ` Volodymyr Babchuk
2021-12-22 11:30 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Volodymyr Babchuk @ 2021-12-22 11:17 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Anthony PERARD, Oleksii Moisieiev, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi Julien,
Julien Grall <julien@xen.org> writes:
> Hi Stefano,
>
> On 21/12/2021 22:39, Stefano Stabellini wrote:
>> On Tue, 21 Dec 2021, Anthony PERARD wrote:
>>> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
>>>>> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>>>>>> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
>>>>>> {
>>>>>> AO_GC;
>>>>>> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
>>>>>> - int i, rc = 0;
>>>>>> + int i, rc = 0, rc_sci = 0;
>>>>>> for (i = 0; i < d_config->num_dtdevs; i++) {
>>>>>> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>>>>>> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
>>>>>> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
>>>>>> - if (rc < 0) {
>>>>>> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
>>>>>> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
>>>>>> +
>>>>>> + if ((rc < 0) && (rc_sci < 0)) {
>>>>>> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
>>>>>> + "xc_domain_add_sci_device failed: %d",
>>>>>> + rc, rc_sci);
>>>>>> goto out;
>>>>>> }
>>>>>> +
>>>>>> + if (rc)
>>>>>> + rc = rc_sci;
>>>>>
>>>>>
>>>>> If I get this code right, it sounds like the dom.cfg's dtdev property is
>>>>> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
>>>>> (and can not) differentiate them. So it has no option but to send two
>>>>> domctls for each device in dtdevs[] hoping to at least one domctl to
>>>>> succeeded. Or I really missed something?
>>>>>
>>>>> It feels to me that:
>>>>> - either new dom.cfg's property could be introduced (scidev?) to describe
>>>>> sci devices together with new parsing logic/management code, so you will end
>>>>> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
>>>>> so no mixing things.
>>>>> - or indeed dtdev logic could be *completely* reused including extending
>>>>> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
>>>>> is used to describe devices for the passthrough (to configure an IOMMU for
>>>>> the device), in that case you wouldn't need an extra
>>>>> XEN_DOMCTL_add_sci_device introduced by current patch.
>> I realize I did my review before reading Oleksandr's comments. I
>> fully
>> agree with his feedback. Having seen how difficult is for users to setup
>> a domU configuration correctly today, I would advise to try to reuse the
>> existing dtdev property instead of adding yet one new property to make
>> the life of our users easier.
>>
>>
>>>>> Personally I would use the first option as I am not sure that second option
>>>>> is conceptually correct && possible. I would leave this for the maintainers
>>>>> to clarify.
>>>>>
>>>>
>>>> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
>>>> seems not to be conceptually correct. Introducing new dom.cfg property
>>>> seems to be the only way to avoid extra Domctl calls.
>>>> I will handle this in v2 if there will be no complains from Maintainers.
>>>
>>> I don't know if there will be a complains in v2 or not :-), but using
>>> something different from dtdev sound good.
>>>
>>> If I understand correctly, dtdev seems to be about passing through an
>>> existing device to a guest, and this new sci device seems to be about having Xen
>>> "emulating" an sci device which the guest will use. So introducing
>>> something new (scidev or other name) sounds good.
>> Users already have to provide 4 properties (dtdev, iomem, irqs,
>> device_tree) to setup device assignment. I think that making it 5
>> properties would not be a step forward :-)
>
> IIRC, in the past, we discussed to fetch the information directly from
> the partial device-tree. Maybe this discussion needs to be revived?
>
> Although, this is a separate topic from this series.
>
>> To me dtdev and XEN_DOMCTL_assign_device are appropriate because
>> they
>> signify device assignment of one or more devices. We are not adding any
>> additional "meaning" to them. It is just that we'll automatically detect
>> and generate any SCMI configurations based on the list of assigned
>> devices. Because if SCMI is enabled and a device is assigned to the
>> guest, then I think we want to add the device to the SCMI list of
>> devices automatically.
>
> I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
> a pitfall with that approach.
>
> At the moment, they are only used for device protected by the
> IOMMU. If the device is not protected by the IOMMU then it will return
> an error.
IIRC there was a change, that allowed to assign device without a IOMMU. At
least we discussed this internally.
>
> Now, with your approach we may have a device that is not protected by
> the IOMMU but require to a SCMI configuration.
You need to protect only DMA-capable devices.
> I don't think it would be sensible to just return "succeed" here
> because technically you are asking to assign a non-protected
> device. But at the same time, it would prevent a user to assign a
> non-DMA capable device.
>
> So how do you suggest to approach this?
Well, in my head assign_device ideally should do the following:
1. Assign IOMMU if it is configured for the device
2. Assign SCMI access rights
(Not related to this patch series, but...)
3. Assign IRQs
4. Assign IO memory ranges.
Points 3. and 4. would allow us to not provide additional irq=[] and
iomem=[] entries in a guest config.
--
Volodymyr Babchuk at EPAM
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-22 11:17 ` Volodymyr Babchuk
@ 2021-12-22 11:30 ` Julien Grall
2021-12-22 12:34 ` Volodymyr Babchuk
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-22 11:30 UTC (permalink / raw)
To: Volodymyr Babchuk
Cc: Stefano Stabellini, Anthony PERARD, Oleksii Moisieiev, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi,
On 22/12/2021 12:17, Volodymyr Babchuk wrote:
> Julien Grall <julien@xen.org> writes:
>> On 21/12/2021 22:39, Stefano Stabellini wrote:
>>> On Tue, 21 Dec 2021, Anthony PERARD wrote:
>>>> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
>>>>>> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>>>>>>> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
>>>>>>> {
>>>>>>> AO_GC;
>>>>>>> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
>>>>>>> - int i, rc = 0;
>>>>>>> + int i, rc = 0, rc_sci = 0;
>>>>>>> for (i = 0; i < d_config->num_dtdevs; i++) {
>>>>>>> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>>>>>>> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
>>>>>>> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
>>>>>>> - if (rc < 0) {
>>>>>>> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
>>>>>>> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
>>>>>>> +
>>>>>>> + if ((rc < 0) && (rc_sci < 0)) {
>>>>>>> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
>>>>>>> + "xc_domain_add_sci_device failed: %d",
>>>>>>> + rc, rc_sci);
>>>>>>> goto out;
>>>>>>> }
>>>>>>> +
>>>>>>> + if (rc)
>>>>>>> + rc = rc_sci;
>>>>>>
>>>>>>
>>>>>> If I get this code right, it sounds like the dom.cfg's dtdev property is
>>>>>> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
>>>>>> (and can not) differentiate them. So it has no option but to send two
>>>>>> domctls for each device in dtdevs[] hoping to at least one domctl to
>>>>>> succeeded. Or I really missed something?
>>>>>>
>>>>>> It feels to me that:
>>>>>> - either new dom.cfg's property could be introduced (scidev?) to describe
>>>>>> sci devices together with new parsing logic/management code, so you will end
>>>>>> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
>>>>>> so no mixing things.
>>>>>> - or indeed dtdev logic could be *completely* reused including extending
>>>>>> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
>>>>>> is used to describe devices for the passthrough (to configure an IOMMU for
>>>>>> the device), in that case you wouldn't need an extra
>>>>>> XEN_DOMCTL_add_sci_device introduced by current patch.
>>> I realize I did my review before reading Oleksandr's comments. I
>>> fully
>>> agree with his feedback. Having seen how difficult is for users to setup
>>> a domU configuration correctly today, I would advise to try to reuse the
>>> existing dtdev property instead of adding yet one new property to make
>>> the life of our users easier.
>>>
>>>
>>>>>> Personally I would use the first option as I am not sure that second option
>>>>>> is conceptually correct && possible. I would leave this for the maintainers
>>>>>> to clarify.
>>>>>>
>>>>>
>>>>> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
>>>>> seems not to be conceptually correct. Introducing new dom.cfg property
>>>>> seems to be the only way to avoid extra Domctl calls.
>>>>> I will handle this in v2 if there will be no complains from Maintainers.
>>>>
>>>> I don't know if there will be a complains in v2 or not :-), but using
>>>> something different from dtdev sound good.
>>>>
>>>> If I understand correctly, dtdev seems to be about passing through an
>>>> existing device to a guest, and this new sci device seems to be about having Xen
>>>> "emulating" an sci device which the guest will use. So introducing
>>>> something new (scidev or other name) sounds good.
>>> Users already have to provide 4 properties (dtdev, iomem, irqs,
>>> device_tree) to setup device assignment. I think that making it 5
>>> properties would not be a step forward :-)
>>
>> IIRC, in the past, we discussed to fetch the information directly from
>> the partial device-tree. Maybe this discussion needs to be revived?
>>
>> Although, this is a separate topic from this series.
>>
>>> To me dtdev and XEN_DOMCTL_assign_device are appropriate because
>>> they
>>> signify device assignment of one or more devices. We are not adding any
>>> additional "meaning" to them. It is just that we'll automatically detect
>>> and generate any SCMI configurations based on the list of assigned
>>> devices. Because if SCMI is enabled and a device is assigned to the
>>> guest, then I think we want to add the device to the SCMI list of
>>> devices automatically.
>>
>> I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
>> a pitfall with that approach.
>>
>> At the moment, they are only used for device protected by the
>> IOMMU. If the device is not protected by the IOMMU then it will return
>> an error.
>
> IIRC there was a change, that allowed to assign device without a IOMMU. At
> least we discussed this internally.
I am not aware of any upstream. Do you have a pointer if there is any
public discussion?
>>
>> Now, with your approach we may have a device that is not protected by
>> the IOMMU but require to a SCMI configuration.
>
> You need to protect only DMA-capable devices.
Xen doesn't know if the device is DMA-capable or not. So...
>
>> I don't think it would be sensible to just return "succeed" here
>> because technically you are asking to assign a non-protected
>> device. But at the same time, it would prevent a user to assign a
>> non-DMA capable device.
>>
>> So how do you suggest to approach this?
>
> Well, in my head assign_device ideally should do the following:
>
> 1. Assign IOMMU if it is configured for the device
... with this approach you are at the risk to let the user passthrough a
device that cannot be protected.
> 2. Assign SCMI access rights
> (Not related to this patch series, but...)
> 3. Assign IRQs
> 4. Assign IO memory ranges.
>
> Points 3. and 4. would allow us to not provide additional irq=[] and
> iomem=[] entries in a guest config.
That could only work if your guest is using the same layout as the host.
Otherwise, there is a risk it will clash with other parts of the memory
layout.
Today, guests started via the toolstack is only using a virtual layout,
so you would first need to add support to use the host memory layout.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-21 13:27 ` Anthony PERARD
@ 2021-12-22 12:20 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-22 12:20 UTC (permalink / raw)
To: Anthony PERARD; +Cc: xen-devel, Wei Liu, Juergen Gross
On Tue, Dec 21, 2021 at 01:27:11PM +0000, Anthony PERARD wrote:
> On Tue, Dec 14, 2021 at 09:34:28AM +0000, Oleksii Moisieiev wrote:
> > This enumeration sets SCI type for the domain. Currently there is
> > two possible options: either 'none' or 'scmi_smc'.
> >
> > 'none' is the default value and it disables SCI support at all.
> >
> > 'scmi_smc' enables access to the Firmware from the domains using SCMI
> > protocol and SMC as transport.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
>
> Thanks for the patch, it looks good too me.
>
> But it is kind of weird that the manual describes something that isn't
> implemented yet. Could you maybe add in the patch description that the
> feature isn't implemented yet or that the feature is implemented in
> follow-up patches?
>
> Also, about the golang binding thingy, could you add a note after a line
> of a three dash "---" that let know the committer to regenerate
> everything that needs re-generating due to change in the *.idl file,
> just in case?
>
Hi Anthony,
I will add golang bindings in v2.
As for the scmi_smc description - is sounds reasonable. I will refactor
it in v2.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-22 11:30 ` Julien Grall
@ 2021-12-22 12:34 ` Volodymyr Babchuk
2021-12-22 13:49 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Volodymyr Babchuk @ 2021-12-22 12:34 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Anthony PERARD, Oleksii Moisieiev, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Julien Grall <julien@xen.org> writes:
> Hi,
>
> On 22/12/2021 12:17, Volodymyr Babchuk wrote:
>> Julien Grall <julien@xen.org> writes:
>>> On 21/12/2021 22:39, Stefano Stabellini wrote:
>>>> On Tue, 21 Dec 2021, Anthony PERARD wrote:
>>>>> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
>>>>>>> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>>>>>>>> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
>>>>>>>> {
>>>>>>>> AO_GC;
>>>>>>>> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
>>>>>>>> - int i, rc = 0;
>>>>>>>> + int i, rc = 0, rc_sci = 0;
>>>>>>>> for (i = 0; i < d_config->num_dtdevs; i++) {
>>>>>>>> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>>>>>>>> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
>>>>>>>> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
>>>>>>>> - if (rc < 0) {
>>>>>>>> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
>>>>>>>> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
>>>>>>>> +
>>>>>>>> + if ((rc < 0) && (rc_sci < 0)) {
>>>>>>>> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
>>>>>>>> + "xc_domain_add_sci_device failed: %d",
>>>>>>>> + rc, rc_sci);
>>>>>>>> goto out;
>>>>>>>> }
>>>>>>>> +
>>>>>>>> + if (rc)
>>>>>>>> + rc = rc_sci;
>>>>>>>
>>>>>>>
>>>>>>> If I get this code right, it sounds like the dom.cfg's dtdev property is
>>>>>>> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
>>>>>>> (and can not) differentiate them. So it has no option but to send two
>>>>>>> domctls for each device in dtdevs[] hoping to at least one domctl to
>>>>>>> succeeded. Or I really missed something?
>>>>>>>
>>>>>>> It feels to me that:
>>>>>>> - either new dom.cfg's property could be introduced (scidev?) to describe
>>>>>>> sci devices together with new parsing logic/management code, so you will end
>>>>>>> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
>>>>>>> so no mixing things.
>>>>>>> - or indeed dtdev logic could be *completely* reused including extending
>>>>>>> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
>>>>>>> is used to describe devices for the passthrough (to configure an IOMMU for
>>>>>>> the device), in that case you wouldn't need an extra
>>>>>>> XEN_DOMCTL_add_sci_device introduced by current patch.
>>>> I realize I did my review before reading Oleksandr's comments. I
>>>> fully
>>>> agree with his feedback. Having seen how difficult is for users to setup
>>>> a domU configuration correctly today, I would advise to try to reuse the
>>>> existing dtdev property instead of adding yet one new property to make
>>>> the life of our users easier.
>>>>
>>>>
>>>>>>> Personally I would use the first option as I am not sure that second option
>>>>>>> is conceptually correct && possible. I would leave this for the maintainers
>>>>>>> to clarify.
>>>>>>>
>>>>>>
>>>>>> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
>>>>>> seems not to be conceptually correct. Introducing new dom.cfg property
>>>>>> seems to be the only way to avoid extra Domctl calls.
>>>>>> I will handle this in v2 if there will be no complains from Maintainers.
>>>>>
>>>>> I don't know if there will be a complains in v2 or not :-), but using
>>>>> something different from dtdev sound good.
>>>>>
>>>>> If I understand correctly, dtdev seems to be about passing through an
>>>>> existing device to a guest, and this new sci device seems to be about having Xen
>>>>> "emulating" an sci device which the guest will use. So introducing
>>>>> something new (scidev or other name) sounds good.
>>>> Users already have to provide 4 properties (dtdev, iomem, irqs,
>>>> device_tree) to setup device assignment. I think that making it 5
>>>> properties would not be a step forward :-)
>>>
>>> IIRC, in the past, we discussed to fetch the information directly from
>>> the partial device-tree. Maybe this discussion needs to be revived?
>>>
>>> Although, this is a separate topic from this series.
>>>
>>>> To me dtdev and XEN_DOMCTL_assign_device are appropriate because
>>>> they
>>>> signify device assignment of one or more devices. We are not adding any
>>>> additional "meaning" to them. It is just that we'll automatically detect
>>>> and generate any SCMI configurations based on the list of assigned
>>>> devices. Because if SCMI is enabled and a device is assigned to the
>>>> guest, then I think we want to add the device to the SCMI list of
>>>> devices automatically.
>>>
>>> I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
>>> a pitfall with that approach.
>>>
>>> At the moment, they are only used for device protected by the
>>> IOMMU. If the device is not protected by the IOMMU then it will return
>>> an error.
>> IIRC there was a change, that allowed to assign device without a
>> IOMMU. At
>> least we discussed this internally.
>
> I am not aware of any upstream. Do you have a pointer if there is any
> public discussion?
No, this is the first public discussion on this topic.
>
>>>
>>> Now, with your approach we may have a device that is not protected by
>>> the IOMMU but require to a SCMI configuration.
>> You need to protect only DMA-capable devices.
>
> Xen doesn't know if the device is DMA-capable or not. So...
>
But it knows if there is a iommus=<> node present for the device.
>>
>>> I don't think it would be sensible to just return "succeed" here
>>> because technically you are asking to assign a non-protected
>>> device. But at the same time, it would prevent a user to assign a
>>> non-DMA capable device.
>>>
>>> So how do you suggest to approach this?
>> Well, in my head assign_device ideally should do the following:
>> 1. Assign IOMMU if it is configured for the device
>
> ... with this approach you are at the risk to let the user passthrough
> a device that cannot be protected.
>
>> 2. Assign SCMI access rights
>> (Not related to this patch series, but...)
>> 3. Assign IRQs
>> 4. Assign IO memory ranges.
>> Points 3. and 4. would allow us to not provide additional irq=[] and
>> iomem=[] entries in a guest config.
>
> That could only work if your guest is using the same layout as the
> host.
Agreed. But in my experience, in most cases this is the true. We worked
with Renesas and IMX hardware and in both cases iomems were
mapped 1:1.
> Otherwise, there is a risk it will clash with other parts of the
> memory layout.
>
> Today, guests started via the toolstack is only using a virtual
> layout, so you would first need to add support to use the host memory
> layout.
I can't say for all the possible configurations in the wild, but I'm
assuming that in most cases it will be fine to use 1:1 mapping. For
those more exotic cases it would be possible to implement some
configuration option like iomem_map=[mfn:gfn].
As Stefano pointed, right now user needs to provide 3 configuration
options to pass a device to a guest: dt_dev, irq, iomem. But in fact,
Xen is already knowing about irq and iomem from device tree.
--
Volodymyr Babchuk at EPAM
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-21 1:37 ` Stefano Stabellini
@ 2021-12-22 13:41 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-22 13:41 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Anthony PERARD, Volodymyr Babchuk,
Bertrand Marquis
Hi Stefano,
On Mon, Dec 20, 2021 at 05:37:50PM -0800, Stefano Stabellini wrote:
> On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > Integration of the SCMI mediator with xen libs:
> > - add hypercalls to aquire SCI channel and set device permissions
> > for DomUs;
> > - add SCMI_SMC nodes to DomUs device-tree based on partial device-tree;
> > - SCI requests redirection from DomUs to Firmware.
> >
> > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > ---
> > tools/include/xenctrl.h | 3 +
> > tools/include/xenguest.h | 2 +
> > tools/libs/ctrl/xc_domain.c | 23 ++++++
> > tools/libs/guest/xg_dom_arm.c | 5 +-
> > tools/libs/light/libxl_arm.c | 122 +++++++++++++++++++++++++++---
> > tools/libs/light/libxl_create.c | 54 ++++++++++++-
> > tools/libs/light/libxl_dom.c | 1 +
> > tools/libs/light/libxl_internal.h | 4 +
> > xen/arch/arm/domctl.c | 15 ++++
> > xen/include/public/domctl.h | 9 +++
> > 10 files changed, 223 insertions(+), 15 deletions(-)
> >
> > diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
> > index 07b96e6671..cdd14f465f 100644
> > --- a/tools/include/xenctrl.h
> > +++ b/tools/include/xenctrl.h
> > @@ -1238,6 +1238,9 @@ int xc_domain_getvnuma(xc_interface *xch,
> > int xc_domain_soft_reset(xc_interface *xch,
> > uint32_t domid);
> >
> > +int xc_domain_add_sci_device(xc_interface *xch,
> > + uint32_t domid, char *path);
> > +
> > #if defined(__i386__) || defined(__x86_64__)
> > /*
> > * PC BIOS standard E820 types and structure.
> > diff --git a/tools/include/xenguest.h b/tools/include/xenguest.h
> > index 61d0a82f48..35c611ac73 100644
> > --- a/tools/include/xenguest.h
> > +++ b/tools/include/xenguest.h
> > @@ -242,6 +242,8 @@ struct xc_dom_image {
> >
> > /* Number of vCPUs */
> > unsigned int max_vcpus;
> > +
> > + xen_pfn_t sci_shmem_gfn;
>
> We should be able to avoid introducing sci_shmem_gfn (more below)
>
>
> > };
> >
> > /* --- arch specific hooks ----------------------------------------- */
> > diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
> > index b155d6afd2..07bb390e17 100644
> > --- a/tools/libs/ctrl/xc_domain.c
> > +++ b/tools/libs/ctrl/xc_domain.c
> > @@ -2206,6 +2206,29 @@ int xc_domain_soft_reset(xc_interface *xch,
> > domctl.domain = domid;
> > return do_domctl(xch, &domctl);
> > }
> > +
> > +int xc_domain_add_sci_device(xc_interface *xch,
> > + uint32_t domid, char *path)
> > +{
> > + size_t size = strlen(path);
> > + int rc;
> > + DECLARE_DOMCTL;
> > + DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
> > +
> > + if ( xc_hypercall_bounce_pre(xch, path) )
> > + return -1;
> > +
> > + domctl.cmd = XEN_DOMCTL_add_sci_device;
> > + domctl.domain = domid;
> > + domctl.u.sci_device_op.size = size;
> > + set_xen_guest_handle(domctl.u.sci_device_op.path, path);
> > + rc = do_domctl(xch, &domctl);
> > +
> > + xc_hypercall_bounce_post(xch, path);
> > +
> > + return rc;
> > +}
>
> I'd skip this xc_ function and hypercall (more below)
>
>
> > /*
> > * Local variables:
> > * mode: C
> > diff --git a/tools/libs/guest/xg_dom_arm.c b/tools/libs/guest/xg_dom_arm.c
> > index 5e3b76355e..368a670c46 100644
> > --- a/tools/libs/guest/xg_dom_arm.c
> > +++ b/tools/libs/guest/xg_dom_arm.c
> > @@ -25,11 +25,12 @@
> >
> > #include "xg_private.h"
> >
> > -#define NR_MAGIC_PAGES 4
> > +#define NR_MAGIC_PAGES 5
> > #define CONSOLE_PFN_OFFSET 0
> > #define XENSTORE_PFN_OFFSET 1
> > #define MEMACCESS_PFN_OFFSET 2
> > #define VUART_PFN_OFFSET 3
> > +#define SCI_SHMEM_OFFSET 4
> >
> > #define LPAE_SHIFT 9
> >
> > @@ -69,11 +70,13 @@ static int alloc_magic_pages(struct xc_dom_image *dom)
> > dom->console_pfn = base + CONSOLE_PFN_OFFSET;
> > dom->xenstore_pfn = base + XENSTORE_PFN_OFFSET;
> > dom->vuart_gfn = base + VUART_PFN_OFFSET;
> > + dom->sci_shmem_gfn = base + SCI_SHMEM_OFFSET;
> >
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->console_pfn);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->xenstore_pfn);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, base + MEMACCESS_PFN_OFFSET);
> > xc_clear_domain_page(dom->xch, dom->guest_domid, dom->vuart_gfn);
> > + xc_clear_domain_page(dom->xch, dom->guest_domid, dom->sci_shmem_gfn);
>
> Given that sci_shmem_gfn doesn't actually require any allocations, just
> a remapping of an existing and already specified physical page, then I
> don't think we need to add a new page to alloc_magic_pages for it.
>
> We can just #define a static address for the SCMI page in the domU
> addres space and use it for the mapping. No need to allocate a new
> page.
>
I think this could be implemented. I will refactor this part in v2.
>
> > xc_hvm_param_set(dom->xch, dom->guest_domid, HVM_PARAM_CONSOLE_PFN,
> > dom->console_pfn);
> > diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> > index eef1de0939..73ef591822 100644
> > --- a/tools/libs/light/libxl_arm.c
> > +++ b/tools/libs/light/libxl_arm.c
> > @@ -8,6 +8,8 @@
> > #include <assert.h>
> > #include <xen/device_tree_defs.h>
> >
> > +#define SCMI_NODE_PATH "/firmware/scmi"
> > +
> > static const char *gicv_to_string(libxl_gic_version gic_version)
> > {
> > switch (gic_version) {
> > @@ -101,6 +103,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
> > return ERROR_FAIL;
> > }
> >
> > + switch (d_config->b_info.sci) {
> > + case LIBXL_SCI_TYPE_NONE:
> > + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_NONE;
> > + break;
> > + case LIBXL_SCI_TYPE_SCMI_SMC:
> > + config->arch.sci_type = XEN_DOMCTL_CONFIG_SCI_SCMI_SMC;
> > + break;
> > + default:
> > + LOG(ERROR, "Unknown SCI type %d",
> > + d_config->b_info.sci);
> > + return ERROR_FAIL;
> > + }
> > +
> > return 0;
> > }
> >
> > @@ -122,6 +137,7 @@ int libxl__arch_domain_save_config(libxl__gc *gc,
> > }
> >
> > state->clock_frequency = config->arch.clock_frequency;
> > + state->sci_agent_paddr = config->arch.sci_agent_paddr;
> >
> > return 0;
> > }
> > @@ -502,9 +518,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> > int res;
> > LOG(DEBUG, "Creating OP-TEE node in dtb");
> >
> > - res = fdt_begin_node(fdt, "firmware");
> > - if (res) return res;
> > -
> > res = fdt_begin_node(fdt, "optee");
> > if (res) return res;
> >
> > @@ -517,9 +530,6 @@ static int make_optee_node(libxl__gc *gc, void *fdt)
> > res = fdt_end_node(fdt);
> > if (res) return res;
> >
> > - res = fdt_end_node(fdt);
> > - if (res) return res;
> > -
> > return 0;
> > }
> >
> > @@ -902,10 +912,9 @@ static int copy_node(libxl__gc *gc, void *fdt, void *pfdt,
> > return 0;
> > }
> >
> > -static int copy_node_by_path(libxl__gc *gc, const char *path,
> > - void *fdt, void *pfdt)
> > +static int get_path_nodeoff(const char *path, void *pfdt)
> > {
> > - int nodeoff, r;
> > + int nodeoff;
> > const char *name = strrchr(path, '/');
> >
> > if (!name)
> > @@ -925,12 +934,101 @@ static int copy_node_by_path(libxl__gc *gc, const char *path,
> > if (strcmp(fdt_get_name(pfdt, nodeoff, NULL), name))
> > return -FDT_ERR_NOTFOUND;
> >
> > + return nodeoff;
> > +}
> > +
> > +static int copy_node_by_path(libxl__gc *gc, const char *path,
> > + void *fdt, void *pfdt)
> > +{
> > + int nodeoff, r;
> > +
> > + nodeoff = get_path_nodeoff(path, pfdt);
> > + if (nodeoff < 0)
> > + return nodeoff;
> > +
> > r = copy_node(gc, fdt, pfdt, nodeoff, 0);
> > if (r) return r;
> >
> > return 0;
> > }
> >
> > +static int get_node_phandle(const char *path, void *pfdt, uint32_t *phandle)
> > +{
> > + int nodeoff;
> > + const char *name = strrchr(path, '/');
> > +
> > + if (!name)
> > + return -FDT_ERR_INTERNAL;
> > +
> > + name++;
> > + nodeoff = fdt_path_offset(pfdt, path);
> > + if (nodeoff < 0)
> > + return nodeoff;
> > +
> > + *phandle = fdt_get_phandle(pfdt, nodeoff);
> > + return 0;
> > +}
> > +
> > +static int make_scmi_shmem_node(libxl__gc *gc, void *fdt, void *pfdt,
> > + struct xc_dom_image *dom)
> > +{
> > + int res;
> > + char buf[64];
> > + uint32_t phandle = 0;
> > +
> > + res = get_node_phandle("/scp-shmem", pfdt, &phandle);
> > + if (res) return res;
>
> I hope we'll be able to avoid requiring the user to write a partial
> device tree with SCMI info in order to use it.
I assuming that all SCMI related data is already present in partial
device-tree. Because all scmi nodes should be added to the original
platform device tree, which is used to build domu and dom0 dtbs.
So the structure should look the following way, taken h3ulcb as an
example;
r8a77951-scmi.dtsi - describe all scmi nodes and update
clock/power/reset etc in the device nodes;
|
\ /
included in r8a77951-ulcb.dts - device tree for rcar h3ulcb board
|
\ /
included in r8a77951-ulcb-xen.dts - xen dts files;
included in r8a77951-ulcb-domu.dts - domu dts file.
So the idea is that scmi configuration should be applied for both
virtualized and non-virtualized systems.
That's why I copy nodes from partial device-tree to domain device-tree.
Another advantage of this approach is that user can configure scmi node
for each domain by add/remove protocols. For example only clock and
resets could work through scmi for DomX. This can easily be configured
when using partial device-tree.
>
> But if we have to go down this route, then we need to add a description
> of scp-shmem under docs/misc/arm/
>
> Also, in general, we cannot mandate or require specific paths in device
> tree and instead we should find nodes based on the compatible string.
> (There are exceptions like /reserved-memory and /firmware but they are
> only a couple.)
>
I agree about the specific path. I will make it to use compatible string
in v2.
>
> > + snprintf(buf, sizeof(buf), "scp-shmem@%lx",
> > + dom->sci_shmem_gfn << XC_PAGE_SHIFT);
> > + res = fdt_begin_node(fdt, buf);
> > + if (res) return res;
> > +
> > + res = fdt_property_compat(gc, fdt, 1, "arm,scmi-shmem");
> > + if (res) return res;
> > +
> > + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> > + GUEST_ROOT_SIZE_CELLS, 1,
> > + dom->sci_shmem_gfn << XC_PAGE_SHIFT, XC_PAGE_SHIFT);
> > + if (res) return res;
> > +
> > + LOG(DEBUG, "scmi: setting phandle = %u\n", phandle);
> > + res = fdt_property_cell(fdt, "phandle", phandle);
> > + if (res) return res;
> > +
> > + res = fdt_end_node(fdt);
> > + if (res) return res;
> > +
> > + return 0;
> > +}
> > +
> > +static int make_firmware_node(libxl__gc *gc, void *fdt, void *pfdt, int tee,
> > + int sci)
> > +{
> > + int res;
> > +
> > + if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci != LIBXL_SCI_TYPE_NONE))
> > + return 0;
>
> Shouldn't this be:
>
> if ((tee != LIBXL_TEE_TYPE_OPTEE) && (sci == LIBXL_SCI_TYPE_NONE))
>
Yeah, I should fix this in v2. Thanks.
>
> > + res = fdt_begin_node(fdt, "firmware");
> > + if (res) return res;
> > +
> > + if (tee == LIBXL_TEE_TYPE_OPTEE) {
> > + res = make_optee_node(gc, fdt);
> > + if (res) return res;
> > + }
> > +
> > + if (sci == LIBXL_SCI_TYPE_SCMI_SMC) {
> > + res = copy_node_by_path(gc, SCMI_NODE_PATH, fdt, pfdt);
> > + if (res) return res;
> > + }
>
> Do we really need to copy the node from the partial device tree? That
> makes it a lot harder to use for the user. Ideally a user would only
> need to specify sci = "scmi_smc" and everything else should be done
> automatically.
>
> Can we automatically generate the SCMI device tree node instead? It
> looks like we should have all the information to be able to do it. If
> not, what is missing?
>
Please see the answer above.
>
> > + res = fdt_end_node(fdt);
> > + if (res) return res;
> > +
> > + return 0;
> > +}
> > +
> > /*
> > * The partial device tree is not copied entirely. Only the relevant bits are
> > * copied to the guest device tree:
> > @@ -1088,8 +1186,10 @@ next_resize:
> > if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART)
> > FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) );
> >
> > - if (info->tee == LIBXL_TEE_TYPE_OPTEE)
> > - FDT( make_optee_node(gc, fdt) );
> > + FDT( make_firmware_node(gc, fdt, pfdt, info->tee, info->sci) );
> > +
> > + if (info->sci == LIBXL_SCI_TYPE_SCMI_SMC)
> > + FDT( make_scmi_shmem_node(gc, fdt, pfdt, dom) );
> >
> > if (d_config->num_pcidevs)
> > FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> > diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
> > index dcd09d32ba..c7372fd344 100644
> > --- a/tools/libs/light/libxl_create.c
> > +++ b/tools/libs/light/libxl_create.c
> > @@ -596,6 +596,37 @@ out:
> > return ret;
> > }
> >
> > +static int map_sci_page(libxl__gc *gc, uint32_t domid, uint64_t paddr,
> > + uint64_t guest_addr)
> > +{
> > + int ret;
> > + uint64_t _paddr_pfn = paddr >> XC_PAGE_SHIFT;
> > + uint64_t _guest_pfn = guest_addr >> XC_PAGE_SHIFT;
> > +
> > + LOG(DEBUG, "iomem %"PRIx64, _paddr_pfn);
> > +
> > + ret = xc_domain_iomem_permission(CTX->xch, domid, _paddr_pfn, 1, 1);
> > + if (ret < 0) {
> > + LOG(ERROR,
> > + "failed give domain access to iomem page %"PRIx64,
> > + _paddr_pfn);
> > + return ret;
> > + }
> > +
> > + ret = xc_domain_memory_mapping(CTX->xch, domid,
> > + _guest_pfn, _paddr_pfn,
> > + 1, 1);
> > + if (ret < 0) {
> > + LOG(ERROR,
> > + "failed to map to domain iomem page %"PRIx64
> > + " to guest address %"PRIx64,
> > + _paddr_pfn, _guest_pfn);
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> > libxl__domain_build_state *state,
> > uint32_t *domid, bool soft_reset)
> > @@ -762,6 +793,16 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
> > goto out;
> > }
> >
> > + if (state->sci_agent_paddr != 0) {
>
> Shouldn't we also check for sci_type == XEN_DOMCTL_CONFIG_SCI_NONE ?
>
> If the user specifies sci_type == XEN_DOMCTL_CONFIG_SCI_SCMI_SMC, we
> shouldn't automatically map any SCMI channels to the guest, right?
>
Sounds good. I will fix this in v2`
>
> > + ret = map_sci_page(gc, *domid, state->sci_agent_paddr,
> > + state->sci_shmem_gfn << XC_PAGE_SHIFT);
> > + if (ret < 0) {
> > + LOGED(ERROR, *domid, "map SCI page fail");
> > + rc = ERROR_FAIL;
> > + goto out;
> > + }
> > + }
> > +
> > dom_path = libxl__xs_get_dompath(gc, *domid);
> > if (!dom_path) {
> > rc = ERROR_FAIL;
> > @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
> > {
> > AO_GC;
> > libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
> > - int i, rc = 0;
> > + int i, rc = 0, rc_sci = 0;
> >
> > for (i = 0; i < d_config->num_dtdevs; i++) {
> > const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
> >
> > LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
> > rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
> > - if (rc < 0) {
> > - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
> > + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
> > +
> > + if ((rc < 0) && (rc_sci < 0)) {
> > + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
> > + "xc_domain_add_sci_device failed: %d",
> > + rc, rc_sci);
> > goto out;
> > }
> > +
> > + if (rc)
> > + rc = rc_sci;
>
> I would make this simpler actually. Do we need to pass dtdev->path
> twice, once for xc_assign_dt_device and a second time for
> xc_domain_add_sci_device?
>
> I would skip adding xc_domain_add_sci_device altogether. A general SCMI
> enable/disable for the domain is necessary, but then we can just get the
> directly assigned devices from xc_assign_dt_device. There is no need to
> specify the list twice. If a device is not manageable via SCMI we can
> detect it automatically because it is going to be missing scmi_devid on
> device tree.
>
In one of my previous email I described the idea to make a list of
sci_devids in dom.cfg. Such as:
sci_devs = [0, 1, 35]
and pass this list to Xen via hypercall, then send permission requests
via scmi for each devid.
The advantages of this approach are the following:
- we don't need scmi_devid in device-tree anymore;
- we do only 1 hypercall to set permissions for devices;
- we don't need to pass device-tree node to Hypervisor, just devid.
What do you think about this approach?
>
> > }
> >
> > out:
> > diff --git a/tools/libs/light/libxl_dom.c b/tools/libs/light/libxl_dom.c
> > index fe9f760f71..b1d288a8b9 100644
> > --- a/tools/libs/light/libxl_dom.c
> > +++ b/tools/libs/light/libxl_dom.c
> > @@ -710,6 +710,7 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
> > state->console_mfn = dom->console_pfn;
> > state->store_mfn = dom->xenstore_pfn;
> > state->vuart_gfn = dom->vuart_gfn;
> > + state->sci_shmem_gfn = dom->sci_shmem_gfn;
> > } else {
> > state->console_mfn = xc_dom_p2m(dom, dom->console_pfn);
> > state->store_mfn = xc_dom_p2m(dom, dom->xenstore_pfn);
> > diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
> > index 0b4671318c..f9f9cc633a 100644
> > --- a/tools/libs/light/libxl_internal.h
> > +++ b/tools/libs/light/libxl_internal.h
> > @@ -1407,6 +1407,10 @@ typedef struct {
> > /* Whether this domain is being migrated/restored, or booting fresh. Only
> > * applicable to the primary domain, not support domains (e.g. stub QEMU). */
> > bool restore;
> > +
> > + /* sci channel paddr to be set to device-tree node */
> > + uint64_t sci_agent_paddr;
> > + xen_pfn_t sci_shmem_gfn;
> > } libxl__domain_build_state;
> >
> > _hidden void libxl__domain_build_state_init(libxl__domain_build_state *s);
> > diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
> > index 6245af6d0b..ba200407da 100644
> > --- a/xen/arch/arm/domctl.c
> > +++ b/xen/arch/arm/domctl.c
> > @@ -4,6 +4,7 @@
> > * Copyright (c) 2012, Citrix Systems
> > */
> >
> > +#include <asm/sci/sci.h>
> > #include <xen/errno.h>
> > #include <xen/guest_access.h>
> > #include <xen/hypercall.h>
> > @@ -175,6 +176,20 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
> >
> > return rc;
> > }
> > + case XEN_DOMCTL_add_sci_device:
> > + {
> > + int rc;
> > + struct dt_device_node *dev;
> > +
> > + rc = dt_find_node_by_gpath(domctl->u.sci_device_op.path,
> > + domctl->u.sci_device_op.size,
> > + &dev);
> > + if ( rc )
> > + return rc;
> > +
> > + return sci_add_dt_device(d, dev);
> > + }
>
> I would skip it and instead I would add a call to sci_add_dt_device in
> the implementation of XEN_DOMCTL_assign_device.
>
please see my comment above.
>
> > default:
> > {
> > int rc;
> > diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> > index b85e6170b0..671c72c3e8 100644
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -1177,6 +1177,13 @@ struct xen_domctl_vmtrace_op {
> > #define XEN_DOMCTL_vmtrace_get_option 5
> > #define XEN_DOMCTL_vmtrace_set_option 6
> > };
> > +
> > +/* XEN_DOMCTL_add_sci_device: set sci device permissions */
> > +struct xen_domctl_sci_device_op {
> > + uint32_t size; /* Length of the path */
> > + XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
> > +};
> > +
> > typedef struct xen_domctl_vmtrace_op xen_domctl_vmtrace_op_t;
> > DEFINE_XEN_GUEST_HANDLE(xen_domctl_vmtrace_op_t);
> >
> > @@ -1265,6 +1272,7 @@ struct xen_domctl {
> > #define XEN_DOMCTL_get_cpu_policy 82
> > #define XEN_DOMCTL_set_cpu_policy 83
> > #define XEN_DOMCTL_vmtrace_op 84
> > +#define XEN_DOMCTL_add_sci_device 85
> > #define XEN_DOMCTL_gdbsx_guestmemio 1000
> > #define XEN_DOMCTL_gdbsx_pausevcpu 1001
> > #define XEN_DOMCTL_gdbsx_unpausevcpu 1002
> > @@ -1326,6 +1334,7 @@ struct xen_domctl {
> > struct xen_domctl_psr_alloc psr_alloc;
> > struct xen_domctl_vuart_op vuart_op;
> > struct xen_domctl_vmtrace_op vmtrace_op;
> > + struct xen_domctl_sci_device_op sci_device_op;
> > uint8_t pad[128];
> > } u;
> > };
> > --
> > 2.27.0
> >
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-22 12:34 ` Volodymyr Babchuk
@ 2021-12-22 13:49 ` Julien Grall
2021-12-23 2:23 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-22 13:49 UTC (permalink / raw)
To: Volodymyr Babchuk
Cc: Stefano Stabellini, Anthony PERARD, Oleksii Moisieiev, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi Volodymyr,
On 22/12/2021 13:34, Volodymyr Babchuk wrote:
>
>
> Julien Grall <julien@xen.org> writes:
>
>> Hi,
>>
>> On 22/12/2021 12:17, Volodymyr Babchuk wrote:
>>> Julien Grall <julien@xen.org> writes:
>>>> On 21/12/2021 22:39, Stefano Stabellini wrote:
>>>>> On Tue, 21 Dec 2021, Anthony PERARD wrote:
>>>>>> On Fri, Dec 17, 2021 at 12:15:25PM +0000, Oleksii Moisieiev wrote:
>>>>>>>> On 14.12.21 11:34, Oleksii Moisieiev wrote:
>>>>>>>>> @@ -1817,17 +1858,24 @@ static void libxl__add_dtdevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid,
>>>>>>>>> {
>>>>>>>>> AO_GC;
>>>>>>>>> libxl__ao_device *aodev = libxl__multidev_prepare(multidev);
>>>>>>>>> - int i, rc = 0;
>>>>>>>>> + int i, rc = 0, rc_sci = 0;
>>>>>>>>> for (i = 0; i < d_config->num_dtdevs; i++) {
>>>>>>>>> const libxl_device_dtdev *dtdev = &d_config->dtdevs[i];
>>>>>>>>> LOGD(DEBUG, domid, "Assign device \"%s\" to domain", dtdev->path);
>>>>>>>>> rc = xc_assign_dt_device(CTX->xch, domid, dtdev->path);
>>>>>>>>> - if (rc < 0) {
>>>>>>>>> - LOGD(ERROR, domid, "xc_assign_dtdevice failed: %d", rc);
>>>>>>>>> + rc_sci = xc_domain_add_sci_device(CTX->xch, domid, dtdev->path);
>>>>>>>>> +
>>>>>>>>> + if ((rc < 0) && (rc_sci < 0)) {
>>>>>>>>> + LOGD(ERROR, domid, "xc_assign_dt_device failed: %d; "
>>>>>>>>> + "xc_domain_add_sci_device failed: %d",
>>>>>>>>> + rc, rc_sci);
>>>>>>>>> goto out;
>>>>>>>>> }
>>>>>>>>> +
>>>>>>>>> + if (rc)
>>>>>>>>> + rc = rc_sci;
>>>>>>>>
>>>>>>>>
>>>>>>>> If I get this code right, it sounds like the dom.cfg's dtdev property is
>>>>>>>> reused to describe sci devices as well, but the libxl__add_dtdevs() does not
>>>>>>>> (and can not) differentiate them. So it has no option but to send two
>>>>>>>> domctls for each device in dtdevs[] hoping to at least one domctl to
>>>>>>>> succeeded. Or I really missed something?
>>>>>>>>
>>>>>>>> It feels to me that:
>>>>>>>> - either new dom.cfg's property could be introduced (scidev?) to describe
>>>>>>>> sci devices together with new parsing logic/management code, so you will end
>>>>>>>> up having new libxl__add_scidevs() to invoke XEN_DOMCTL_add_sci_device,
>>>>>>>> so no mixing things.
>>>>>>>> - or indeed dtdev logic could be *completely* reused including extending
>>>>>>>> XEN_DOMCTL_assign_device to cover your use-case, although sounds generic, it
>>>>>>>> is used to describe devices for the passthrough (to configure an IOMMU for
>>>>>>>> the device), in that case you wouldn't need an extra
>>>>>>>> XEN_DOMCTL_add_sci_device introduced by current patch.
>>>>> I realize I did my review before reading Oleksandr's comments. I
>>>>> fully
>>>>> agree with his feedback. Having seen how difficult is for users to setup
>>>>> a domU configuration correctly today, I would advise to try to reuse the
>>>>> existing dtdev property instead of adding yet one new property to make
>>>>> the life of our users easier.
>>>>>
>>>>>
>>>>>>>> Personally I would use the first option as I am not sure that second option
>>>>>>>> is conceptually correct && possible. I would leave this for the maintainers
>>>>>>>> to clarify.
>>>>>>>>
>>>>>>>
>>>>>>> Thank you for the point. I agree that reusing XEN_DOMCTL_assign_device
>>>>>>> seems not to be conceptually correct. Introducing new dom.cfg property
>>>>>>> seems to be the only way to avoid extra Domctl calls.
>>>>>>> I will handle this in v2 if there will be no complains from Maintainers.
>>>>>>
>>>>>> I don't know if there will be a complains in v2 or not :-), but using
>>>>>> something different from dtdev sound good.
>>>>>>
>>>>>> If I understand correctly, dtdev seems to be about passing through an
>>>>>> existing device to a guest, and this new sci device seems to be about having Xen
>>>>>> "emulating" an sci device which the guest will use. So introducing
>>>>>> something new (scidev or other name) sounds good.
>>>>> Users already have to provide 4 properties (dtdev, iomem, irqs,
>>>>> device_tree) to setup device assignment. I think that making it 5
>>>>> properties would not be a step forward :-)
>>>>
>>>> IIRC, in the past, we discussed to fetch the information directly from
>>>> the partial device-tree. Maybe this discussion needs to be revived?
>>>>
>>>> Although, this is a separate topic from this series.
>>>>
>>>>> To me dtdev and XEN_DOMCTL_assign_device are appropriate because
>>>>> they
>>>>> signify device assignment of one or more devices. We are not adding any
>>>>> additional "meaning" to them. It is just that we'll automatically detect
>>>>> and generate any SCMI configurations based on the list of assigned
>>>>> devices. Because if SCMI is enabled and a device is assigned to the
>>>>> guest, then I think we want to add the device to the SCMI list of
>>>>> devices automatically.
>>>>
>>>> I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
>>>> a pitfall with that approach.
>>>>
>>>> At the moment, they are only used for device protected by the
>>>> IOMMU. If the device is not protected by the IOMMU then it will return
>>>> an error.
>>> IIRC there was a change, that allowed to assign device without a
>>> IOMMU. At
>>> least we discussed this internally.
>>
>> I am not aware of any upstream. Do you have a pointer if there is any
>> public discussion?
>
> No, this is the first public discussion on this topic.
>
>>
>>>>
>>>> Now, with your approach we may have a device that is not protected by
>>>> the IOMMU but require to a SCMI configuration.
>>> You need to protect only DMA-capable devices.
>>
>> Xen doesn't know if the device is DMA-capable or not. So...
>>
>
> But it knows if there is a iommus=<> node present for the device.
Not really. Not all the platforms have IOMMUs and not all the platforms
with IOMMU have DMA-capable device protected by an IOMMU.
>
>>>
>>>> I don't think it would be sensible to just return "succeed" here
>>>> because technically you are asking to assign a non-protected
>>>> device. But at the same time, it would prevent a user to assign a
>>>> non-DMA capable device.
>>>>
>>>> So how do you suggest to approach this?
>>> Well, in my head assign_device ideally should do the following:
>>> 1. Assign IOMMU if it is configured for the device
>>
>> ... with this approach you are at the risk to let the user passthrough
>> a device that cannot be protected.
>>
>>> 2. Assign SCMI access rights
>>> (Not related to this patch series, but...)
>>> 3. Assign IRQs
>>> 4. Assign IO memory ranges.
>>> Points 3. and 4. would allow us to not provide additional irq=[] and
>>> iomem=[] entries in a guest config.
>>
>> That could only work if your guest is using the same layout as the
>> host.
>
> Agreed. But in my experience, in most cases this is the true. We worked
> with Renesas and IMX hardware and in both cases iomems were
> mapped 1:1.
>
>> Otherwise, there is a risk it will clash with other parts of the
>> memory layout.
>>
>
>> Today, guests started via the toolstack is only using a virtual
>> layout, so you would first need to add support to use the host memory
>> layout.
>
> I can't say for all the possible configurations in the wild, but I'm
> assuming that in most cases it will be fine to use 1:1 mapping. For
> those more exotic cases it would be possible to implement some
> configuration option like iomem_map=[mfn:gfn].
Well, the Xen memory layout is not something that is stable nor AFAIK
based on any memory layout. In fact, there is no such generic layout on Arm.
It is quite possible that it will work well with 1:1 MMIO on some
platform (e.g. Renesas, IMX) but you can't expect to work for every Xen
release or all the platforms.
>
> As Stefano pointed, right now user needs to provide 3 configuration
> options to pass a device to a guest: dt_dev, irq, iomem. But in fact,
> Xen is already knowing about irq and iomem from device tree.
I understand and this is not ideal. I would be happy to consider your
approach. However, this will have to enabled only when the guest is
re-using the host layout.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-22 13:49 ` Julien Grall
@ 2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 19:06 ` Stefano Stabellini
2021-12-23 19:11 ` Oleksii Moisieiev
0 siblings, 2 replies; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-23 2:23 UTC (permalink / raw)
To: Julien Grall
Cc: Volodymyr Babchuk, Stefano Stabellini, Anthony PERARD,
Oleksii Moisieiev, Oleksandr, xen-devel, Wei Liu, Juergen Gross,
Andrew Cooper, George Dunlap, Jan Beulich, Bertrand Marquis
On Wed, 22 Dec 2021, Julien Grall wrote:
> > > > > > To me dtdev and XEN_DOMCTL_assign_device are appropriate because
> > > > > > they
> > > > > > signify device assignment of one or more devices. We are not adding
> > > > > > any
> > > > > > additional "meaning" to them. It is just that we'll automatically
> > > > > > detect
> > > > > > and generate any SCMI configurations based on the list of assigned
> > > > > > devices. Because if SCMI is enabled and a device is assigned to the
> > > > > > guest, then I think we want to add the device to the SCMI list of
> > > > > > devices automatically.
> > > > >
> > > > > I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
> > > > > a pitfall with that approach.
> > > > >
> > > > > At the moment, they are only used for device protected by the
> > > > > IOMMU. If the device is not protected by the IOMMU then it will return
> > > > > an error.
> > > > IIRC there was a change, that allowed to assign device without a
> > > > IOMMU. At
> > > > least we discussed this internally.
> > >
> > > I am not aware of any upstream. Do you have a pointer if there is any
> > > public discussion?
> >
> > No, this is the first public discussion on this topic.
> >
> > >
> > > > >
> > > > > Now, with your approach we may have a device that is not protected by
> > > > > the IOMMU but require to a SCMI configuration.
> > > > You need to protect only DMA-capable devices.
> > >
> > > Xen doesn't know if the device is DMA-capable or not. So...
> > >
> >
> > But it knows if there is a iommus=<> node present for the device.
>
> Not really. Not all the platforms have IOMMUs and not all the platforms with
> IOMMU have DMA-capable device protected by an IOMMU.
>
> >
> > > >
> > > > > I don't think it would be sensible to just return "succeed" here
> > > > > because technically you are asking to assign a non-protected
> > > > > device. But at the same time, it would prevent a user to assign a
> > > > > non-DMA capable device.
> > > > >
> > > > > So how do you suggest to approach this?
> > > > Well, in my head assign_device ideally should do the following:
> > > > 1. Assign IOMMU if it is configured for the device
> > >
> > > ... with this approach you are at the risk to let the user passthrough
> > > a device that cannot be protected.
> > >
> > > > 2. Assign SCMI access rights
> > > > (Not related to this patch series, but...)
> > > > 3. Assign IRQs
> > > > 4. Assign IO memory ranges.
> > > > Points 3. and 4. would allow us to not provide additional irq=[] and
> > > > iomem=[] entries in a guest config.
> > >
> > > That could only work if your guest is using the same layout as the
> > > host.
> >
> > Agreed. But in my experience, in most cases this is the true. We worked
> > with Renesas and IMX hardware and in both cases iomems were
> > mapped 1:1.
> >
> > > Otherwise, there is a risk it will clash with other parts of the
> > > memory layout.
> > >
> >
> > > Today, guests started via the toolstack is only using a virtual
> > > layout, so you would first need to add support to use the host memory
> > > layout.
> >
> > I can't say for all the possible configurations in the wild, but I'm
> > assuming that in most cases it will be fine to use 1:1 mapping. For
> > those more exotic cases it would be possible to implement some
> > configuration option like iomem_map=[mfn:gfn].
> Well, the Xen memory layout is not something that is stable nor AFAIK based on
> any memory layout. In fact, there is no such generic layout on Arm.
>
> It is quite possible that it will work well with 1:1 MMIO on some platform
> (e.g. Renesas, IMX) but you can't expect to work for every Xen release or all
> the platforms.
Yeah this is a true problem. Thankfully with Penny's series we'll be
able to create domUs with the host memory layout (although dom0less
only but it is a step forward).
> > As Stefano pointed, right now user needs to provide 3 configuration
> > options to pass a device to a guest: dt_dev, irq, iomem. But in fact,
> > Xen is already knowing about irq and iomem from device tree.
>
> I understand and this is not ideal. I would be happy to consider your
> approach. However, this will have to enabled only when the guest is re-using
> the host layout.
It looks like we all agree that today configuring device assignment with
Xen is too complicated and would like for it to be simpler. This thread
has some excellent ideas on how to address the issue. Skip to the end if
you are only interested in this patch series.
# Future Ideas
A great suggestion by Julien is to start supporting the dom0less partial
device tree format in xl/libxl as well so that we can have a single
"device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
dtdev).
Even with that implemented, the user has still to provide a partial dtb,
xen,reg and xen,path. I think this is a great step forward and we should
do that, if nothing else to make it easier to switch between dom0less
and normal domU configurations. But the number of options and
information that the user has to provide is still similar to what we
have today.
After that, I think we need something along the lines of what Volodymyr
suggested. Let's say that the user only provides "dtdev" and
"device_tree" in dom.cfg. We could:
- read interrupts from the "interrupts" property like we do for dom0less
- read "xen,reg" for the mapping, if "xen,reg" is missing, read "reg"
and assume 1:1 (we could try the mapping and print an error on
failure, or we could only do 1:1 if the domain is entirely 1:1)
- optionally read "xen,path" to populate dtdev
- if an IOMMU configuration is present, then also configure the IOMMU
for the device, if not then "xen,force-assign-without-iommu" must be
present
- assign SCMI access rights
Now we only have:
- device_tree in dom.cfg
- dtdev in dom.cfg (or xen,path in the partial DTB)
- xen,force-assign-without-iommu in the partial DTB
It would be good if we could remove "xen,force-assign-without-iommu"
because at this stage it is the only Xen-specific property remaining in
the partial DTB. If we could get rid of it, it would make it easier to
write/generate the partial DTB because it becomes a strict subset of the
corresponding host node. It would enable us to automatically generate it
(we are going to do experiments with it at Xilinx in the next few
months) and it would be easier to explain to users how to write it.
The partial DTB usually starts as a copy of the corresponding host node
plus some edits. The fewer edits are required, the better.
But it is difficult because of the reasons mentioned by Julien. In Xen
we cannot know if a device is not behind an IOMMU because is not a DMA
master (so safe to assign) or because the system simply doesn't have
enough IOMMU coverage (so not safe to assign). Thankfully the hardware
has been improving in recent years and there are more and more platforms
with full IOMMU coverage. I think we should make it easier for users on
these well-behave platforms.
At least, we could move the "xen,force-assign-without-iommu" option from
the partial DTB to the VM config file dom.cfg. Something like:
force-assign-without-iommu="true"
or
platform-iommu-safe="true"
That way, it is global rather than per-device and doesn't have to be
added by the user by hand when creating the partial DTB.
# This patch series
Now in regards to this specific series and the SCMI options today, I
think it is OK to have a per-domain sci="scmi_smc", but I think we
should try to avoid another detailed list of device paths or list of
IDs in addition to dtdev.
dtdev already specifies the device tree nodes to be assigned to the
guest. Based on that list we can also do SCMI assignment.
As Julien pointed out, the issue is: what if a device needs SCMI
assignment but is not a DMA master (hence there is no IOMMU
configuration on the host)?
Attempting to assign a DMA mastering device without IOMMU protection is
not just unsafe, it will actively cause memory corruptions in most
cases. It is an erroneous configuration.
Of course we should try to stop people from running erroneous
configurations, but we should do so without penalizing all users.
With that in mind, also considering that dtdev is the list of devices to
be assigned, I think dtdev should be the list of all devices, even
non-DMA masters. It makes a lot of sense also because today is really
difficult to explain to a user that "yes, dtdev is the devices to be
assigned but no, if the device is a UART you don't have to add it to
dtdev because it is not a DMA master". It would be a lot easier if the
list of assigned devices was comprehensive, including both DMA masters
and not DMA masters.
So I think we should either:
a) extend dtdev to cover all devices, including non-DMA masters
b) or add a new "device_assignment" property which is like dtdev but is
the list of both DMA masters and non-DMA masters
Either way, when non-DMA masters are present in the
dtdev/device_assignment list we could either:
c) require force-assign-without-iommu="true" in dom.cfg
d) or print a warning like:
"WARNING: device assignment safety for device XXX cannot be
verified. Please make sure XXX is not a DMA mastering device."
All these options are good I think. My preference is a) + d), so extend
dtdev and print a warning if non-DMA masters are in the list. We don't
necessarily need a new hypercall but that is also an option.
I think this discussion was a long time coming and I am glad we are
having it now. We have a lot of room for improvement! I don't want to
scope-creep this patch series, but I hope that this last bit could be
done as part of this series if we find agreement in the community.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-22 11:04 ` Oleksii Moisieiev
@ 2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 18:45 ` Volodymyr Babchuk
` (2 more replies)
0 siblings, 3 replies; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-23 2:23 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > Hi Stefano,
> > >
> > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > Hi Stefano,
> > > > >
> > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > power-domains without access to CPG.
> > > > > > >
> > > > > > > The following features are implemented:
> > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > resets and power-domains via SCMI;
> > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > >
> > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > ---
> > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > >
> > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > SCI mediators implemented in XEN.
> > > > > > >
> > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > +
> > > > > > > endmenu
> > > > > > >
> > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > new file mode 100644
> > > > > > > index 0000000000..9563067ddc
> > > > > > > --- /dev/null
> > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > @@ -0,0 +1,10 @@
> > > > > > > +config SCMI_SMC
> > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > + default n
> > > > > > > + depends on SCI
> > > > > > > + ---help---
> > > > > > > +
> > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > + for communication.
> > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > @@ -1 +1,2 @@
> > > > > > > obj-y += sci.o
> > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > new file mode 100644
> > > > > > > index 0000000000..2eb01ea82d
> > > > > > > --- /dev/null
> > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > @@ -0,0 +1,795 @@
> > > > > > > +/*
> > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > + *
> > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > + *
> > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > + *
> > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > + * (at your option) any later version.
> > > > > > > + *
> > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > + * GNU General Public License for more details.
> > > > > > > + */
> > > > > > > +
> > > > > > > +#include <asm/sci/sci.h>
> > > > > > > +#include <asm/smccc.h>
> > > > > > > +#include <asm/io.h>
> > > > > > > +#include <xen/bitops.h>
> > > > > > > +#include <xen/config.h>
> > > > > > > +#include <xen/sched.h>
> > > > > > > +#include <xen/device_tree.h>
> > > > > > > +#include <xen/iocap.h>
> > > > > > > +#include <xen/init.h>
> > > > > > > +#include <xen/err.h>
> > > > > > > +#include <xen/lib.h>
> > > > > > > +#include <xen/list.h>
> > > > > > > +#include <xen/mm.h>
> > > > > > > +#include <xen/string.h>
> > > > > > > +#include <xen/time.h>
> > > > > > > +#include <xen/vmap.h>
> > > > > > > +
> > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > +
> > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > +
> > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > +
> > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > >
> > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > >
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > >
> > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > seems to be "arm,scmi-shmem".
> > > > > >
> > > > >
> > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > channels:
> > > > >
> > > > > reserved-memory {
> > > > > /* reserved region for scmi channels*/
> > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > no-map;
> > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > };
> > > > > };
> > > > >
> > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > the current scmi channel:
> > > > >
> > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > };
> > > > >
> > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > assigned to this agent.
> > > >
> > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > compatible string, not as a node name, and it would need to be described
> > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > >
> > > > But from your description I don't think it is necessary. We can just use
> > > > "arm,scmi-shmem" to describe all the required regions:
> > > >
> > > > reserved-memory {
> > > > scp-shmem@0x53FF0000 {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > };
> > > > scp-shmem@0x53FF1000 {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > };
> > > > scp-shmem@0x53FF2000 {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > };
> > > > ...
> > > >
> > > > In other words, if all the individual channel pages are described as
> > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > "linux,scmi_mem"?
> > > >
> > >
> > > That was my first implementation. But I've met a problem with
> > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > there were some if, checking if memory weren't reserved.
> > > That's why I ended up splitting nodes reserved memory region and actual
> > > shmem page.
> > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > which has no compatible string and provides no-map property.
> > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > space for the domain.
> > >
> > > Very interesting question about should I introduce linux,scmi_mem node
> > > and scmi_devid property to the
> > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > Those node and property are needed only for Xen and useless for
> > > non-virtualized systems. I can add this node and property description to
> > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > What do you think about it?
> >
> > Reply below
> >
> > [...]
> >
> >
> > > > In general we can't use properties that are not part of the device tree
> > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > >
> > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > activities to get "linux,scmi_mem" upstream under
> > > > Documentation/devicetree/bindings in Linux?
> > > >
> > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > Documentation/devicetree/bindings (probably
> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > work on the Xen code that makes use of it.
> > > >
> > > > Does it make sense?
> > > >
> > >
> > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> >
> > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > Xen specific. In general, it would be best not to introduce Xen specific
> > properties into generic bindings. It is a problem both from a
> > specification perspective (because it has hard to handle Xen specific
> > cases in fully generic bindings, especially as those bindings are
> > maintained as part of the Linux kernel) and from a user perspective
> > (because now the user has to deal with a Xen-specific dtb, or has to
> > modify the host dtb to add Xen-specific information by hand.)
> >
> >
> > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > looks like a generic property that should be needed for the Linux SCMI
> > driver too. Why the Linux driver doesn't need it?
> >
>
> scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> On non-virtualized systems - there is no need of this call, because OS
> is the only one entity, running on the system.
OK. Even if it is only required for virtualized systems, I think that
scmi_devid is important enough that should be part of the upstream
binding. I think it is worth starting an email thread on the LKML with
Rob Herring and the SCMI maintainers to discuss the addition of
scmi_devid to the binding.
> I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> list of device_ids to dom.cfg, such as:
> sci_devs = [ 0, 1, 15, 35 ];
>
> Using this approach, we can remove scmi_devid from the device tree and
> just pass a list of scmi_devids to XEN using additional hypercall.
> We can probably make hypercall taking devid list as input parameter.
> This will take only 1 hypercall to setup sci permissions.
But how would a user know which are the right SCMI IDs to add to the
sci_devs list? Would the user have to go and read the reference manual
of the platform to find the SCMI IDs and then write sci_devs by hand?
If that is the case, then I think that it would be better to add
scmi_devid to device tree.
In general, I think this configuration should happen automatically
without user intervention. The user should just specify "enable SCMI"
and it should work.
> > In regards to linux,scmi_mem, I think it would be best to do without it
> > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > way, I don't think we should need linux,scmi_mem.
>
> This requires further investigation. I will try to make implementation
> without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> reuslts with you.
OK, thanks.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-22 10:24 ` Oleksii Moisieiev
@ 2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 19:13 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-23 2:23 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> On Mon, Dec 20, 2021 at 04:54:25PM -0800, Stefano Stabellini wrote:
> > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > This enumeration sets SCI type for the domain. Currently there is
> > > two possible options: either 'none' or 'scmi_smc'.
> > >
> > > 'none' is the default value and it disables SCI support at all.
> > >
> > > 'scmi_smc' enables access to the Firmware from the domains using SCMI
> > > protocol and SMC as transport.
> > >
> > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > ---
> > > docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> > > tools/include/libxl.h | 5 +++++
> > > tools/libs/light/libxl_types.idl | 6 ++++++
> > > tools/xl/xl_parse.c | 9 +++++++++
> > > 4 files changed, 42 insertions(+)
> > >
> > > diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> > > index b98d161398..92d0593875 100644
> > > --- a/docs/man/xl.cfg.5.pod.in
> > > +++ b/docs/man/xl.cfg.5.pod.in
> > > @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
> > >
> > > =back
> > >
> > > +=item B<sci="STRING">
> > > +
> > > +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> > > +allows domain to manage various functions that are provided by HW platform.
> > > +
> > > +=over 4
> > > +
> > > +=item B<none>
> > > +
> > > +Don't allow guest to use SCI if present on the platform. This is the default
> > > +value.
> > > +
> > > +=item B<scmi_smc>
> > > +
> > > +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> > > +Inferface - allows domain to manage various functions that are provided by HW
> > > +platform, such as clocks, resets and power-domains. Xen will mediate access to
> > > +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> > > +SMC is used as transport.
> >
> > Would it make sense to actually enable SCMI-SMC support for the guest by
> > default if the guest has any devices directly assigned?
>
> Hi Stefano,
>
> Previously we discussed with Oleksandr about introducing new dom.cfg
> parameter, such as scidev[] lists all sci related nodes, which will help to
> avoid extra domctl calls.
The concerning part is (too much) information the user needs to provide
in device tree or dom.cfg. We can be flexible with the number of extra
domctl calls, but of course it would be good to avoid them too.
> But there is still a question how mediator types should be set for
> different domains? I know currently system supports only one mediator
> type, but different implementation should be also possible.
I think it is fine to have an option sci="scmi_smc" in dom.cfg, that is
not a problem. The issue is a detailed list of SCMI IDs or a second list
of device tree paths, because those are far harder to write correctly.
> For example, if we have to start dom0 and domd using scmi_mailbox
> mediator and domU using scmi_smc, because our system supports only 2
> mailboxes.
Yeah that's fine, it could be done with the sci option you are
suggesting.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-23 2:23 ` Stefano Stabellini
@ 2021-12-23 18:45 ` Volodymyr Babchuk
2021-12-23 19:06 ` Oleksii Moisieiev
2022-01-19 12:04 ` Oleksii Moisieiev
2 siblings, 0 replies; 95+ messages in thread
From: Volodymyr Babchuk @ 2021-12-23 18:45 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Oleksii Moisieiev, xen-devel, Julien Grall, Bertrand Marquis
Hi Stefano, Oleksii,
Stefano Stabellini <sstabellini@kernel.org> writes:
> On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
>> On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
>> > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
>> > > Hi Stefano,
>> > >
>> > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
>> > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
>> > > > > Hi Stefano,
>> > > > >
>> > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
>> > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
[...]
>> > > > In general we can't use properties that are not part of the device tree
>> > > > spec, either
>> > > > https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$
>> > > > [devicetree[.]org] or
>> > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$
>> > > > [git[.]kernel[.]org]
>> > > >
>> > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
>> > > > activities to get "linux,scmi_mem" upstream under
>> > > > Documentation/devicetree/bindings in Linux?
>> > > >
>> > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
>> > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
>> > > > Documentation/devicetree/bindings (probably
>> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
>> > > > work on the Xen code that makes use of it.
>> > > >
>> > > > Does it make sense?
>> > > >
>> > >
>> > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
>> > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
>> >
>> > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
>> > Xen specific. In general, it would be best not to introduce Xen specific
>> > properties into generic bindings. It is a problem both from a
>> > specification perspective (because it has hard to handle Xen specific
>> > cases in fully generic bindings, especially as those bindings are
>> > maintained as part of the Linux kernel) and from a user perspective
>> > (because now the user has to deal with a Xen-specific dtb, or has to
>> > modify the host dtb to add Xen-specific information by hand.)
>> >
>> >
>> > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
>> > looks like a generic property that should be needed for the Linux SCMI
>> > driver too. Why the Linux driver doesn't need it?
>> >
>>
>> scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
>> On non-virtualized systems - there is no need of this call, because OS
>> is the only one entity, running on the system.
>
> OK. Even if it is only required for virtualized systems, I think that
> scmi_devid is important enough that should be part of the upstream
> binding. I think it is worth starting an email thread on the LKML with
> Rob Herring and the SCMI maintainers to discuss the addition of
> scmi_devid to the binding.
Agree there. Also I want to point that there are other hypervisors, like
KVM, which may benefit from this.
Another approach is to name this node "xen,scmdi_devid", but I don't
like it.
>> I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
>> list of device_ids to dom.cfg, such as:
>> sci_devs = [ 0, 1, 15, 35 ];
>>
Well, yes. We discussed this possibility, but personally I stick to
idea of re-using dt_dev, as we discussed in the different thread.
>> Using this approach, we can remove scmi_devid from the device tree and
>> just pass a list of scmi_devids to XEN using additional hypercall.
>> We can probably make hypercall taking devid list as input parameter.
>> This will take only 1 hypercall to setup sci permissions.
>
> But how would a user know which are the right SCMI IDs to add to the
> sci_devs list? Would the user have to go and read the reference manual
> of the platform to find the SCMI IDs and then write sci_devs by hand?
> If that is the case, then I think that it would be better to add
> scmi_devid to device tree.
>
Yes, ideally this needs to be done by BSP vendor in the device tree.
--
Volodymyr Babchuk at EPAM
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-23 2:23 ` Stefano Stabellini
@ 2021-12-23 19:06 ` Stefano Stabellini
2021-12-24 13:30 ` Julien Grall
2021-12-23 19:11 ` Oleksii Moisieiev
1 sibling, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-23 19:06 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Volodymyr Babchuk, Anthony PERARD,
Oleksii Moisieiev, Oleksandr, xen-devel, Wei Liu, Juergen Gross,
Andrew Cooper, George Dunlap, Jan Beulich, Bertrand Marquis
On Wed, 22 Dec 2021, Stefano Stabellini wrote:
> # Future Ideas
>
> A great suggestion by Julien is to start supporting the dom0less partial
> device tree format in xl/libxl as well so that we can have a single
> "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
> dtdev).
>
> Even with that implemented, the user has still to provide a partial dtb,
> xen,reg and xen,path. I think this is a great step forward and we should
> do that, if nothing else to make it easier to switch between dom0less
> and normal domU configurations. But the number of options and
> information that the user has to provide is still similar to what we
> have today.
I have just realized that if we start to parse the partial DTB in
xl/libxl the same way that we do for dom0less guests (parse "xen,path",
"xen,reg", and "interrupts", making dtdev, irqs and iomem optional)
actually we can achieve the goal below thanks to the combination:
"xen,path" + "xen,force-assign-without-iommu".
In other words, with dom0less we already have a way to specify the link
to the host node even if the device is not a DMA master. We can do that
by specifying both xen,path and xen,force-assign-without-iommu for a
device.
This is just FYI. I am not suggesting we should introduce dom0less-style
partial DTBs in order to get SCMI support in guests (although it would
be great to have). I think the best way forward for this series is one
of the combinations below, like a) + d), or a) + c)
[...]
> So I think we should either:
>
> a) extend dtdev to cover all devices, including non-DMA masters
> b) or add a new "device_assignment" property which is like dtdev but is
> the list of both DMA masters and non-DMA masters
>
> Either way, when non-DMA masters are present in the
> dtdev/device_assignment list we could either:
> c) require force-assign-without-iommu="true" in dom.cfg
> d) or print a warning like:
> "WARNING: device assignment safety for device XXX cannot be
> verified. Please make sure XXX is not a DMA mastering device."
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 18:45 ` Volodymyr Babchuk
@ 2021-12-23 19:06 ` Oleksii Moisieiev
2021-12-24 0:16 ` Stefano Stabellini
2022-01-19 12:04 ` Oleksii Moisieiev
2 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-23 19:06 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > Hi Stefano,
> > > >
> > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > Hi Stefano,
> > > > > >
> > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > power-domains without access to CPG.
> > > > > > > >
> > > > > > > > The following features are implemented:
> > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > resets and power-domains via SCMI;
> > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > >
> > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > ---
> > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > >
> > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > SCI mediators implemented in XEN.
> > > > > > > >
> > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > +
> > > > > > > > endmenu
> > > > > > > >
> > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..9563067ddc
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > +config SCMI_SMC
> > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > + default n
> > > > > > > > + depends on SCI
> > > > > > > > + ---help---
> > > > > > > > +
> > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > + for communication.
> > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > @@ -1 +1,2 @@
> > > > > > > > obj-y += sci.o
> > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > +/*
> > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > + *
> > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > + *
> > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > + *
> > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > + * (at your option) any later version.
> > > > > > > > + *
> > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > + * GNU General Public License for more details.
> > > > > > > > + */
> > > > > > > > +
> > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > +#include <asm/smccc.h>
> > > > > > > > +#include <asm/io.h>
> > > > > > > > +#include <xen/bitops.h>
> > > > > > > > +#include <xen/config.h>
> > > > > > > > +#include <xen/sched.h>
> > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > +#include <xen/iocap.h>
> > > > > > > > +#include <xen/init.h>
> > > > > > > > +#include <xen/err.h>
> > > > > > > > +#include <xen/lib.h>
> > > > > > > > +#include <xen/list.h>
> > > > > > > > +#include <xen/mm.h>
> > > > > > > > +#include <xen/string.h>
> > > > > > > > +#include <xen/time.h>
> > > > > > > > +#include <xen/vmap.h>
> > > > > > > > +
> > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > +
> > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > +
> > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > +
> > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > >
> > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > >
> > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > >
> > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > seems to be "arm,scmi-shmem".
> > > > > > >
> > > > > >
> > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > channels:
> > > > > >
> > > > > > reserved-memory {
> > > > > > /* reserved region for scmi channels*/
> > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > no-map;
> > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > };
> > > > > > };
> > > > > >
> > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > the current scmi channel:
> > > > > >
> > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > };
> > > > > >
> > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > assigned to this agent.
> > > > >
> > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > compatible string, not as a node name, and it would need to be described
> > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > >
> > > > > But from your description I don't think it is necessary. We can just use
> > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > >
> > > > > reserved-memory {
> > > > > scp-shmem@0x53FF0000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF1000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF2000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > };
> > > > > ...
> > > > >
> > > > > In other words, if all the individual channel pages are described as
> > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > "linux,scmi_mem"?
> > > > >
> > > >
> > > > That was my first implementation. But I've met a problem with
> > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > there were some if, checking if memory weren't reserved.
> > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > shmem page.
> > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > which has no compatible string and provides no-map property.
> > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > space for the domain.
> > > >
> > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > and scmi_devid property to the
> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > Those node and property are needed only for Xen and useless for
> > > > non-virtualized systems. I can add this node and property description to
> > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > What do you think about it?
> > >
> > > Reply below
> > >
> > > [...]
> > >
> > >
> > > > > In general we can't use properties that are not part of the device tree
> > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > >
> > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > activities to get "linux,scmi_mem" upstream under
> > > > > Documentation/devicetree/bindings in Linux?
> > > > >
> > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > Documentation/devicetree/bindings (probably
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > work on the Xen code that makes use of it.
> > > > >
> > > > > Does it make sense?
> > > > >
> > > >
> > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > >
> > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > Xen specific. In general, it would be best not to introduce Xen specific
> > > properties into generic bindings. It is a problem both from a
> > > specification perspective (because it has hard to handle Xen specific
> > > cases in fully generic bindings, especially as those bindings are
> > > maintained as part of the Linux kernel) and from a user perspective
> > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > modify the host dtb to add Xen-specific information by hand.)
> > >
> > >
> > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > looks like a generic property that should be needed for the Linux SCMI
> > > driver too. Why the Linux driver doesn't need it?
> > >
> >
> > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > On non-virtualized systems - there is no need of this call, because OS
> > is the only one entity, running on the system.
>
> OK. Even if it is only required for virtualized systems, I think that
> scmi_devid is important enough that should be part of the upstream
> binding. I think it is worth starting an email thread on the LKML with
> Rob Herring and the SCMI maintainers to discuss the addition of
> scmi_devid to the binding.
>
Ok I will start the thread about scmi_devid.
>
> > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > list of device_ids to dom.cfg, such as:
> > sci_devs = [ 0, 1, 15, 35 ];
> >
> > Using this approach, we can remove scmi_devid from the device tree and
> > just pass a list of scmi_devids to XEN using additional hypercall.
> > We can probably make hypercall taking devid list as input parameter.
> > This will take only 1 hypercall to setup sci permissions.
>
> But how would a user know which are the right SCMI IDs to add to the
> sci_devs list? Would the user have to go and read the reference manual
> of the platform to find the SCMI IDs and then write sci_devs by hand?
> If that is the case, then I think that it would be better to add
> scmi_devid to device tree.
>
> In general, I think this configuration should happen automatically
> without user intervention. The user should just specify "enable SCMI"
> and it should work.
>
Ok. This sounds reasonable.
>
> > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > way, I don't think we should need linux,scmi_mem.
> >
> > This requires further investigation. I will try to make implementation
> > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > reuslts with you.
>
> OK, thanks.
One more question: As you probably seen - Jan had a complains about SCI
term. He said SCI is ambiguous with ACPI's System
Control Interrupt. I think of using SC (as System Control) instead. What
do you think about it?
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 19:06 ` Stefano Stabellini
@ 2021-12-23 19:11 ` Oleksii Moisieiev
1 sibling, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-23 19:11 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Volodymyr Babchuk, Anthony PERARD, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi Stefano,
On Wed, Dec 22, 2021 at 06:23:13PM -0800, Stefano Stabellini wrote:
> On Wed, 22 Dec 2021, Julien Grall wrote:
> > > > > > > To me dtdev and XEN_DOMCTL_assign_device are appropriate because
> > > > > > > they
> > > > > > > signify device assignment of one or more devices. We are not adding
> > > > > > > any
> > > > > > > additional "meaning" to them. It is just that we'll automatically
> > > > > > > detect
> > > > > > > and generate any SCMI configurations based on the list of assigned
> > > > > > > devices. Because if SCMI is enabled and a device is assigned to the
> > > > > > > guest, then I think we want to add the device to the SCMI list of
> > > > > > > devices automatically.
> > > > > >
> > > > > > I am OK with re-using dtdev/XEN_DOMCTL_assign_device however there is
> > > > > > a pitfall with that approach.
> > > > > >
> > > > > > At the moment, they are only used for device protected by the
> > > > > > IOMMU. If the device is not protected by the IOMMU then it will return
> > > > > > an error.
> > > > > IIRC there was a change, that allowed to assign device without a
> > > > > IOMMU. At
> > > > > least we discussed this internally.
> > > >
> > > > I am not aware of any upstream. Do you have a pointer if there is any
> > > > public discussion?
> > >
> > > No, this is the first public discussion on this topic.
> > >
> > > >
> > > > > >
> > > > > > Now, with your approach we may have a device that is not protected by
> > > > > > the IOMMU but require to a SCMI configuration.
> > > > > You need to protect only DMA-capable devices.
> > > >
> > > > Xen doesn't know if the device is DMA-capable or not. So...
> > > >
> > >
> > > But it knows if there is a iommus=<> node present for the device.
> >
> > Not really. Not all the platforms have IOMMUs and not all the platforms with
> > IOMMU have DMA-capable device protected by an IOMMU.
> >
> > >
> > > > >
> > > > > > I don't think it would be sensible to just return "succeed" here
> > > > > > because technically you are asking to assign a non-protected
> > > > > > device. But at the same time, it would prevent a user to assign a
> > > > > > non-DMA capable device.
> > > > > >
> > > > > > So how do you suggest to approach this?
> > > > > Well, in my head assign_device ideally should do the following:
> > > > > 1. Assign IOMMU if it is configured for the device
> > > >
> > > > ... with this approach you are at the risk to let the user passthrough
> > > > a device that cannot be protected.
> > > >
> > > > > 2. Assign SCMI access rights
> > > > > (Not related to this patch series, but...)
> > > > > 3. Assign IRQs
> > > > > 4. Assign IO memory ranges.
> > > > > Points 3. and 4. would allow us to not provide additional irq=[] and
> > > > > iomem=[] entries in a guest config.
> > > >
> > > > That could only work if your guest is using the same layout as the
> > > > host.
> > >
> > > Agreed. But in my experience, in most cases this is the true. We worked
> > > with Renesas and IMX hardware and in both cases iomems were
> > > mapped 1:1.
> > >
> > > > Otherwise, there is a risk it will clash with other parts of the
> > > > memory layout.
> > > >
> > >
> > > > Today, guests started via the toolstack is only using a virtual
> > > > layout, so you would first need to add support to use the host memory
> > > > layout.
> > >
> > > I can't say for all the possible configurations in the wild, but I'm
> > > assuming that in most cases it will be fine to use 1:1 mapping. For
> > > those more exotic cases it would be possible to implement some
> > > configuration option like iomem_map=[mfn:gfn].
> > Well, the Xen memory layout is not something that is stable nor AFAIK based on
> > any memory layout. In fact, there is no such generic layout on Arm.
> >
> > It is quite possible that it will work well with 1:1 MMIO on some platform
> > (e.g. Renesas, IMX) but you can't expect to work for every Xen release or all
> > the platforms.
>
> Yeah this is a true problem. Thankfully with Penny's series we'll be
> able to create domUs with the host memory layout (although dom0less
> only but it is a step forward).
>
>
> > > As Stefano pointed, right now user needs to provide 3 configuration
> > > options to pass a device to a guest: dt_dev, irq, iomem. But in fact,
> > > Xen is already knowing about irq and iomem from device tree.
> >
> > I understand and this is not ideal. I would be happy to consider your
> > approach. However, this will have to enabled only when the guest is re-using
> > the host layout.
>
> It looks like we all agree that today configuring device assignment with
> Xen is too complicated and would like for it to be simpler. This thread
> has some excellent ideas on how to address the issue. Skip to the end if
> you are only interested in this patch series.
>
>
> # Future Ideas
>
> A great suggestion by Julien is to start supporting the dom0less partial
> device tree format in xl/libxl as well so that we can have a single
> "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
> dtdev).
>
> Even with that implemented, the user has still to provide a partial dtb,
> xen,reg and xen,path. I think this is a great step forward and we should
> do that, if nothing else to make it easier to switch between dom0less
> and normal domU configurations. But the number of options and
> information that the user has to provide is still similar to what we
> have today.
>
> After that, I think we need something along the lines of what Volodymyr
> suggested. Let's say that the user only provides "dtdev" and
> "device_tree" in dom.cfg. We could:
>
> - read interrupts from the "interrupts" property like we do for dom0less
> - read "xen,reg" for the mapping, if "xen,reg" is missing, read "reg"
> and assume 1:1 (we could try the mapping and print an error on
> failure, or we could only do 1:1 if the domain is entirely 1:1)
> - optionally read "xen,path" to populate dtdev
> - if an IOMMU configuration is present, then also configure the IOMMU
> for the device, if not then "xen,force-assign-without-iommu" must be
> present
> - assign SCMI access rights
>
>
> Now we only have:
> - device_tree in dom.cfg
> - dtdev in dom.cfg (or xen,path in the partial DTB)
> - xen,force-assign-without-iommu in the partial DTB
>
>
> It would be good if we could remove "xen,force-assign-without-iommu"
> because at this stage it is the only Xen-specific property remaining in
> the partial DTB. If we could get rid of it, it would make it easier to
> write/generate the partial DTB because it becomes a strict subset of the
> corresponding host node. It would enable us to automatically generate it
> (we are going to do experiments with it at Xilinx in the next few
> months) and it would be easier to explain to users how to write it.
> The partial DTB usually starts as a copy of the corresponding host node
> plus some edits. The fewer edits are required, the better.
>
> But it is difficult because of the reasons mentioned by Julien. In Xen
> we cannot know if a device is not behind an IOMMU because is not a DMA
> master (so safe to assign) or because the system simply doesn't have
> enough IOMMU coverage (so not safe to assign). Thankfully the hardware
> has been improving in recent years and there are more and more platforms
> with full IOMMU coverage. I think we should make it easier for users on
> these well-behave platforms.
>
> At least, we could move the "xen,force-assign-without-iommu" option from
> the partial DTB to the VM config file dom.cfg. Something like:
>
> force-assign-without-iommu="true"
> or
> platform-iommu-safe="true"
>
> That way, it is global rather than per-device and doesn't have to be
> added by the user by hand when creating the partial DTB.
>
>
> # This patch series
>
> Now in regards to this specific series and the SCMI options today, I
> think it is OK to have a per-domain sci="scmi_smc", but I think we
> should try to avoid another detailed list of device paths or list of
> IDs in addition to dtdev.
>
> dtdev already specifies the device tree nodes to be assigned to the
> guest. Based on that list we can also do SCMI assignment.
>
> As Julien pointed out, the issue is: what if a device needs SCMI
> assignment but is not a DMA master (hence there is no IOMMU
> configuration on the host)?
>
> Attempting to assign a DMA mastering device without IOMMU protection is
> not just unsafe, it will actively cause memory corruptions in most
> cases. It is an erroneous configuration.
>
> Of course we should try to stop people from running erroneous
> configurations, but we should do so without penalizing all users.
>
> With that in mind, also considering that dtdev is the list of devices to
> be assigned, I think dtdev should be the list of all devices, even
> non-DMA masters. It makes a lot of sense also because today is really
> difficult to explain to a user that "yes, dtdev is the devices to be
> assigned but no, if the device is a UART you don't have to add it to
> dtdev because it is not a DMA master". It would be a lot easier if the
> list of assigned devices was comprehensive, including both DMA masters
> and not DMA masters.
>
> So I think we should either:
>
> a) extend dtdev to cover all devices, including non-DMA masters
> b) or add a new "device_assignment" property which is like dtdev but is
> the list of both DMA masters and non-DMA masters
>
> Either way, when non-DMA masters are present in the
> dtdev/device_assignment list we could either:
> c) require force-assign-without-iommu="true" in dom.cfg
> d) or print a warning like:
> "WARNING: device assignment safety for device XXX cannot be
> verified. Please make sure XXX is not a DMA mastering device."
>
>
> All these options are good I think. My preference is a) + d), so extend
> dtdev and print a warning if non-DMA masters are in the list. We don't
> necessarily need a new hypercall but that is also an option.
>
> I think this discussion was a long time coming and I am glad we are
> having it now. We have a lot of room for improvement! I don't want to
> scope-creep this patch series, but I hope that this last bit could be
> done as part of this series if we find agreement in the community.
For me a) + d) looks good. I will implement it in v2 if everybody ok
with this approach.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg
2021-12-23 2:23 ` Stefano Stabellini
@ 2021-12-23 19:13 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-23 19:13 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Wei Liu, Anthony PERARD, Juergen Gross
Hi Stefano,
On Wed, Dec 22, 2021 at 06:23:33PM -0800, Stefano Stabellini wrote:
> On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > On Mon, Dec 20, 2021 at 04:54:25PM -0800, Stefano Stabellini wrote:
> > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > This enumeration sets SCI type for the domain. Currently there is
> > > > two possible options: either 'none' or 'scmi_smc'.
> > > >
> > > > 'none' is the default value and it disables SCI support at all.
> > > >
> > > > 'scmi_smc' enables access to the Firmware from the domains using SCMI
> > > > protocol and SMC as transport.
> > > >
> > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > ---
> > > > docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
> > > > tools/include/libxl.h | 5 +++++
> > > > tools/libs/light/libxl_types.idl | 6 ++++++
> > > > tools/xl/xl_parse.c | 9 +++++++++
> > > > 4 files changed, 42 insertions(+)
> > > >
> > > > diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> > > > index b98d161398..92d0593875 100644
> > > > --- a/docs/man/xl.cfg.5.pod.in
> > > > +++ b/docs/man/xl.cfg.5.pod.in
> > > > @@ -1614,6 +1614,28 @@ This feature is a B<technology preview>.
> > > >
> > > > =back
> > > >
> > > > +=item B<sci="STRING">
> > > > +
> > > > +B<Arm only> Set SCI type for the guest. SCI is System Control Protocol -
> > > > +allows domain to manage various functions that are provided by HW platform.
> > > > +
> > > > +=over 4
> > > > +
> > > > +=item B<none>
> > > > +
> > > > +Don't allow guest to use SCI if present on the platform. This is the default
> > > > +value.
> > > > +
> > > > +=item B<scmi_smc>
> > > > +
> > > > +Enables SCMI-SMC support for the guest. SCMI is System Control Management
> > > > +Inferface - allows domain to manage various functions that are provided by HW
> > > > +platform, such as clocks, resets and power-domains. Xen will mediate access to
> > > > +clocks, power-domains and resets between Domains and ATF. Disabled by default.
> > > > +SMC is used as transport.
> > >
> > > Would it make sense to actually enable SCMI-SMC support for the guest by
> > > default if the guest has any devices directly assigned?
> >
> > Hi Stefano,
> >
> > Previously we discussed with Oleksandr about introducing new dom.cfg
> > parameter, such as scidev[] lists all sci related nodes, which will help to
> > avoid extra domctl calls.
>
> The concerning part is (too much) information the user needs to provide
> in device tree or dom.cfg. We can be flexible with the number of extra
> domctl calls, but of course it would be good to avoid them too.
>
>
> > But there is still a question how mediator types should be set for
> > different domains? I know currently system supports only one mediator
> > type, but different implementation should be also possible.
>
> I think it is fine to have an option sci="scmi_smc" in dom.cfg, that is
> not a problem. The issue is a detailed list of SCMI IDs or a second list
> of device tree paths, because those are far harder to write correctly.
>
>
> > For example, if we have to start dom0 and domd using scmi_mailbox
> > mediator and domU using scmi_smc, because our system supports only 2
> > mailboxes.
>
> Yeah that's fine, it could be done with the sci option you are
> suggesting.
Thank you for suggestions. I will use this approach in v2.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-23 19:06 ` Oleksii Moisieiev
@ 2021-12-24 0:16 ` Stefano Stabellini
2021-12-24 13:29 ` Julien Grall
2021-12-24 14:07 ` Oleksii Moisieiev
0 siblings, 2 replies; 95+ messages in thread
From: Stefano Stabellini @ 2021-12-24 0:16 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Thu, 23 Dec 2021, Oleksii Moisieiev wrote:
> On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > Hi Stefano,
> > > > >
> > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > Hi Stefano,
> > > > > > >
> > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > power-domains without access to CPG.
> > > > > > > > >
> > > > > > > > > The following features are implemented:
> > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > ---
> > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > >
> > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > >
> > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > +
> > > > > > > > > endmenu
> > > > > > > > >
> > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > +config SCMI_SMC
> > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > + default n
> > > > > > > > > + depends on SCI
> > > > > > > > > + ---help---
> > > > > > > > > +
> > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > + for communication.
> > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > obj-y += sci.o
> > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > +/*
> > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > + *
> > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > + *
> > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > + *
> > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > + * (at your option) any later version.
> > > > > > > > > + *
> > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > + */
> > > > > > > > > +
> > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > +#include <asm/io.h>
> > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > +#include <xen/config.h>
> > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > +#include <xen/init.h>
> > > > > > > > > +#include <xen/err.h>
> > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > +#include <xen/list.h>
> > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > +#include <xen/string.h>
> > > > > > > > > +#include <xen/time.h>
> > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > +
> > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > +
> > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > +
> > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > +
> > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > >
> > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > >
> > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > >
> > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > >
> > > > > > >
> > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > channels:
> > > > > > >
> > > > > > > reserved-memory {
> > > > > > > /* reserved region for scmi channels*/
> > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > no-map;
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > };
> > > > > > > };
> > > > > > >
> > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > the current scmi channel:
> > > > > > >
> > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > };
> > > > > > >
> > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > assigned to this agent.
> > > > > >
> > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > >
> > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > >
> > > > > > reserved-memory {
> > > > > > scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF1000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF2000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > };
> > > > > > ...
> > > > > >
> > > > > > In other words, if all the individual channel pages are described as
> > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > "linux,scmi_mem"?
> > > > > >
> > > > >
> > > > > That was my first implementation. But I've met a problem with
> > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > there were some if, checking if memory weren't reserved.
> > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > shmem page.
> > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > which has no compatible string and provides no-map property.
> > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > space for the domain.
> > > > >
> > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > and scmi_devid property to the
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > Those node and property are needed only for Xen and useless for
> > > > > non-virtualized systems. I can add this node and property description to
> > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > What do you think about it?
> > > >
> > > > Reply below
> > > >
> > > > [...]
> > > >
> > > >
> > > > > > In general we can't use properties that are not part of the device tree
> > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > >
> > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > Documentation/devicetree/bindings in Linux?
> > > > > >
> > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > Documentation/devicetree/bindings (probably
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > work on the Xen code that makes use of it.
> > > > > >
> > > > > > Does it make sense?
> > > > > >
> > > > >
> > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > >
> > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > properties into generic bindings. It is a problem both from a
> > > > specification perspective (because it has hard to handle Xen specific
> > > > cases in fully generic bindings, especially as those bindings are
> > > > maintained as part of the Linux kernel) and from a user perspective
> > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > modify the host dtb to add Xen-specific information by hand.)
> > > >
> > > >
> > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > looks like a generic property that should be needed for the Linux SCMI
> > > > driver too. Why the Linux driver doesn't need it?
> > > >
> > >
> > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > On non-virtualized systems - there is no need of this call, because OS
> > > is the only one entity, running on the system.
> >
> > OK. Even if it is only required for virtualized systems, I think that
> > scmi_devid is important enough that should be part of the upstream
> > binding. I think it is worth starting an email thread on the LKML with
> > Rob Herring and the SCMI maintainers to discuss the addition of
> > scmi_devid to the binding.
> >
>
> Ok I will start the thread about scmi_devid.
> >
> > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > list of device_ids to dom.cfg, such as:
> > > sci_devs = [ 0, 1, 15, 35 ];
> > >
> > > Using this approach, we can remove scmi_devid from the device tree and
> > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > We can probably make hypercall taking devid list as input parameter.
> > > This will take only 1 hypercall to setup sci permissions.
> >
> > But how would a user know which are the right SCMI IDs to add to the
> > sci_devs list? Would the user have to go and read the reference manual
> > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > If that is the case, then I think that it would be better to add
> > scmi_devid to device tree.
> >
> > In general, I think this configuration should happen automatically
> > without user intervention. The user should just specify "enable SCMI"
> > and it should work.
> >
>
> Ok. This sounds reasonable.
>
> >
> > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > way, I don't think we should need linux,scmi_mem.
> > >
> > > This requires further investigation. I will try to make implementation
> > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > reuslts with you.
> >
> > OK, thanks.
>
> One more question: As you probably seen - Jan had a complains about SCI
> term. He said SCI is ambiguous with ACPI's System
> Control Interrupt.
I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
"SCI" is the first time I saw it with this patch series.
> I think of using SC (as System Control) instead. What do you think
> about it?
Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
doesn't give me enough context to guess what it is.
Or we could broaden the scope and call it "firmware_interface"?
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 0:16 ` Stefano Stabellini
@ 2021-12-24 13:29 ` Julien Grall
2021-12-24 13:59 ` Oleksii Moisieiev
2021-12-24 14:07 ` Oleksii Moisieiev
1 sibling, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-24 13:29 UTC (permalink / raw)
To: Stefano Stabellini, Oleksii Moisieiev
Cc: xen-devel, Volodymyr Babchuk, Bertrand Marquis
Hi,
On 24/12/2021 01:16, Stefano Stabellini wrote:
>> One more question: As you probably seen - Jan had a complains about SCI
>> term. He said SCI is ambiguous with ACPI's System
>> Control Interrupt.
>
> I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
> "SCI" is the first time I saw it with this patch series.
>
>
>> I think of using SC (as System Control) instead. What do you think
>> about it?
>
> Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
> doesn't give me enough context to guess what it is.
I might be missing some context. Why are naming everything SCI rather
than SMCI?
>
> Or we could broaden the scope and call it "firmware_interface"?
How would this be used? Will it be a list of interface that will be
exposed to the guest?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-23 19:06 ` Stefano Stabellini
@ 2021-12-24 13:30 ` Julien Grall
2022-01-19 9:40 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-24 13:30 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Volodymyr Babchuk, Anthony PERARD, Oleksii Moisieiev, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi,
On 23/12/2021 20:06, Stefano Stabellini wrote:
> On Wed, 22 Dec 2021, Stefano Stabellini wrote:
>> # Future Ideas
>>
>> A great suggestion by Julien is to start supporting the dom0less partial
>> device tree format in xl/libxl as well so that we can have a single
>> "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
>> dtdev).
>>
>> Even with that implemented, the user has still to provide a partial dtb,
>> xen,reg and xen,path. I think this is a great step forward and we should
>> do that, if nothing else to make it easier to switch between dom0less
>> and normal domU configurations. But the number of options and
>> information that the user has to provide is still similar to what we
>> have today.
>
> I have just realized that if we start to parse the partial DTB in
> xl/libxl the same way that we do for dom0less guests (parse "xen,path",
> "xen,reg", and "interrupts", making dtdev, irqs and iomem optional)
> actually we can achieve the goal below thanks to the combination:
> "xen,path" + "xen,force-assign-without-iommu".
>
> In other words, with dom0less we already have a way to specify the link
> to the host node even if the device is not a DMA master. We can do that
> by specifying both xen,path and xen,force-assign-without-iommu for a
> device.
>
> This is just FYI. I am not suggesting we should introduce dom0less-style
> partial DTBs in order to get SCMI support in guests (although it would
> be great to have). I think the best way forward for this series is one
> of the combinations below, like a) + d), or a) + c)
I strongly prefer a) + c) because a warning is easy to miss/ignore. At
least with the extra property the user made an action to think about it
and agree that this is the way do it.
It is also easier to spot if we ask the user to provide the
configuration file.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 13:29 ` Julien Grall
@ 2021-12-24 13:59 ` Oleksii Moisieiev
2021-12-24 14:28 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-24 13:59 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, xen-devel, Volodymyr Babchuk, Bertrand Marquis
Hi Julien,
On Fri, Dec 24, 2021 at 02:29:13PM +0100, Julien Grall wrote:
> Hi,
>
> On 24/12/2021 01:16, Stefano Stabellini wrote:
> > > One more question: As you probably seen - Jan had a complains about SCI
> > > term. He said SCI is ambiguous with ACPI's System
> > > Control Interrupt.
> >
> > I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
> > "SCI" is the first time I saw it with this patch series.
> >
> >
> > > I think of using SC (as System Control) instead. What do you think
> > > about it?
> >
> > Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
> > doesn't give me enough context to guess what it is.
>
> I might be missing some context. Why are naming everything SCI rather than
> SMCI?
Because we're expecting other interfaces and transport to be
implemented, such as for example:
scmi_mailbox, scpi_smc, scpi_mailbox, ti_sci_smc etc.
>
> >
> > Or we could broaden the scope and call it "firmware_interface"?
> How would this be used? Will it be a list of interface that will be exposed
> to the guest?
>
The idea is to set mediator type for each Domain, so for example Xen can
use scmi_mailbox to communicate with SCP, Dom0 and DomD are also using
scmi_mailbox, but DomU using scmi_smc mediator because we have only 3
mailboxes in system. This is not implemented yet, right now, we are
introducing only scmi_smc support. In future, multiple mediator support
can be added to Xen.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 0:16 ` Stefano Stabellini
2021-12-24 13:29 ` Julien Grall
@ 2021-12-24 14:07 ` Oleksii Moisieiev
1 sibling, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-24 14:07 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
On Thu, Dec 23, 2021 at 04:16:45PM -0800, Stefano Stabellini wrote:
> On Thu, 23 Dec 2021, Oleksii Moisieiev wrote:
> > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > Hi Stefano,
> > > > > >
> > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > Hi Stefano,
> > > > > > > >
> > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > >
> > > > > > > > > > The following features are implemented:
> > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > ---
> > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > >
> > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > >
> > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > +
> > > > > > > > > > endmenu
> > > > > > > > > >
> > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > + default n
> > > > > > > > > > + depends on SCI
> > > > > > > > > > + ---help---
> > > > > > > > > > +
> > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > + for communication.
> > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > obj-y += sci.o
> > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > +/*
> > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > + *
> > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > + *
> > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > + *
> > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > + *
> > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > + */
> > > > > > > > > > +
> > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > +
> > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > +
> > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > +
> > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > +
> > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > >
> > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > >
> > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > >
> > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > >
> > > > > > > >
> > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > channels:
> > > > > > > >
> > > > > > > > reserved-memory {
> > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > no-map;
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > };
> > > > > > > > };
> > > > > > > >
> > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > the current scmi channel:
> > > > > > > >
> > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > >
> > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > assigned to this agent.
> > > > > > >
> > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > >
> > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > >
> > > > > > > reserved-memory {
> > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > };
> > > > > > > ...
> > > > > > >
> > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > "linux,scmi_mem"?
> > > > > > >
> > > > > >
> > > > > > That was my first implementation. But I've met a problem with
> > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > there were some if, checking if memory weren't reserved.
> > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > shmem page.
> > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > which has no compatible string and provides no-map property.
> > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > space for the domain.
> > > > > >
> > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > and scmi_devid property to the
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > Those node and property are needed only for Xen and useless for
> > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > What do you think about it?
> > > > >
> > > > > Reply below
> > > > >
> > > > > [...]
> > > > >
> > > > >
> > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > >
> > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > >
> > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > work on the Xen code that makes use of it.
> > > > > > >
> > > > > > > Does it make sense?
> > > > > > >
> > > > > >
> > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > >
> > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > properties into generic bindings. It is a problem both from a
> > > > > specification perspective (because it has hard to handle Xen specific
> > > > > cases in fully generic bindings, especially as those bindings are
> > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > >
> > > > >
> > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > driver too. Why the Linux driver doesn't need it?
> > > > >
> > > >
> > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > On non-virtualized systems - there is no need of this call, because OS
> > > > is the only one entity, running on the system.
> > >
> > > OK. Even if it is only required for virtualized systems, I think that
> > > scmi_devid is important enough that should be part of the upstream
> > > binding. I think it is worth starting an email thread on the LKML with
> > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > scmi_devid to the binding.
> > >
> >
> > Ok I will start the thread about scmi_devid.
> > >
> > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > list of device_ids to dom.cfg, such as:
> > > > sci_devs = [ 0, 1, 15, 35 ];
> > > >
> > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > We can probably make hypercall taking devid list as input parameter.
> > > > This will take only 1 hypercall to setup sci permissions.
> > >
> > > But how would a user know which are the right SCMI IDs to add to the
> > > sci_devs list? Would the user have to go and read the reference manual
> > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > If that is the case, then I think that it would be better to add
> > > scmi_devid to device tree.
> > >
> > > In general, I think this configuration should happen automatically
> > > without user intervention. The user should just specify "enable SCMI"
> > > and it should work.
> > >
> >
> > Ok. This sounds reasonable.
> >
> > >
> > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > way, I don't think we should need linux,scmi_mem.
> > > >
> > > > This requires further investigation. I will try to make implementation
> > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > reuslts with you.
> > >
> > > OK, thanks.
> >
> > One more question: As you probably seen - Jan had a complains about SCI
> > term. He said SCI is ambiguous with ACPI's System
> > Control Interrupt.
>
> I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
> "SCI" is the first time I saw it with this patch series.
>
>
> > I think of using SC (as System Control) instead. What do you think
> > about it?
>
> Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
> doesn't give me enough context to guess what it is.
>
> Or we could broaden the scope and call it "firmware_interface"?
ARM_SCI sounds good for me.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 13:59 ` Oleksii Moisieiev
@ 2021-12-24 14:28 ` Julien Grall
2021-12-24 16:49 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-24 14:28 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On 24/12/2021 14:59, Oleksii Moisieiev wrote:
> Hi Julien,
Hello,
> On Fri, Dec 24, 2021 at 02:29:13PM +0100, Julien Grall wrote:
>> Hi,
>>
>> On 24/12/2021 01:16, Stefano Stabellini wrote:
>>>> One more question: As you probably seen - Jan had a complains about SCI
>>>> term. He said SCI is ambiguous with ACPI's System
>>>> Control Interrupt.
>>>
>>> I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
>>> "SCI" is the first time I saw it with this patch series.
>>>
>>>
>>>> I think of using SC (as System Control) instead. What do you think
>>>> about it?
>>>
>>> Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
>>> doesn't give me enough context to guess what it is.
>>
>> I might be missing some context. Why are naming everything SCI rather than
>> SMCI?
>
> Because we're expecting other interfaces and transport to be
> implemented, such as for example:
> scmi_mailbox, scpi_smc, scpi_mailbox, ti_sci_smc etc.
Oh, now that explain why there is a layer of indirection in Xen. It
wasn't very clear from the cover letter why it was present.
>
>>
>>>
>>> Or we could broaden the scope and call it "firmware_interface"?
>> How would this be used? Will it be a list of interface that will be exposed
>> to the guest?
>>
>
> The idea is to set mediator type for each Domain, so for example Xen can
> use scmi_mailbox to communicate with SCP, Dom0 and DomD are also using
> scmi_mailbox, but DomU using scmi_smc mediator because we have only 3
> mailboxes in system. This is not implemented yet, right now, we are
> introducing only scmi_smc support. In future, multiple mediator support
> can be added to Xen.
Ok. So will there be only one interface at the time for a given domain.
Is that correct?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-20 15:41 ` Oleksii Moisieiev
@ 2021-12-24 14:42 ` Julien Grall
2021-12-24 17:02 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2021-12-24 14:42 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> Hi Julien,
Hello,
> On Fri, Dec 17, 2021 at 04:38:31PM +0000, Julien Grall wrote:
>>
>>
>> On 17/12/2021 13:58, Oleksii Moisieiev wrote:
>>> Hi Julien,
>>
>> Hi,
>>
>>> On Fri, Dec 17, 2021 at 01:37:35PM +0000, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 17/12/2021 13:23, Oleksii Moisieiev wrote:
>>>>>>> +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
>>>>>>> +{
>>>>>>> + return iomem_permit_access(d, paddr_to_pfn(addr),
>>>>>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
>>>>>>> + uint64_t len)
>>>>>>> +{
>>>>>>> + return iomem_deny_access(d, paddr_to_pfn(addr),
>>>>>>> + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
>>>>>>> +}
>>>>>>
>>>>>> I wonder, why we need an extra level of indirection here. And if this is
>>>>>> really needed, I wonder why map(unmap)_memory* name was chosen, as there is
>>>>>> no memory mapping/unmapping really happens here.
>>>>>>
>>>>>
>>>>> I've added extra indirection to hide math like
>>>>> paddr_to_pfn(PAGE_ALIGN(addr + len -1)
>>>>> so you don't have to math in each call. unmap_memory_from_domain called
>>>>> from 2 places, so I moved both calls to separate function.
>>>>> Although, I agree that map/unmap is not perfect name. I consider
>>>>> renaming it to mem_permit_acces and mam_deny_access.
>>>>
>>>> I haven't looked at the rest of the series. But this discussion caught my
>>>> eye. This code implies that the address is page-aligned but the length not.
>>>> Is that intended?
>>>>
>>>> That said, if you give permission to the domain on a full page then it means
>>>> it may be able to access address it should not. Can you explain why this is
>>>> fine?
>>>>
>>>
>>> The idea was that xen receives some memory from the dt_node linux,scmi_mem,
>>> then we split memory between the agents, so each agent get 1 page (we
>>> allocate 0x10 pages right now).
>>
>> Thanks for the clarification. Does this imply the guest will be able to
>> write message directly to the firmware?
>
> We used DEN0056C Specification as base. Available on: https://developer.arm.com/documentation/den0056/latest.
> SCMI transport is described in Section 5.1. We implemented Shared Memory transport.
> Firmware has N pages of the shared memory, used to communicate with Agents.
> It allocates N agents and assign a page for each agent, such as:
> -------------------------------------
> | Agent H | Agent 1 | Agent 2 | ... |
> -------------------------------------
> Agent H is the privilleged Hypervisor agent, which is used to do the base commands,
> such as getting Agent list, set\unset permissions etc.
> Hypervisor assign agent to the guest and maps page, related to the agent to the Guest.
> So the Guest, which is Agent 1 will get an access to Agent 1 page.
>
> Guest places SCMI message to Agent 1 memory, then sends SMC message.
> Hypervisor process SMC request, add agent id to the message parameters and redirects it to the Firmware.
> Based on the agent_id Firmware knows which page it should read.
> Then after permission check ( if the resetId/clockID/powerID etc from message
> is assigned to agent_id ) it does changes to the HW and places response to Agent
> shared memory and marks channel as FREE ( by setting free bit in shared memory ).
> Once channel is marked as free - Guest read response from the shared memory.
So, IIUC, the hypervisor will not control what is written in the shared
memory. It will only control the SMC parameters. Is my understanding
correct?
>
> Non-virtualized systems will work as well. OS should send SMC directly to the Firmware.
>
>>
>> If so, this brings a few more questions:
>> 1) What will the guest write in it? Can it contains addresses?
> Guest can write scmi request to the shared memory, which include the following data:
> 1) protocol_id - which protocol is requested, such as clock, power, reset etc
> 2) message_id - action that should be done to HW, such as do_reset or get_clock
> 3) message data - which includes reset_id/clock_id/power_id etc. that should be changed.
> Reset IDs and Clock IDs are assigned in Firmware. Guest receives ID, corresponding to the device from the device-tree.
> dt_node as an example:
> &avb {
> scmi_devid = <0>;
> clocks = <&scmi_clock 0>;
> power-domains = <&scmi_power 0>;
> resets = <&scmi_reset 0>;
> };
>
>> 2) What are the expected memory attribute for the regions?
>
> xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
I think you misunderstood my comment. Memory can be mapped with various
type (e.g. Device, Memory) and attribute (cacheable, non-cacheable...).
What will the firmware expect? What will the guest OS usually?
The reason I am asking is the attributes have to matched to avoid any
coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping,
Xen will configure the stage-2 to use Device nGnRnE. As the result, the
result memory access will be Device nGnRnE which is very strict.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 14:28 ` Julien Grall
@ 2021-12-24 16:49 ` Oleksii Moisieiev
2022-01-03 14:23 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-24 16:49 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On Fri, Dec 24, 2021 at 03:28:56PM +0100, Julien Grall wrote:
>
>
> On 24/12/2021 14:59, Oleksii Moisieiev wrote:
> > Hi Julien,
>
> Hello,
>
> > On Fri, Dec 24, 2021 at 02:29:13PM +0100, Julien Grall wrote:
> > > Hi,
> > >
> > > On 24/12/2021 01:16, Stefano Stabellini wrote:
> > > > > One more question: As you probably seen - Jan had a complains about SCI
> > > > > term. He said SCI is ambiguous with ACPI's System
> > > > > Control Interrupt.
> > > >
> > > > I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
> > > > "SCI" is the first time I saw it with this patch series.
> > > >
> > > >
> > > > > I think of using SC (as System Control) instead. What do you think
> > > > > about it?
> > > >
> > > > Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
> > > > doesn't give me enough context to guess what it is.
> > >
> > > I might be missing some context. Why are naming everything SCI rather than
> > > SMCI?
> >
> > Because we're expecting other interfaces and transport to be
> > implemented, such as for example:
> > scmi_mailbox, scpi_smc, scpi_mailbox, ti_sci_smc etc.
>
> Oh, now that explain why there is a layer of indirection in Xen. It wasn't
> very clear from the cover letter why it was present.
>
Please see below.
> >
> > >
> > > >
> > > > Or we could broaden the scope and call it "firmware_interface"?
> > > How would this be used? Will it be a list of interface that will be exposed
> > > to the guest?
> > >
> >
> > The idea is to set mediator type for each Domain, so for example Xen can
> > use scmi_mailbox to communicate with SCP, Dom0 and DomD are also using
> > scmi_mailbox, but DomU using scmi_smc mediator because we have only 3
> > mailboxes in system. This is not implemented yet, right now, we are
> > introducing only scmi_smc support. In future, multiple mediator support
> > can be added to Xen.
>
> Ok. So will there be only one interface at the time for a given domain. Is
> that correct?
>
Correct. The idea is that we provice only one interface to the Domain,
so different domains can use different protocols and transport. Those
interfaces can be different than the interface Xen uses to connect to SCP.
This allows us to vary the configuration. So for example:
Let's take system, that support only 2 mailboxes and communication with
SCP can use only mailboxes as transport. We intent to use scmi protocol
to manage HW. In this case we use 2 mailboxes for Xen-> SCP
communication, and for Dom0 -> Xen. Domu can be configured to use
scmi_smc, so the communication should be the following:
DomU --smc--> Xen -mailbox--> SCP Firmware.
Let's say we want to add DomainX with OS XXX, which using yyy protocol
with zzz transport. Then we can configure DomX wuth yyy_zzz mediator, so
the communication will be the following:
DomX --yyy--> Xen -mailbox--> SCP Firmware
Where Xen knows how to convert message from yyy protocol to scmi protocol.
I considered the alternative way, when we can configure domain with
several mediators, so each Domain can be configured to use, for example,
scmi_smc for power-domains and scpi_smc for clocks and resets. But I
don't see real use-cases for this configuration.
What do you think about that?
Best regards,
Oleksii
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 14:42 ` Julien Grall
@ 2021-12-24 17:02 ` Oleksii Moisieiev
2022-01-03 13:14 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2021-12-24 17:02 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > Hi Julien,
>
> Hello,
>
> > On Fri, Dec 17, 2021 at 04:38:31PM +0000, Julien Grall wrote:
> > >
> > >
> > > On 17/12/2021 13:58, Oleksii Moisieiev wrote:
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > > On Fri, Dec 17, 2021 at 01:37:35PM +0000, Julien Grall wrote:
> > > > > Hi,
> > > > >
> > > > > On 17/12/2021 13:23, Oleksii Moisieiev wrote:
> > > > > > > > +static int map_memory_to_domain(struct domain *d, uint64_t addr, uint64_t len)
> > > > > > > > +{
> > > > > > > > + return iomem_permit_access(d, paddr_to_pfn(addr),
> > > > > > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static int unmap_memory_from_domain(struct domain *d, uint64_t addr,
> > > > > > > > + uint64_t len)
> > > > > > > > +{
> > > > > > > > + return iomem_deny_access(d, paddr_to_pfn(addr),
> > > > > > > > + paddr_to_pfn(PAGE_ALIGN(addr + len -1)));
> > > > > > > > +}
> > > > > > >
> > > > > > > I wonder, why we need an extra level of indirection here. And if this is
> > > > > > > really needed, I wonder why map(unmap)_memory* name was chosen, as there is
> > > > > > > no memory mapping/unmapping really happens here.
> > > > > > >
> > > > > >
> > > > > > I've added extra indirection to hide math like
> > > > > > paddr_to_pfn(PAGE_ALIGN(addr + len -1)
> > > > > > so you don't have to math in each call. unmap_memory_from_domain called
> > > > > > from 2 places, so I moved both calls to separate function.
> > > > > > Although, I agree that map/unmap is not perfect name. I consider
> > > > > > renaming it to mem_permit_acces and mam_deny_access.
> > > > >
> > > > > I haven't looked at the rest of the series. But this discussion caught my
> > > > > eye. This code implies that the address is page-aligned but the length not.
> > > > > Is that intended?
> > > > >
> > > > > That said, if you give permission to the domain on a full page then it means
> > > > > it may be able to access address it should not. Can you explain why this is
> > > > > fine?
> > > > >
> > > >
> > > > The idea was that xen receives some memory from the dt_node linux,scmi_mem,
> > > > then we split memory between the agents, so each agent get 1 page (we
> > > > allocate 0x10 pages right now).
> > >
> > > Thanks for the clarification. Does this imply the guest will be able to
> > > write message directly to the firmware?
> >
> > We used DEN0056C Specification as base. Available on: https://urldefense.com/v3/__https://developer.arm.com/documentation/den0056/latest__;!!GF_29dbcQIUBPA!m9pWoxBEjb8Sd1CoV5cpU8MbmLCjohYQxv2ci9tDvMmZ9oCEitqyydZ3rQWXCM5bxvIn$ [developer[.]arm[.]com].
> > SCMI transport is described in Section 5.1. We implemented Shared Memory transport.
> > Firmware has N pages of the shared memory, used to communicate with Agents.
> > It allocates N agents and assign a page for each agent, such as:
> > -------------------------------------
> > | Agent H | Agent 1 | Agent 2 | ... |
> > -------------------------------------
> > Agent H is the privilleged Hypervisor agent, which is used to do the base commands,
> > such as getting Agent list, set\unset permissions etc.
> > Hypervisor assign agent to the guest and maps page, related to the agent to the Guest.
> > So the Guest, which is Agent 1 will get an access to Agent 1 page.
> >
> > Guest places SCMI message to Agent 1 memory, then sends SMC message.
> > Hypervisor process SMC request, add agent id to the message parameters and redirects it to the Firmware.
> > Based on the agent_id Firmware knows which page it should read.
> > Then after permission check ( if the resetId/clockID/powerID etc from message
> > is assigned to agent_id ) it does changes to the HW and places response to Agent
> > shared memory and marks channel as FREE ( by setting free bit in shared memory ).
> > Once channel is marked as free - Guest read response from the shared memory.
>
> So, IIUC, the hypervisor will not control what is written in the shared
> memory. It will only control the SMC parameters. Is my understanding
> correct?
>
For scmi_smc it will not. But potentially it can make some changes, or
convert to the different protocol.
> >
> > Non-virtualized systems will work as well. OS should send SMC directly to the Firmware.
> >
> > >
> > > If so, this brings a few more questions:
> > > 1) What will the guest write in it? Can it contains addresses?
> > Guest can write scmi request to the shared memory, which include the following data:
> > 1) protocol_id - which protocol is requested, such as clock, power, reset etc
> > 2) message_id - action that should be done to HW, such as do_reset or get_clock
> > 3) message data - which includes reset_id/clock_id/power_id etc. that should be changed.
> > Reset IDs and Clock IDs are assigned in Firmware. Guest receives ID, corresponding to the device from the device-tree.
> > dt_node as an example:
> > &avb {
> > scmi_devid = <0>;
> > clocks = <&scmi_clock 0>;
> > power-domains = <&scmi_power 0>;
> > resets = <&scmi_reset 0>;
> > };
> >
> > > 2) What are the expected memory attribute for the regions?
> >
> > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
>
> I think you misunderstood my comment. Memory can be mapped with various type
> (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> the firmware expect? What will the guest OS usually?
>
> The reason I am asking is the attributes have to matched to avoid any
> coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> will configure the stage-2 to use Device nGnRnE. As the result, the result
> memory access will be Device nGnRnE which is very strict.
>
Let me share with you the configuration example:
scmi expects memory to be configured in the device-tree:
cpu_scp_shm: scp-shmem@0xXXXXXXX {
compatible = "arm,scmi-shmem";
reg = <0x0 0xXXXXXX 0x0 0x1000>;
};
where XXXXXX address I allocate in alloc_magic_pages function.
Then I get paddr of the scmi channel for this domain and use
XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
Hope I wass able to answer your question.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 17:02 ` Oleksii Moisieiev
@ 2022-01-03 13:14 ` Julien Grall
2022-01-06 13:53 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2022-01-03 13:14 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi,
On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
>> On 20/12/2021 16:41, Oleksii Moisieiev wrote:
>>>> 2) What are the expected memory attribute for the regions?
>>>
>>> xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
>>
>> I think you misunderstood my comment. Memory can be mapped with various type
>> (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
>> the firmware expect? What will the guest OS usually?
>>
>> The reason I am asking is the attributes have to matched to avoid any
>> coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
>> will configure the stage-2 to use Device nGnRnE. As the result, the result
>> memory access will be Device nGnRnE which is very strict.
>>
>
> Let me share with you the configuration example:
> scmi expects memory to be configured in the device-tree:
>
> cpu_scp_shm: scp-shmem@0xXXXXXXX {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0xXXXXXX 0x0 0x1000>;
> };
>
> where XXXXXX address I allocate in alloc_magic_pages function.
The goal of alloc_magic_pages() is to allocate RAM. However, what you
want is a guest physical address and then map a part of the shared page.
I can see two options here:
1) Hardcode the SCMI region in the memory map
2) Create a new region in the memory map that can be used for
reserving memory for mapping.
We still have plenty of space in the guest memory map. So the former is
probably going to be fine for now.
> Then I get paddr of the scmi channel for this domain and use
> XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
> > Hope I wass able to answer your question.
What you provided me is how the guest OS will locate the shared memory.
This still doesn't tell me which memory attribute will be used to map
the page in Stage-1 (guest page-tables).
To find that out, you want to look at the driver and how the mapping is
done. The Linux driver (drivers/firmware/arm_scmi) is using
devm_ioremap() (see smc_chan_setup()).
Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
(arm64) which is one of the most restrictive memory attribute.
This means the firmware should be able to deal with the most restrictive
attribute and therefore using XEN_DOMCTL_memory_mapping to map the
shared page in stage-2 should be fine.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-24 16:49 ` Oleksii Moisieiev
@ 2022-01-03 14:23 ` Julien Grall
2022-01-06 15:19 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2022-01-03 14:23 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Volodymyr Babchuk, Bertrand Marquis
Hi,
On 24/12/2021 16:49, Oleksii Moisieiev wrote:
> On Fri, Dec 24, 2021 at 03:28:56PM +0100, Julien Grall wrote:
>> On 24/12/2021 14:59, Oleksii Moisieiev wrote:
>>> Hi Julien,
>>
>> Hello,
>>
>>> On Fri, Dec 24, 2021 at 02:29:13PM +0100, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 24/12/2021 01:16, Stefano Stabellini wrote:
>>>>>> One more question: As you probably seen - Jan had a complains about SCI
>>>>>> term. He said SCI is ambiguous with ACPI's System
>>>>>> Control Interrupt.
>>>>>
>>>>> I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
>>>>> "SCI" is the first time I saw it with this patch series.
>>>>>
>>>>>
>>>>>> I think of using SC (as System Control) instead. What do you think
>>>>>> about it?
>>>>>
>>>>> Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
>>>>> doesn't give me enough context to guess what it is.
>>>>
>>>> I might be missing some context. Why are naming everything SCI rather than
>>>> SMCI?
>>>
>>> Because we're expecting other interfaces and transport to be
>>> implemented, such as for example:
>>> scmi_mailbox, scpi_smc, scpi_mailbox, ti_sci_smc etc.
>>
>> Oh, now that explain why there is a layer of indirection in Xen. It wasn't
>> very clear from the cover letter why it was present.
>>
> Please see below.
>>>
>>>>
>>>>>
>>>>> Or we could broaden the scope and call it "firmware_interface"?
>>>> How would this be used? Will it be a list of interface that will be exposed
>>>> to the guest?
>>>>
>>>
>>> The idea is to set mediator type for each Domain, so for example Xen can
>>> use scmi_mailbox to communicate with SCP, Dom0 and DomD are also using
>>> scmi_mailbox, but DomU using scmi_smc mediator because we have only 3
>>> mailboxes in system. This is not implemented yet, right now, we are
>>> introducing only scmi_smc support. In future, multiple mediator support
>>> can be added to Xen.
>>
>> Ok. So will there be only one interface at the time for a given domain. Is
>> that correct?
>>
> Correct. The idea is that we provice only one interface to the Domain,
> so different domains can use different protocols and transport
I think this is similar to TEE mediator. Specifying in the guest
configuration the TEE protocol (e.g. OP-TEE) allows to sanity check what
the guest wants match the host. However, I am not convinced there is a
need to expose different protocols for domains running on the same platform.
> Those
> interfaces can be different than the interface Xen uses to connect to SCP.
I can understand how this looks appealing. However, it may not always be
possible to convert the guest protocol to the host protocol. And even if
it is possible, it is increasing the complexity and risk.
> This allows us to vary the configuration.
> So for example:
> Let's take system, that support only 2 mailboxes and communication with
> SCP can use only mailboxes as transport. We intent to use scmi protocol
> to manage HW. In this case we use 2 mailboxes for Xen-> SCP
> communication, and for Dom0 -> Xen.
> Domu can be configured to use
> scmi_smc, so the communication should be the following:
> DomU --smc--> Xen -mailbox--> SCP Firmware.
> Let's say we want to add DomainX with OS XXX, which using yyy protocol
> with zzz transport. Then we can configure DomX wuth yyy_zzz mediator, so
> the communication will be the following:
> DomX --yyy--> Xen -mailbox--> SCP Firmware
I am a bit confused with your example here. From my understanding, I
would say 'smc' is the transport. But in your example above, you suggest
this is the protocol. What did I miss?
> Where Xen knows how to convert message from yyy protocol to scmi protocol
As I wrote above, when possible, converting protocol could be complex
and risky. So do you have a concrete use case for that?
That said, changing the transport (e.g. smc <-> mailbox) would make more
sense to me.
>
> I considered the alternative way, when we can configure domain with
> several mediators, so each Domain can be configured to use, for example,
> scmi_smc for power-domains and scpi_smc for clocks and resets. But I
> don't see real use-cases for this configuration.
Do you mean a platform using both or exposing both to the guest?
>
> What do you think about that?
So I am a bit unsure how "firmware_interface" would be specified. IMO,
the user should only specificy which interface the guest will use (e.g.
SCMI via SMC) even if the host end up to use a different transport (e.g.
SCMI via mailbox). IOW, the option would not tell how to convert it.
Is it what you had in mind?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-03 13:14 ` Julien Grall
@ 2022-01-06 13:53 ` Oleksii Moisieiev
2022-01-06 14:02 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-06 13:53 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi Julien,
On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> Hi,
>
> On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > 2) What are the expected memory attribute for the regions?
> > > >
> > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > >
> > > I think you misunderstood my comment. Memory can be mapped with various type
> > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > the firmware expect? What will the guest OS usually?
> > >
> > > The reason I am asking is the attributes have to matched to avoid any
> > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > memory access will be Device nGnRnE which is very strict.
> > >
> >
> > Let me share with you the configuration example:
> > scmi expects memory to be configured in the device-tree:
> >
> > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > };
> >
> > where XXXXXX address I allocate in alloc_magic_pages function.
>
> The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> is a guest physical address and then map a part of the shared page.
Do you mean that I can't use alloc_magic_pages to allocate shared
memory region for SCMI?
>
> I can see two options here:
> 1) Hardcode the SCMI region in the memory map
> 2) Create a new region in the memory map that can be used for reserving
> memory for mapping.
Could you please explain what do you mean under the "new region in the
memory map"?
>
> We still have plenty of space in the guest memory map. So the former is
> probably going to be fine for now.
>
> > Then I get paddr of the scmi channel for this domain and use
> > XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
> > > Hope I wass able to answer your question.
>
> What you provided me is how the guest OS will locate the shared memory. This
> still doesn't tell me which memory attribute will be used to map the page in
> Stage-1 (guest page-tables).
>
> To find that out, you want to look at the driver and how the mapping is
> done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
> (see smc_chan_setup()).
>
> Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
> (arm64) which is one of the most restrictive memory attribute.
>
> This means the firmware should be able to deal with the most restrictive
> attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
> page in stage-2 should be fine.
>
I'm using vmap call to map channel memory (see smc_create_channel()).
vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
Considering that protocol is synchronous and only one agent per channel is
expected - this works fine for now.
But I agree that the same memory attributes should be used in xen and
kernel. I fill fix that in v2.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-06 13:53 ` Oleksii Moisieiev
@ 2022-01-06 14:02 ` Julien Grall
2022-01-06 15:43 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2022-01-06 14:02 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> Hi Julien,
Hi,
>
> On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
>> Hi,
>>
>> On 24/12/2021 17:02, Oleksii Moisieiev wrote:
>>> On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
>>>> On 20/12/2021 16:41, Oleksii Moisieiev wrote:
>>>>>> 2) What are the expected memory attribute for the regions?
>>>>>
>>>>> xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
>>>>
>>>> I think you misunderstood my comment. Memory can be mapped with various type
>>>> (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
>>>> the firmware expect? What will the guest OS usually?
>>>>
>>>> The reason I am asking is the attributes have to matched to avoid any
>>>> coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
>>>> will configure the stage-2 to use Device nGnRnE. As the result, the result
>>>> memory access will be Device nGnRnE which is very strict.
>>>>
>>>
>>> Let me share with you the configuration example:
>>> scmi expects memory to be configured in the device-tree:
>>>
>>> cpu_scp_shm: scp-shmem@0xXXXXXXX {
>>> compatible = "arm,scmi-shmem";
>>> reg = <0x0 0xXXXXXX 0x0 0x1000>;
>>> };
>>>
>>> where XXXXXX address I allocate in alloc_magic_pages function.
>>
>> The goal of alloc_magic_pages() is to allocate RAM. However, what you want
>> is a guest physical address and then map a part of the shared page.
>
> Do you mean that I can't use alloc_magic_pages to allocate shared
> memory region for SCMI?
Correct. alloc_magic_pages() will allocate a RAM page and then assign to
the guest. From your description, this is not what you want because you
will call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
>
>>
>> I can see two options here:
>> 1) Hardcode the SCMI region in the memory map
>> 2) Create a new region in the memory map that can be used for reserving
>> memory for mapping.
>
> Could you please explain what do you mean under the "new region in the
> memory map"?
I mean reserving some guest physical address that could be used for map
host physical address (e.g. SCMI region, GIC CPU interface...).
So rather than hardcoding the address, we have something more flexible.
>
>>
>> We still have plenty of space in the guest memory map. So the former is
>> probably going to be fine for now.
>>
>>> Then I get paddr of the scmi channel for this domain and use
>>> XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
>>> > Hope I wass able to answer your question.
>>
>> What you provided me is how the guest OS will locate the shared memory. This
>> still doesn't tell me which memory attribute will be used to map the page in
>> Stage-1 (guest page-tables).
>>
>> To find that out, you want to look at the driver and how the mapping is
>> done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
>> (see smc_chan_setup()).
>>
>> Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
>> (arm64) which is one of the most restrictive memory attribute.
>>
>> This means the firmware should be able to deal with the most restrictive
>> attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
>> page in stage-2 should be fine.
>>
>
> I'm using vmap call to map channel memory (see smc_create_channel()).
> vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
You want to use ioremap().
> Considering that protocol is synchronous and only one agent per channel is
> expected - this works fine for now.
> But I agree that the same memory attributes should be used in xen and
> kernel. I fill fix that in v2.
I am a bit confused. Are you mapping the full shared memory area in Xen?
If yes, why do you need to map the memory that is going to be shared
with a domain?
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-03 14:23 ` Julien Grall
@ 2022-01-06 15:19 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-06 15:19 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On Mon, Jan 03, 2022 at 02:23:01PM +0000, Julien Grall wrote:
> Hi,
>
> On 24/12/2021 16:49, Oleksii Moisieiev wrote:
> > On Fri, Dec 24, 2021 at 03:28:56PM +0100, Julien Grall wrote:
> > > On 24/12/2021 14:59, Oleksii Moisieiev wrote:
> > > > Hi Julien,
> > >
> > > Hello,
> > >
> > > > On Fri, Dec 24, 2021 at 02:29:13PM +0100, Julien Grall wrote:
> > > > > Hi,
> > > > >
> > > > > On 24/12/2021 01:16, Stefano Stabellini wrote:
> > > > > > > One more question: As you probably seen - Jan had a complains about SCI
> > > > > > > term. He said SCI is ambiguous with ACPI's System
> > > > > > > Control Interrupt.
> > > > > >
> > > > > > I see his point. As a term I see "SCMI" often and sometimes "SCPI" but
> > > > > > "SCI" is the first time I saw it with this patch series.
> > > > > >
> > > > > >
> > > > > > > I think of using SC (as System Control) instead. What do you think
> > > > > > > about it?
> > > > > >
> > > > > > Yeah, I am not great at naming things but maybe "ARM_SCI"? "SC" alone
> > > > > > doesn't give me enough context to guess what it is.
> > > > >
> > > > > I might be missing some context. Why are naming everything SCI rather than
> > > > > SMCI?
> > > >
> > > > Because we're expecting other interfaces and transport to be
> > > > implemented, such as for example:
> > > > scmi_mailbox, scpi_smc, scpi_mailbox, ti_sci_smc etc.
> > >
> > > Oh, now that explain why there is a layer of indirection in Xen. It wasn't
> > > very clear from the cover letter why it was present.
> > >
> > Please see below.
> > > >
> > > > >
> > > > > >
> > > > > > Or we could broaden the scope and call it "firmware_interface"?
> > > > > How would this be used? Will it be a list of interface that will be exposed
> > > > > to the guest?
> > > > >
> > > >
> > > > The idea is to set mediator type for each Domain, so for example Xen can
> > > > use scmi_mailbox to communicate with SCP, Dom0 and DomD are also using
> > > > scmi_mailbox, but DomU using scmi_smc mediator because we have only 3
> > > > mailboxes in system. This is not implemented yet, right now, we are
> > > > introducing only scmi_smc support. In future, multiple mediator support
> > > > can be added to Xen.
> > >
> > > Ok. So will there be only one interface at the time for a given domain. Is
> > > that correct?
> > >
> > Correct. The idea is that we provice only one interface to the Domain,
> > so different domains can use different protocols and transport
>
> I think this is similar to TEE mediator. Specifying in the guest
> configuration the TEE protocol (e.g. OP-TEE) allows to sanity check what the
> guest wants match the host. However, I am not convinced there is a need to
> expose different protocols for domains running on the same platform.
>
Yes. You're right, I based on TEE mediator during implementation.
Please see below my thoughts about different protocols and transports.
> > Those
> > interfaces can be different than the interface Xen uses to connect to SCP.
>
> I can understand how this looks appealing. However, it may not always be
> possible to convert the guest protocol to the host protocol. And even if it
> is possible, it is increasing the complexity and risk.
>
> > This allows us to vary the configuration.
> > So for example:
> > Let's take system, that support only 2 mailboxes and communication with
> > SCP can use only mailboxes as transport. We intent to use scmi protocol
> > to manage HW. In this case we use 2 mailboxes for Xen-> SCP
> > communication, and for Dom0 -> Xen.
> > Domu can be configured to use
> > scmi_smc, so the communication should be the following:
> > DomU --smc--> Xen -mailbox--> SCP Firmware.
> > Let's say we want to add DomainX with OS XXX, which using yyy protocol
> > with zzz transport. Then we can configure DomX wuth yyy_zzz mediator, so
> > the communication will be the following:
> > DomX --yyy--> Xen -mailbox--> SCP Firmware
>
> I am a bit confused with your example here. From my understanding, I would
> say 'smc' is the transport. But in your example above, you suggest this is
> the protocol. What did I miss?
Sorry, that's my fault. I merged 2 exapmles in 1, in which the first example
shows different transports and the second one shows different protocols.
The first expample describes the use-case when we have firmware, using
mailboxes as transport, but the system we have has number of Domains >
than number of the mailboxes.
Let's say Firmware and all Domains are using scmi protocol.
In this case I see 2 possible configurations:
1) Firmware and some Domains are using mailbox as transport, other
Domains, when all mailboxes are occupied, are using smc as transport.
Xen will be responsible to receive smc message and send request to
Firmware using mailbox and vice versa.
2) Firmware using mailbox to communicate with Xen, all Domains are using
smc. In this case only one mailbox will be used.
Number 2 looks better from my standpoint. In that case we have scmi_smc
mediator, which exposes smc transport to domains and uses mailbox to
communicate with Firmware.
The second part is that we can change protocol when it's simple and
safe. But now I see that this is not a good Idea to make Xen responsible
for protocol converting.
>
> > Where Xen knows how to convert message from yyy protocol to scmi protocol
>
> As I wrote above, when possible, converting protocol could be complex and
> risky. So do you have a concrete use case for that?
>
> That said, changing the transport (e.g. smc <-> mailbox) would make more
> sense to me.
>
I think you're right.
> >
> > I considered the alternative way, when we can configure domain with
> > several mediators, so each Domain can be configured to use, for example,
> > scmi_smc for power-domains and scpi_smc for clocks and resets. But I
> > don't see real use-cases for this configuration.
>
> Do you mean a platform using both or exposing both to the guest?
>
Yes, that's the alternative way I see. Similar to linux kernel, where
you are allowed to use for example scmi for clocks and scpi for resets.
I don't see the real use-cases for this config, but Xen can follow Linux
kernel configuration approach and allow to register 2 mediators: one for
scmi_smc and one for scpi_smc.
In this case Domain configuration will look like this:
sci = [
"scmi_smc",
"scpi_smc",
];
So both scmi_smc and scpi_smc mediators are exposing protocols to the
domains and communicating with Firmware.
Current implementation supports only 1 mediator, but in future I think
we should consider the posibility of having several mediators.
Where each mediator has it's own channel (transport+protocol) to
Firmware and exposes transport+protocol to the guests.
> >
> > What do you think about that?
> So I am a bit unsure how "firmware_interface" would be specified. IMO, the
> user should only specificy which interface the guest will use (e.g. SCMI via
> SMC) even if the host end up to use a different transport (e.g. SCMI via
> mailbox). IOW, the option would not tell how to convert it.
>
> Is it what you had in mind?
>
Yes. That's what I mean.
firmware_interface can be specified in xen Device-tree. Different SCI
mediators can be implemented and added to the device-tree. Each mediator has channel (protocol +
transports) to the Firmware and exposes protocol + transport to the guests.
Protocol to the Firmware should be the same as exposed protocol,
transport may differ and is implementation specific.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-06 14:02 ` Julien Grall
@ 2022-01-06 15:43 ` Oleksii Moisieiev
2022-01-06 16:04 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-06 15:43 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
>
>
> On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> > Hi Julien,
>
> Hi,
>
> >
> > On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> > > Hi,
> > >
> > > On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > > > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > > > 2) What are the expected memory attribute for the regions?
> > > > > >
> > > > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > > > >
> > > > > I think you misunderstood my comment. Memory can be mapped with various type
> > > > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > > > the firmware expect? What will the guest OS usually?
> > > > >
> > > > > The reason I am asking is the attributes have to matched to avoid any
> > > > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > > > memory access will be Device nGnRnE which is very strict.
> > > > >
> > > >
> > > > Let me share with you the configuration example:
> > > > scmi expects memory to be configured in the device-tree:
> > > >
> > > > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > > > };
> > > >
> > > > where XXXXXX address I allocate in alloc_magic_pages function.
> > >
> > > The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> > > is a guest physical address and then map a part of the shared page.
> >
> > Do you mean that I can't use alloc_magic_pages to allocate shared
> > memory region for SCMI?
> Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
> guest. From your description, this is not what you want because you will
> call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
>
Ok thanks, I will refactor this part in v2.
> >
> > >
> > > I can see two options here:
> > > 1) Hardcode the SCMI region in the memory map
> > > 2) Create a new region in the memory map that can be used for reserving
> > > memory for mapping.
> >
> > Could you please explain what do you mean under the "new region in the
> > memory map"?
>
> I mean reserving some guest physical address that could be used for map host
> physical address (e.g. SCMI region, GIC CPU interface...).
>
> So rather than hardcoding the address, we have something more flexible.
>
Ok, I will fix that in v2.
> >
> > >
> > > We still have plenty of space in the guest memory map. So the former is
> > > probably going to be fine for now.
> > >
> > > > Then I get paddr of the scmi channel for this domain and use
> > > > XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
> > > > > Hope I wass able to answer your question.
> > >
> > > What you provided me is how the guest OS will locate the shared memory. This
> > > still doesn't tell me which memory attribute will be used to map the page in
> > > Stage-1 (guest page-tables).
> > >
> > > To find that out, you want to look at the driver and how the mapping is
> > > done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
> > > (see smc_chan_setup()).
> > >
> > > Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
> > > (arm64) which is one of the most restrictive memory attribute.
> > >
> > > This means the firmware should be able to deal with the most restrictive
> > > attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
> > > page in stage-2 should be fine.
> > >
> >
> > I'm using vmap call to map channel memory (see smc_create_channel()).
> > vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
>
> You want to use ioremap().
>
I've used ioremap originally, but changed it to vmap because ioremap
doesn't support memcpy.
What if I use __vmap with MT_DEVICE_nGnRE flag?
> > Considering that protocol is synchronous and only one agent per channel is
> > expected - this works fine for now.
> > But I agree that the same memory attributes should be used in xen and
> > kernel. I fill fix that in v2.
>
> I am a bit confused. Are you mapping the full shared memory area in Xen? If
> yes, why do you need to map the memory that is going to be shared with a
> domain?
>
Xen should have an access to all agent channels because it should send
SCMI_BASE_DISCOVER_AGENT to each channel and receive agent_id during
scmi_probe call.
Best regards,
Oleksii
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-06 15:43 ` Oleksii Moisieiev
@ 2022-01-06 16:04 ` Julien Grall
2022-01-06 16:28 ` Oleksii Moisieiev
2022-01-19 10:37 ` Oleksii Moisieiev
0 siblings, 2 replies; 95+ messages in thread
From: Julien Grall @ 2022-01-06 16:04 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi,
On 06/01/2022 15:43, Oleksii Moisieiev wrote:
> On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
>>
>>
>> On 06/01/2022 13:53, Oleksii Moisieiev wrote:
>>> Hi Julien,
>>
>> Hi,
>>
>>>
>>> On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 24/12/2021 17:02, Oleksii Moisieiev wrote:
>>>>> On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
>>>>>> On 20/12/2021 16:41, Oleksii Moisieiev wrote:
>>>>>>>> 2) What are the expected memory attribute for the regions?
>>>>>>>
>>>>>>> xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
>>>>>>
>>>>>> I think you misunderstood my comment. Memory can be mapped with various type
>>>>>> (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
>>>>>> the firmware expect? What will the guest OS usually?
>>>>>>
>>>>>> The reason I am asking is the attributes have to matched to avoid any
>>>>>> coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
>>>>>> will configure the stage-2 to use Device nGnRnE. As the result, the result
>>>>>> memory access will be Device nGnRnE which is very strict.
>>>>>>
>>>>>
>>>>> Let me share with you the configuration example:
>>>>> scmi expects memory to be configured in the device-tree:
>>>>>
>>>>> cpu_scp_shm: scp-shmem@0xXXXXXXX {
>>>>> compatible = "arm,scmi-shmem";
>>>>> reg = <0x0 0xXXXXXX 0x0 0x1000>;
>>>>> };
>>>>>
>>>>> where XXXXXX address I allocate in alloc_magic_pages function.
>>>>
>>>> The goal of alloc_magic_pages() is to allocate RAM. However, what you want
>>>> is a guest physical address and then map a part of the shared page.
>>>
>>> Do you mean that I can't use alloc_magic_pages to allocate shared
>>> memory region for SCMI?
>> Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
>> guest. From your description, this is not what you want because you will
>> call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
>>
>
> Ok thanks, I will refactor this part in v2.
>
>>>
>>>>
>>>> I can see two options here:
>>>> 1) Hardcode the SCMI region in the memory map
>>>> 2) Create a new region in the memory map that can be used for reserving
>>>> memory for mapping.
>>>
>>> Could you please explain what do you mean under the "new region in the
>>> memory map"?
>>
>> I mean reserving some guest physical address that could be used for map host
>> physical address (e.g. SCMI region, GIC CPU interface...).
>>
>> So rather than hardcoding the address, we have something more flexible.
>>
>
> Ok, I will fix that in v2.
Just for avoidance of doubt. I was clarify option 2 and not requesting
to implement. That said, if you want to implement option 2 I would be
happy to review it.
>
>>>
>>>>
>>>> We still have plenty of space in the guest memory map. So the former is
>>>> probably going to be fine for now.
>>>>
>>>>> Then I get paddr of the scmi channel for this domain and use
>>>>> XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
>>>>> > Hope I wass able to answer your question.
>>>>
>>>> What you provided me is how the guest OS will locate the shared memory. This
>>>> still doesn't tell me which memory attribute will be used to map the page in
>>>> Stage-1 (guest page-tables).
>>>>
>>>> To find that out, you want to look at the driver and how the mapping is
>>>> done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
>>>> (see smc_chan_setup()).
>>>>
>>>> Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
>>>> (arm64) which is one of the most restrictive memory attribute.
>>>>
>>>> This means the firmware should be able to deal with the most restrictive
>>>> attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
>>>> page in stage-2 should be fine.
>>>>
>>>
>>> I'm using vmap call to map channel memory (see smc_create_channel()).
>>> vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
>>
>> You want to use ioremap().
>>
>
> I've used ioremap originally, but changed it to vmap because ioremap
> doesn't support memcpy.
> What if I use __vmap with MT_DEVICE_nGnRE flag?
That's not going to help. Our implementation of memcpy() is using
unaligned access (which is forbidden on Device memory).
You will need something similar to memcpy_toio() in Linux. I don't think
we have one today in Xen, so I would suggest to import the
implementation from Linux.
>
>>> Considering that protocol is synchronous and only one agent per channel is
>>> expected - this works fine for now.
>>> But I agree that the same memory attributes should be used in xen and
>>> kernel. I fill fix that in v2.
>>
>> I am a bit confused. Are you mapping the full shared memory area in Xen? If
>> yes, why do you need to map the memory that is going to be shared with a
>> domain?
>>
>
> Xen should have an access to all agent channels because it should send
> SCMI_BASE_DISCOVER_AGENT to each channel and receive agent_id during
> scmi_probe call.
Hmmm... Just to confirm, this will only happen during Xen boot? IOW, Xen
will never write to the channel when a domain is running?
If yes, then I think it would be best to unmap the channel once they are
used. This would prevent all sort of issues (e.g. Xen mistakenly written
in them).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-06 16:04 ` Julien Grall
@ 2022-01-06 16:28 ` Oleksii Moisieiev
2022-01-19 10:37 ` Oleksii Moisieiev
1 sibling, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-06 16:28 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi Julien,
On Thu, Jan 06, 2022 at 04:04:34PM +0000, Julien Grall wrote:
> Hi,
>
> On 06/01/2022 15:43, Oleksii Moisieiev wrote:
> > On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
> > >
> > >
> > > On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > >
> > > > On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> > > > > Hi,
> > > > >
> > > > > On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > > > > > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > > > > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > > > > > 2) What are the expected memory attribute for the regions?
> > > > > > > >
> > > > > > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > > > > > >
> > > > > > > I think you misunderstood my comment. Memory can be mapped with various type
> > > > > > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > > > > > the firmware expect? What will the guest OS usually?
> > > > > > >
> > > > > > > The reason I am asking is the attributes have to matched to avoid any
> > > > > > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > > > > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > > > > > memory access will be Device nGnRnE which is very strict.
> > > > > > >
> > > > > >
> > > > > > Let me share with you the configuration example:
> > > > > > scmi expects memory to be configured in the device-tree:
> > > > > >
> > > > > > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > > > > > };
> > > > > >
> > > > > > where XXXXXX address I allocate in alloc_magic_pages function.
> > > > >
> > > > > The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> > > > > is a guest physical address and then map a part of the shared page.
> > > >
> > > > Do you mean that I can't use alloc_magic_pages to allocate shared
> > > > memory region for SCMI?
> > > Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
> > > guest. From your description, this is not what you want because you will
> > > call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
> > >
> >
> > Ok thanks, I will refactor this part in v2.
> >
> > > >
> > > > >
> > > > > I can see two options here:
> > > > > 1) Hardcode the SCMI region in the memory map
> > > > > 2) Create a new region in the memory map that can be used for reserving
> > > > > memory for mapping.
> > > >
> > > > Could you please explain what do you mean under the "new region in the
> > > > memory map"?
> > >
> > > I mean reserving some guest physical address that could be used for map host
> > > physical address (e.g. SCMI region, GIC CPU interface...).
> > >
> > > So rather than hardcoding the address, we have something more flexible.
> > >
> >
> > Ok, I will fix that in v2.
>
> Just for avoidance of doubt. I was clarify option 2 and not requesting to
> implement. That said, if you want to implement option 2 I would be happy to
> review it.
>
I think it's time for me to start working on v2. I think option 2 can be
implemented. If not - I will look for an alternative way.
> >
> > > >
> > > > >
> > > > > We still have plenty of space in the guest memory map. So the former is
> > > > > probably going to be fine for now.
> > > > >
> > > > > > Then I get paddr of the scmi channel for this domain and use
> > > > > > XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
> > > > > > > Hope I wass able to answer your question.
> > > > >
> > > > > What you provided me is how the guest OS will locate the shared memory. This
> > > > > still doesn't tell me which memory attribute will be used to map the page in
> > > > > Stage-1 (guest page-tables).
> > > > >
> > > > > To find that out, you want to look at the driver and how the mapping is
> > > > > done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
> > > > > (see smc_chan_setup()).
> > > > >
> > > > > Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
> > > > > (arm64) which is one of the most restrictive memory attribute.
> > > > >
> > > > > This means the firmware should be able to deal with the most restrictive
> > > > > attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
> > > > > page in stage-2 should be fine.
> > > > >
> > > >
> > > > I'm using vmap call to map channel memory (see smc_create_channel()).
> > > > vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
> > >
> > > You want to use ioremap().
> > >
> >
> > I've used ioremap originally, but changed it to vmap because ioremap
> > doesn't support memcpy.
> > What if I use __vmap with MT_DEVICE_nGnRE flag?
>
> That's not going to help. Our implementation of memcpy() is using unaligned
> access (which is forbidden on Device memory).
>
> You will need something similar to memcpy_toio() in Linux. I don't think we
> have one today in Xen, so I would suggest to import the implementation from
> Linux.
>
Ok. Then I'll import memcpy_toio from Linux kernel.
> >
> > > > Considering that protocol is synchronous and only one agent per channel is
> > > > expected - this works fine for now.
> > > > But I agree that the same memory attributes should be used in xen and
> > > > kernel. I fill fix that in v2.
> > >
> > > I am a bit confused. Are you mapping the full shared memory area in Xen? If
> > > yes, why do you need to map the memory that is going to be shared with a
> > > domain?
> > >
> >
> > Xen should have an access to all agent channels because it should send
> > SCMI_BASE_DISCOVER_AGENT to each channel and receive agent_id during
> > scmi_probe call.
>
> Hmmm... Just to confirm, this will only happen during Xen boot? IOW, Xen
> will never write to the channel when a domain is running?
>
Yes. Only during Xen boot.
> If yes, then I think it would be best to unmap the channel once they are
> used. This would prevent all sort of issues (e.g. Xen mistakenly written in
> them).
>
That's a good advise. Thank you.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2021-12-24 13:30 ` Julien Grall
@ 2022-01-19 9:40 ` Oleksii Moisieiev
2022-01-20 1:53 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-19 9:40 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Volodymyr Babchuk, Anthony PERARD, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
Hi Julien,
On Fri, Dec 24, 2021 at 02:30:50PM +0100, Julien Grall wrote:
> Hi,
>
> On 23/12/2021 20:06, Stefano Stabellini wrote:
> > On Wed, 22 Dec 2021, Stefano Stabellini wrote:
> > > # Future Ideas
> > >
> > > A great suggestion by Julien is to start supporting the dom0less partial
> > > device tree format in xl/libxl as well so that we can have a single
> > > "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
> > > dtdev).
> > >
> > > Even with that implemented, the user has still to provide a partial dtb,
> > > xen,reg and xen,path. I think this is a great step forward and we should
> > > do that, if nothing else to make it easier to switch between dom0less
> > > and normal domU configurations. But the number of options and
> > > information that the user has to provide is still similar to what we
> > > have today.
> >
> > I have just realized that if we start to parse the partial DTB in
> > xl/libxl the same way that we do for dom0less guests (parse "xen,path",
> > "xen,reg", and "interrupts", making dtdev, irqs and iomem optional)
> > actually we can achieve the goal below thanks to the combination:
> > "xen,path" + "xen,force-assign-without-iommu".
> >
> > In other words, with dom0less we already have a way to specify the link
> > to the host node even if the device is not a DMA master. We can do that
> > by specifying both xen,path and xen,force-assign-without-iommu for a
> > device.
> >
> > This is just FYI. I am not suggesting we should introduce dom0less-style
> > partial DTBs in order to get SCMI support in guests (although it would
> > be great to have). I think the best way forward for this series is one
> > of the combinations below, like a) + d), or a) + c)
>
> I strongly prefer a) + c) because a warning is easy to miss/ignore. At least
> with the extra property the user made an action to think about it and agree
> that this is the way do it.
>
> It is also easier to spot if we ask the user to provide the configuration
> file.
>
Let me share my thoughts about c), which is:
c) require force-assign-without-iommu="true" in dom.cfg
Adding this parameter to domain config means removing
xen,force-assign-without-iommu param from partial DTB. This will affect
dom0less configuration, which I can't test for now without extra effort.
What I suggest is to implement a) + d) in this patch series, which is:
a) extend dtdev to cover all devices, including non-DMA masters
d) or print a warning like:
"WARNING: device assignment safety for device XXX cannot be
verified. Please make sure XXX is not a DMA mastering device."
And introduce a) + c) with the next patch series where dom0less scmi
support will be done.
Maybe leave a comment in code that force-assign-without-iommu config parameter
should be implemened.
What do you think about this?
--
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-06 16:04 ` Julien Grall
2022-01-06 16:28 ` Oleksii Moisieiev
@ 2022-01-19 10:37 ` Oleksii Moisieiev
2022-01-20 2:10 ` Stefano Stabellini
1 sibling, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-19 10:37 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr, xen-devel, Stefano Stabellini, Volodymyr Babchuk,
Bertrand Marquis
Hi Julien,
On Thu, Jan 06, 2022 at 04:04:34PM +0000, Julien Grall wrote:
> Hi,
>
> On 06/01/2022 15:43, Oleksii Moisieiev wrote:
> > On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
> > >
> > >
> > > On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > >
> > > > On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> > > > > Hi,
> > > > >
> > > > > On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > > > > > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > > > > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > > > > > 2) What are the expected memory attribute for the regions?
> > > > > > > >
> > > > > > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > > > > > >
> > > > > > > I think you misunderstood my comment. Memory can be mapped with various type
> > > > > > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > > > > > the firmware expect? What will the guest OS usually?
> > > > > > >
> > > > > > > The reason I am asking is the attributes have to matched to avoid any
> > > > > > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > > > > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > > > > > memory access will be Device nGnRnE which is very strict.
> > > > > > >
> > > > > >
> > > > > > Let me share with you the configuration example:
> > > > > > scmi expects memory to be configured in the device-tree:
> > > > > >
> > > > > > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > > > > > };
> > > > > >
> > > > > > where XXXXXX address I allocate in alloc_magic_pages function.
> > > > >
> > > > > The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> > > > > is a guest physical address and then map a part of the shared page.
> > > >
> > > > Do you mean that I can't use alloc_magic_pages to allocate shared
> > > > memory region for SCMI?
> > > Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
> > > guest. From your description, this is not what you want because you will
> > > call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
> > >
> >
> > Ok thanks, I will refactor this part in v2.
> >
> > > >
> > > > >
> > > > > I can see two options here:
> > > > > 1) Hardcode the SCMI region in the memory map
> > > > > 2) Create a new region in the memory map that can be used for reserving
> > > > > memory for mapping.
> > > >
> > > > Could you please explain what do you mean under the "new region in the
> > > > memory map"?
> > >
> > > I mean reserving some guest physical address that could be used for map host
> > > physical address (e.g. SCMI region, GIC CPU interface...).
> > >
> > > So rather than hardcoding the address, we have something more flexible.
> > >
> >
> > Ok, I will fix that in v2.
>
> Just for avoidance of doubt. I was clarify option 2 and not requesting to
> implement. That said, if you want to implement option 2 I would be happy to
> review it.
>
I'm thinking about the best way to reserve address for the domain.
We have xen_pfn_t shared_info_pfn in struct xc_dom_image which is not
used for Arm architecture. It can be set from shared_info_arm callback,
defined in xg_dom_arm.c.
I can use shared_info to store address of the SCMI and then use map_sci_page to
call XEN_DOMCTL_memory_mapping.
This will allow me to reuse existing functionality and do not allocate
extra RAM.
What do you think about that?
--
Best regards,
Oleksii.
> >
> > > >
> > > > >
> > > > > We still have plenty of space in the guest memory map. So the former is
> > > > > probably going to be fine for now.
> > > > >
> > > > > > Then I get paddr of the scmi channel for this domain and use
> > > > > > XEN_DOMCTL_memory_mapping to map scmi channel address to gfn.
> > > > > > > Hope I wass able to answer your question.
> > > > >
> > > > > What you provided me is how the guest OS will locate the shared memory. This
> > > > > still doesn't tell me which memory attribute will be used to map the page in
> > > > > Stage-1 (guest page-tables).
> > > > >
> > > > > To find that out, you want to look at the driver and how the mapping is
> > > > > done. The Linux driver (drivers/firmware/arm_scmi) is using devm_ioremap()
> > > > > (see smc_chan_setup()).
> > > > >
> > > > > Under the hood, the function devm_ioremap() is using PROT_DEVICE_nGnRE
> > > > > (arm64) which is one of the most restrictive memory attribute.
> > > > >
> > > > > This means the firmware should be able to deal with the most restrictive
> > > > > attribute and therefore using XEN_DOMCTL_memory_mapping to map the shared
> > > > > page in stage-2 should be fine.
> > > > >
> > > >
> > > > I'm using vmap call to map channel memory (see smc_create_channel()).
> > > > vmap call sets PAGE_HYPERVISOR flag which sets MT_NORMAL (0x7) flag.
> > >
> > > You want to use ioremap().
> > >
> >
> > I've used ioremap originally, but changed it to vmap because ioremap
> > doesn't support memcpy.
> > What if I use __vmap with MT_DEVICE_nGnRE flag?
>
> That's not going to help. Our implementation of memcpy() is using unaligned
> access (which is forbidden on Device memory).
>
> You will need something similar to memcpy_toio() in Linux. I don't think we
> have one today in Xen, so I would suggest to import the implementation from
> Linux.
>
> >
> > > > Considering that protocol is synchronous and only one agent per channel is
> > > > expected - this works fine for now.
> > > > But I agree that the same memory attributes should be used in xen and
> > > > kernel. I fill fix that in v2.
> > >
> > > I am a bit confused. Are you mapping the full shared memory area in Xen? If
> > > yes, why do you need to map the memory that is going to be shared with a
> > > domain?
> > >
> >
> > Xen should have an access to all agent channels because it should send
> > SCMI_BASE_DISCOVER_AGENT to each channel and receive agent_id during
> > scmi_probe call.
>
> Hmmm... Just to confirm, this will only happen during Xen boot? IOW, Xen
> will never write to the channel when a domain is running?
>
> If yes, then I think it would be best to unmap the channel once they are
> used. This would prevent all sort of issues (e.g. Xen mistakenly written in
> them).
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 18:45 ` Volodymyr Babchuk
2021-12-23 19:06 ` Oleksii Moisieiev
@ 2022-01-19 12:04 ` Oleksii Moisieiev
2022-01-20 1:28 ` Stefano Stabellini
2 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-19 12:04 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > Hi Stefano,
> > > >
> > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > Hi Stefano,
> > > > > >
> > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > power-domains without access to CPG.
> > > > > > > >
> > > > > > > > The following features are implemented:
> > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > resets and power-domains via SCMI;
> > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > >
> > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > ---
> > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > >
> > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > SCI mediators implemented in XEN.
> > > > > > > >
> > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > +
> > > > > > > > endmenu
> > > > > > > >
> > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..9563067ddc
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > +config SCMI_SMC
> > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > + default n
> > > > > > > > + depends on SCI
> > > > > > > > + ---help---
> > > > > > > > +
> > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > + for communication.
> > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > @@ -1 +1,2 @@
> > > > > > > > obj-y += sci.o
> > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > +/*
> > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > + *
> > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > + *
> > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > + *
> > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > + * (at your option) any later version.
> > > > > > > > + *
> > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > + * GNU General Public License for more details.
> > > > > > > > + */
> > > > > > > > +
> > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > +#include <asm/smccc.h>
> > > > > > > > +#include <asm/io.h>
> > > > > > > > +#include <xen/bitops.h>
> > > > > > > > +#include <xen/config.h>
> > > > > > > > +#include <xen/sched.h>
> > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > +#include <xen/iocap.h>
> > > > > > > > +#include <xen/init.h>
> > > > > > > > +#include <xen/err.h>
> > > > > > > > +#include <xen/lib.h>
> > > > > > > > +#include <xen/list.h>
> > > > > > > > +#include <xen/mm.h>
> > > > > > > > +#include <xen/string.h>
> > > > > > > > +#include <xen/time.h>
> > > > > > > > +#include <xen/vmap.h>
> > > > > > > > +
> > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > +
> > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > +
> > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > +
> > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > >
> > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > >
> > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > >
> > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > seems to be "arm,scmi-shmem".
> > > > > > >
> > > > > >
> > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > channels:
> > > > > >
> > > > > > reserved-memory {
> > > > > > /* reserved region for scmi channels*/
> > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > no-map;
> > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > };
> > > > > > };
> > > > > >
> > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > the current scmi channel:
> > > > > >
> > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > };
> > > > > >
> > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > assigned to this agent.
> > > > >
> > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > compatible string, not as a node name, and it would need to be described
> > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > >
> > > > > But from your description I don't think it is necessary. We can just use
> > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > >
> > > > > reserved-memory {
> > > > > scp-shmem@0x53FF0000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF1000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF2000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > };
> > > > > ...
> > > > >
> > > > > In other words, if all the individual channel pages are described as
> > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > "linux,scmi_mem"?
> > > > >
> > > >
> > > > That was my first implementation. But I've met a problem with
> > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > there were some if, checking if memory weren't reserved.
> > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > shmem page.
> > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > which has no compatible string and provides no-map property.
> > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > space for the domain.
> > > >
> > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > and scmi_devid property to the
> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > Those node and property are needed only for Xen and useless for
> > > > non-virtualized systems. I can add this node and property description to
> > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > What do you think about it?
> > >
> > > Reply below
> > >
> > > [...]
> > >
> > >
> > > > > In general we can't use properties that are not part of the device tree
> > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > >
> > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > activities to get "linux,scmi_mem" upstream under
> > > > > Documentation/devicetree/bindings in Linux?
> > > > >
> > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > Documentation/devicetree/bindings (probably
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > work on the Xen code that makes use of it.
> > > > >
> > > > > Does it make sense?
> > > > >
> > > >
> > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > >
> > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > Xen specific. In general, it would be best not to introduce Xen specific
> > > properties into generic bindings. It is a problem both from a
> > > specification perspective (because it has hard to handle Xen specific
> > > cases in fully generic bindings, especially as those bindings are
> > > maintained as part of the Linux kernel) and from a user perspective
> > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > modify the host dtb to add Xen-specific information by hand.)
> > >
> > >
> > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > looks like a generic property that should be needed for the Linux SCMI
> > > driver too. Why the Linux driver doesn't need it?
> > >
> >
> > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > On non-virtualized systems - there is no need of this call, because OS
> > is the only one entity, running on the system.
>
> OK. Even if it is only required for virtualized systems, I think that
> scmi_devid is important enough that should be part of the upstream
> binding. I think it is worth starting an email thread on the LKML with
> Rob Herring and the SCMI maintainers to discuss the addition of
> scmi_devid to the binding.
>
>
> > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > list of device_ids to dom.cfg, such as:
> > sci_devs = [ 0, 1, 15, 35 ];
> >
> > Using this approach, we can remove scmi_devid from the device tree and
> > just pass a list of scmi_devids to XEN using additional hypercall.
> > We can probably make hypercall taking devid list as input parameter.
> > This will take only 1 hypercall to setup sci permissions.
>
> But how would a user know which are the right SCMI IDs to add to the
> sci_devs list? Would the user have to go and read the reference manual
> of the platform to find the SCMI IDs and then write sci_devs by hand?
> If that is the case, then I think that it would be better to add
> scmi_devid to device tree.
>
> In general, I think this configuration should happen automatically
> without user intervention. The user should just specify "enable SCMI"
> and it should work.
>
>
> > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > way, I don't think we should need linux,scmi_mem.
> >
> > This requires further investigation. I will try to make implementation
> > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > reuslts with you.
>
> OK, thanks.
Hi Stefano,
As I did some investigation about using reserved-memory area
linux,scmi_mem and now I need your advice.
I see 2 possible implementations for now:
1) Add memory-region parameter to cpu_scp_shm node which points to the
reserved memory region.
So device-tree will look like this:
reserved-memory {
/* reserved region for scmi channels*/
scmi_memory: region@53FF0000{
no-map;
reg = <0x0 0x53FF0000 0x0 0x10000>;
};
};
cpu_scp_shm: scp-shmem@0x53FF0000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF0000 0x0 0x1000>;
memory-region = <&scmi_memory>;
};
So cpu_scp_shm node has a reference to scmi_memory region. This mean
that xen can find reserved memory region without adding additional names
to the device-tree bindings.
memory-region parameter as a reference to reserved memory and region
creation described in:
https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
This approach I've implemented already and it works.
2) The second approach is the format you suggested:
> > > > > reserved-memory {
> > > > > scp-shmem@0x53FF0000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF1000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > };
> > > > > scp-shmem@0x53FF2000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > };
> > > > > ...
This approach has an advantage that xen ARM_SCI driver do not know about
how channels are placed in the reserved memory, but introduces some
disadvantages:
a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
in the device-tree. In current implementation I have separate scmi.dtsi
file which introduces scmi support for both XEN-based and
non-virtualized systems. Having 14 extra channels in the device-tree may
be confusing.
b) In case if we have all 15 channels, described in partial device-tree,
we should not copy any node to the domain device-tree. I think it will
be better to generate arm,scmi-shmem node in the Domain device-tree. The
problem is that arm,scmi-smc node, which is using arm,scmi-shmem node
can't be generated. I prefer it to be copied from the partial
device-tree because it includes some platform specific configuration,
such as func-id and list of the protocols (for example different
platforms may require different list of the protocols). So in this case
we will have 1 node copied and 1 node generated.
I think even for dom0less we should use arm,scmi-smc node from the
device-tree because protocol configuration and funcid is related to the
platform.
I prefer the second approach and will try to make it if it's OK to copy
arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
node.
What do you think about that?
Also I wanted to mention that I'm not planning to make ARM_SCI support for
dom0less in terms of this patch series bacause I can't test
dom0less configuration for now. So let me know if some of my
functionality breaks dom0less.
--
Best regards
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-19 12:04 ` Oleksii Moisieiev
@ 2022-01-20 1:28 ` Stefano Stabellini
2022-01-20 10:21 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-20 1:28 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > Hi Stefano,
> > > > >
> > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > Hi Stefano,
> > > > > > >
> > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > power-domains without access to CPG.
> > > > > > > > >
> > > > > > > > > The following features are implemented:
> > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > ---
> > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > >
> > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > >
> > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > +
> > > > > > > > > endmenu
> > > > > > > > >
> > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > +config SCMI_SMC
> > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > + default n
> > > > > > > > > + depends on SCI
> > > > > > > > > + ---help---
> > > > > > > > > +
> > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > + for communication.
> > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > obj-y += sci.o
> > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > +/*
> > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > + *
> > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > + *
> > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > + *
> > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > + * (at your option) any later version.
> > > > > > > > > + *
> > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > + */
> > > > > > > > > +
> > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > +#include <asm/io.h>
> > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > +#include <xen/config.h>
> > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > +#include <xen/init.h>
> > > > > > > > > +#include <xen/err.h>
> > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > +#include <xen/list.h>
> > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > +#include <xen/string.h>
> > > > > > > > > +#include <xen/time.h>
> > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > +
> > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > +
> > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > +
> > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > +
> > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > >
> > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > >
> > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > >
> > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > >
> > > > > > >
> > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > channels:
> > > > > > >
> > > > > > > reserved-memory {
> > > > > > > /* reserved region for scmi channels*/
> > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > no-map;
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > };
> > > > > > > };
> > > > > > >
> > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > the current scmi channel:
> > > > > > >
> > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > };
> > > > > > >
> > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > assigned to this agent.
> > > > > >
> > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > >
> > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > >
> > > > > > reserved-memory {
> > > > > > scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF1000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF2000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > };
> > > > > > ...
> > > > > >
> > > > > > In other words, if all the individual channel pages are described as
> > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > "linux,scmi_mem"?
> > > > > >
> > > > >
> > > > > That was my first implementation. But I've met a problem with
> > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > there were some if, checking if memory weren't reserved.
> > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > shmem page.
> > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > which has no compatible string and provides no-map property.
> > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > space for the domain.
> > > > >
> > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > and scmi_devid property to the
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > Those node and property are needed only for Xen and useless for
> > > > > non-virtualized systems. I can add this node and property description to
> > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > What do you think about it?
> > > >
> > > > Reply below
> > > >
> > > > [...]
> > > >
> > > >
> > > > > > In general we can't use properties that are not part of the device tree
> > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > >
> > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > Documentation/devicetree/bindings in Linux?
> > > > > >
> > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > Documentation/devicetree/bindings (probably
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > work on the Xen code that makes use of it.
> > > > > >
> > > > > > Does it make sense?
> > > > > >
> > > > >
> > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > >
> > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > properties into generic bindings. It is a problem both from a
> > > > specification perspective (because it has hard to handle Xen specific
> > > > cases in fully generic bindings, especially as those bindings are
> > > > maintained as part of the Linux kernel) and from a user perspective
> > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > modify the host dtb to add Xen-specific information by hand.)
> > > >
> > > >
> > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > looks like a generic property that should be needed for the Linux SCMI
> > > > driver too. Why the Linux driver doesn't need it?
> > > >
> > >
> > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > On non-virtualized systems - there is no need of this call, because OS
> > > is the only one entity, running on the system.
> >
> > OK. Even if it is only required for virtualized systems, I think that
> > scmi_devid is important enough that should be part of the upstream
> > binding. I think it is worth starting an email thread on the LKML with
> > Rob Herring and the SCMI maintainers to discuss the addition of
> > scmi_devid to the binding.
> >
> >
> > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > list of device_ids to dom.cfg, such as:
> > > sci_devs = [ 0, 1, 15, 35 ];
> > >
> > > Using this approach, we can remove scmi_devid from the device tree and
> > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > We can probably make hypercall taking devid list as input parameter.
> > > This will take only 1 hypercall to setup sci permissions.
> >
> > But how would a user know which are the right SCMI IDs to add to the
> > sci_devs list? Would the user have to go and read the reference manual
> > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > If that is the case, then I think that it would be better to add
> > scmi_devid to device tree.
> >
> > In general, I think this configuration should happen automatically
> > without user intervention. The user should just specify "enable SCMI"
> > and it should work.
> >
> >
> > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > way, I don't think we should need linux,scmi_mem.
> > >
> > > This requires further investigation. I will try to make implementation
> > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > reuslts with you.
> >
> > OK, thanks.
>
> Hi Stefano,
>
> As I did some investigation about using reserved-memory area
> linux,scmi_mem and now I need your advice.
>
> I see 2 possible implementations for now:
> 1) Add memory-region parameter to cpu_scp_shm node which points to the
> reserved memory region.
> So device-tree will look like this:
>
> reserved-memory {
> /* reserved region for scmi channels*/
> scmi_memory: region@53FF0000{
> no-map;
> reg = <0x0 0x53FF0000 0x0 0x10000>;
> };
> };
> cpu_scp_shm: scp-shmem@0x53FF0000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF0000 0x0 0x1000>;
> memory-region = <&scmi_memory>;
> };
>
> So cpu_scp_shm node has a reference to scmi_memory region. This mean
> that xen can find reserved memory region without adding additional names
> to the device-tree bindings.
> memory-region parameter as a reference to reserved memory and region
> creation described in:
> https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
>
> This approach I've implemented already and it works.
This approach would require a discussion with the upstream device tree
maintainers. Likely, we would need to add a note about the usage of the
"memory-region" property to arm,scmi.yaml.
Also, I have the feeling that they would ask to add the "memory-region"
property directly to the "arm,scmi-smc" node, as an alternative (or
in addition) to the existing "shmem" property.
That said, from my point of view this approach is also a viable option.
I don't see any major problems.
The main question (after reading everything else that you wrote below)
is whether the "arm,scmi-smc" node in this case could be automatically
generated.
> 2) The second approach is the format you suggested:
> > > > > > reserved-memory {
> > > > > > scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF1000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > };
> > > > > > scp-shmem@0x53FF2000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > };
> > > > > > ...
>
> This approach has an advantage that xen ARM_SCI driver do not know about
> how channels are placed in the reserved memory, but introduces some
> disadvantages:
> a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> in the device-tree. In current implementation I have separate scmi.dtsi
> file which introduces scmi support for both XEN-based and
> non-virtualized systems. Having 14 extra channels in the device-tree may
> be confusing.
I can see that while it would be ideal for Xen to see all 14+1 channels
in device tree (on the host device tree), we wouldn't want to expose all
of them to the domains, not even to dom0.
How many channels do we want dom0 to see by the way? For this
discussion, I'll just assume for now that dom0 only sees 1 channel like
the domUs.
Now we have a problem: how do we go about "filtering" the
"arm,scmi-shmem" device tree nodes? Which is also what you are asking
below in point b).
> b) In case if we have all 15 channels, described in partial device-tree,
I think you meant "described in the host device tree", right?
> we should not copy any node to the domain device-tree. I think it will
> be better to generate arm,scmi-shmem node in the Domain device-tree.
Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
device tree description for the DomU/Dom0 based on the channels
allocated to the domain.
> The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> node can't be generated. I prefer it to be copied from the partial
> device-tree because it includes some platform specific configuration,
> such as func-id and list of the protocols (for example different
> platforms may require different list of the protocols). So in this
> case we will have 1 node copied and 1 node generated.
>
> I think even for dom0less we should use arm,scmi-smc node from the
> device-tree because protocol configuration and funcid is related to the
> platform.
I am not sure I understood what you wrote. You are saying that the
"arm,scmi-smc" node includes some platform specific configurations so
it cannot be automatically generated by Xen (or by the tools) and
instead it needs to be manually provided as part of the partial dtb for
the domU. Is that correct?
If so, I would like to understand the reasons behind it. Manual
device tree editing is problematic.
I looked for "func-id" in
Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
find any results. Do you have an example of the platform specific
configuration or protocol configuration that would make it difficult to
automatically generate the "arm,scmi-smc" node for the domains?
Also, is this a problem just for approach #2 or also for approach #1?
If it is a problem only for approach #2, then let's just go with
approach #1.
> I prefer the second approach and will try to make it if it's OK to copy
> arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> node.
>
> What do you think about that?
From a device tree specification perspective, I think both approaches
are OK (with a minor comment on the first approach as I wrote above.)
But from a Xen perspective I think it is important that we don't require
the user to manually provide the SCMI configuration in the partial DTB.
It would be better if we could generate it automatically from Xen or the
tools (or even an independent script). Or copy the "arm,scmi-smc" node
from the host device tree to the domU device tree without modifications.
So if using approach #1 allows us to automatically generate the
"arm,scmi-smc" node for the guest, then I think it's best for sure.
> Also I wanted to mention that I'm not planning to make ARM_SCI support for
> dom0less in terms of this patch series bacause I can't test
> dom0less configuration for now. So let me know if some of my
> functionality breaks dom0less.
That's fine. I don't mean to scope-creep your patch series, which is
extremely valuable as is.
That said, I would be happy to provide you with a very simple dom0less
configuration for your platform to enable you to test, or alternatively
I could write a patch to add dom0less domU support if you are happy to
help reviewing and testing it.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2022-01-19 9:40 ` Oleksii Moisieiev
@ 2022-01-20 1:53 ` Stefano Stabellini
2022-01-20 10:27 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-20 1:53 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Julien Grall, Stefano Stabellini, Volodymyr Babchuk,
Anthony PERARD, Oleksandr, xen-devel, Wei Liu, Juergen Gross,
Andrew Cooper, George Dunlap, Jan Beulich, Bertrand Marquis
On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> On Fri, Dec 24, 2021 at 02:30:50PM +0100, Julien Grall wrote:
> > Hi,
> >
> > On 23/12/2021 20:06, Stefano Stabellini wrote:
> > > On Wed, 22 Dec 2021, Stefano Stabellini wrote:
> > > > # Future Ideas
> > > >
> > > > A great suggestion by Julien is to start supporting the dom0less partial
> > > > device tree format in xl/libxl as well so that we can have a single
> > > > "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
> > > > dtdev).
> > > >
> > > > Even with that implemented, the user has still to provide a partial dtb,
> > > > xen,reg and xen,path. I think this is a great step forward and we should
> > > > do that, if nothing else to make it easier to switch between dom0less
> > > > and normal domU configurations. But the number of options and
> > > > information that the user has to provide is still similar to what we
> > > > have today.
> > >
> > > I have just realized that if we start to parse the partial DTB in
> > > xl/libxl the same way that we do for dom0less guests (parse "xen,path",
> > > "xen,reg", and "interrupts", making dtdev, irqs and iomem optional)
> > > actually we can achieve the goal below thanks to the combination:
> > > "xen,path" + "xen,force-assign-without-iommu".
> > >
> > > In other words, with dom0less we already have a way to specify the link
> > > to the host node even if the device is not a DMA master. We can do that
> > > by specifying both xen,path and xen,force-assign-without-iommu for a
> > > device.
> > >
> > > This is just FYI. I am not suggesting we should introduce dom0less-style
> > > partial DTBs in order to get SCMI support in guests (although it would
> > > be great to have). I think the best way forward for this series is one
> > > of the combinations below, like a) + d), or a) + c)
> >
> > I strongly prefer a) + c) because a warning is easy to miss/ignore. At least
> > with the extra property the user made an action to think about it and agree
> > that this is the way do it.
> >
> > It is also easier to spot if we ask the user to provide the configuration
> > file.
> >
>
> Let me share my thoughts about c), which is:
> c) require force-assign-without-iommu="true" in dom.cfg
>
> Adding this parameter to domain config means removing
> xen,force-assign-without-iommu param from partial DTB.
Why? No I don't think so.
> This will affect dom0less configuration, which I can't test for now
> without extra effort.
We are just talking about adding:
force-assign-without-iommu="true"
to the xl config file. For instance:
dtdev = [ "/amba/serial@ff000000" ]
iomem = ["0xff000,1"]
force-assign-without-iommu="true"
It is unrelated to the dom0less partial DTB. There is no need to test
dom0less to do this.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-19 10:37 ` Oleksii Moisieiev
@ 2022-01-20 2:10 ` Stefano Stabellini
2022-01-20 10:25 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-20 2:10 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Julien Grall, Oleksandr, xen-devel, Stefano Stabellini,
Volodymyr Babchuk, Bertrand Marquis
On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> On Thu, Jan 06, 2022 at 04:04:34PM +0000, Julien Grall wrote:
> > On 06/01/2022 15:43, Oleksii Moisieiev wrote:
> > > On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
> > > > On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> > > > > Hi Julien,
> > > > >
> > > > > On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > > > > > > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > > > > > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > > > > > > 2) What are the expected memory attribute for the regions?
> > > > > > > > >
> > > > > > > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > > > > > > >
> > > > > > > > I think you misunderstood my comment. Memory can be mapped with various type
> > > > > > > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > > > > > > the firmware expect? What will the guest OS usually?
> > > > > > > >
> > > > > > > > The reason I am asking is the attributes have to matched to avoid any
> > > > > > > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > > > > > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > > > > > > memory access will be Device nGnRnE which is very strict.
> > > > > > > > :w
> > > > > > >
> > > > > > > Let me share with you the configuration example:
> > > > > > > scmi expects memory to be configured in the device-tree:
> > > > > > >
> > > > > > > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > > > > > > };
> > > > > > >
> > > > > > > where XXXXXX address I allocate in alloc_magic_pages function.
> > > > > >
> > > > > > The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> > > > > > is a guest physical address and then map a part of the shared page.
> > > > >
> > > > > Do you mean that I can't use alloc_magic_pages to allocate shared
> > > > > memory region for SCMI?
> > > > Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
> > > > guest. From your description, this is not what you want because you will
> > > > call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
> > > >
> > >
> > > Ok thanks, I will refactor this part in v2.
> > >
> > > > >
> > > > > >
> > > > > > I can see two options here:
> > > > > > 1) Hardcode the SCMI region in the memory map
> > > > > > 2) Create a new region in the memory map that can be used for reserving
> > > > > > memory for mapping.
> > > > >
> > > > > Could you please explain what do you mean under the "new region in the
> > > > > memory map"?
> > > >
> > > > I mean reserving some guest physical address that could be used for map host
> > > > physical address (e.g. SCMI region, GIC CPU interface...).
> > > >
> > > > So rather than hardcoding the address, we have something more flexible.
> > > >
> > >
> > > Ok, I will fix that in v2.
> >
> > Just for avoidance of doubt. I was clarify option 2 and not requesting to
> > implement. That said, if you want to implement option 2 I would be happy to
> > review it.
> >
>
> I'm thinking about the best way to reserve address for the domain.
> We have xen_pfn_t shared_info_pfn in struct xc_dom_image which is not
> used for Arm architecture. It can be set from shared_info_arm callback,
> defined in xg_dom_arm.c.
> I can use shared_info to store address of the SCMI and then use map_sci_page to
> call XEN_DOMCTL_memory_mapping.
>
> This will allow me to reuse existing functionality and do not allocate
> extra RAM.
>
> What do you think about that?
I cannot speak for Julien but I think he meant something else (Julien
please correct me if I am wrong.) Exposing addresses via device tree is
not a problem.
Normally we pick a fixed address for guest resources, for instance
GUEST_GICD_BASE, see xen/include/public/arch-arm.h. We could do that for
SCMI as well and it is basically approach 1).
However, it is a bit inflexible and could cause issues with things like
direct-map (https://marc.info/?l=xen-devel&m=163997768108997). A more
flexible way would be for the SCMI guest address to be dynamically
generated somehow.
I am not sure how Julien envisioned the address to be generated exactly.
Thanks to Oleksandr's work we have a way to find large regions of "free"
address space. It is currently used for grant-table mappings. Maybe we
could use a subset of it for SCMI? It might be best to wait for Julien's
answer as he might have a better idea.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-20 1:28 ` Stefano Stabellini
@ 2022-01-20 10:21 ` Oleksii Moisieiev
2022-01-20 22:29 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-20 10:21 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > Hi Stefano,
> > > > > >
> > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > Hi Stefano,
> > > > > > > >
> > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > >
> > > > > > > > > > The following features are implemented:
> > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > ---
> > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > >
> > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > >
> > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > +
> > > > > > > > > > endmenu
> > > > > > > > > >
> > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > + default n
> > > > > > > > > > + depends on SCI
> > > > > > > > > > + ---help---
> > > > > > > > > > +
> > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > + for communication.
> > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > obj-y += sci.o
> > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > +/*
> > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > + *
> > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > + *
> > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > + *
> > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > + *
> > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > + */
> > > > > > > > > > +
> > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > +
> > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > +
> > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > +
> > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > +
> > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > >
> > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > >
> > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > >
> > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > >
> > > > > > > >
> > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > channels:
> > > > > > > >
> > > > > > > > reserved-memory {
> > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > no-map;
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > };
> > > > > > > > };
> > > > > > > >
> > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > the current scmi channel:
> > > > > > > >
> > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > >
> > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > assigned to this agent.
> > > > > > >
> > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > >
> > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > >
> > > > > > > reserved-memory {
> > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > };
> > > > > > > ...
> > > > > > >
> > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > "linux,scmi_mem"?
> > > > > > >
> > > > > >
> > > > > > That was my first implementation. But I've met a problem with
> > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > there were some if, checking if memory weren't reserved.
> > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > shmem page.
> > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > which has no compatible string and provides no-map property.
> > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > space for the domain.
> > > > > >
> > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > and scmi_devid property to the
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > Those node and property are needed only for Xen and useless for
> > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > What do you think about it?
> > > > >
> > > > > Reply below
> > > > >
> > > > > [...]
> > > > >
> > > > >
> > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > >
> > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > >
> > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > work on the Xen code that makes use of it.
> > > > > > >
> > > > > > > Does it make sense?
> > > > > > >
> > > > > >
> > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > >
> > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > properties into generic bindings. It is a problem both from a
> > > > > specification perspective (because it has hard to handle Xen specific
> > > > > cases in fully generic bindings, especially as those bindings are
> > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > >
> > > > >
> > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > driver too. Why the Linux driver doesn't need it?
> > > > >
> > > >
> > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > On non-virtualized systems - there is no need of this call, because OS
> > > > is the only one entity, running on the system.
> > >
> > > OK. Even if it is only required for virtualized systems, I think that
> > > scmi_devid is important enough that should be part of the upstream
> > > binding. I think it is worth starting an email thread on the LKML with
> > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > scmi_devid to the binding.
> > >
> > >
> > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > list of device_ids to dom.cfg, such as:
> > > > sci_devs = [ 0, 1, 15, 35 ];
> > > >
> > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > We can probably make hypercall taking devid list as input parameter.
> > > > This will take only 1 hypercall to setup sci permissions.
> > >
> > > But how would a user know which are the right SCMI IDs to add to the
> > > sci_devs list? Would the user have to go and read the reference manual
> > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > If that is the case, then I think that it would be better to add
> > > scmi_devid to device tree.
> > >
> > > In general, I think this configuration should happen automatically
> > > without user intervention. The user should just specify "enable SCMI"
> > > and it should work.
> > >
> > >
> > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > way, I don't think we should need linux,scmi_mem.
> > > >
> > > > This requires further investigation. I will try to make implementation
> > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > reuslts with you.
> > >
> > > OK, thanks.
> >
> > Hi Stefano,
> >
> > As I did some investigation about using reserved-memory area
> > linux,scmi_mem and now I need your advice.
> >
> > I see 2 possible implementations for now:
> > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > reserved memory region.
> > So device-tree will look like this:
> >
> > reserved-memory {
> > /* reserved region for scmi channels*/
> > scmi_memory: region@53FF0000{
> > no-map;
> > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > };
> > };
> > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > memory-region = <&scmi_memory>;
> > };
> >
> > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > that xen can find reserved memory region without adding additional names
> > to the device-tree bindings.
> > memory-region parameter as a reference to reserved memory and region
> > creation described in:
> > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> >
> > This approach I've implemented already and it works.
>
> This approach would require a discussion with the upstream device tree
> maintainers. Likely, we would need to add a note about the usage of the
> "memory-region" property to arm,scmi.yaml.
>
> Also, I have the feeling that they would ask to add the "memory-region"
> property directly to the "arm,scmi-smc" node, as an alternative (or
> in addition) to the existing "shmem" property.
>
> That said, from my point of view this approach is also a viable option.
> I don't see any major problems.
>
> The main question (after reading everything else that you wrote below)
> is whether the "arm,scmi-smc" node in this case could be automatically
> generated.
>
arm,scmi-smc node can be generated in both cases. I think I'd leave it
as backup in case if the second approach will not work.
>
> > 2) The second approach is the format you suggested:
> > > > > > > reserved-memory {
> > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > };
> > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > };
> > > > > > > ...
> >
> > This approach has an advantage that xen ARM_SCI driver do not know about
> > how channels are placed in the reserved memory, but introduces some
> > disadvantages:
> > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > in the device-tree. In current implementation I have separate scmi.dtsi
> > file which introduces scmi support for both XEN-based and
> > non-virtualized systems. Having 14 extra channels in the device-tree may
> > be confusing.
>
> I can see that while it would be ideal for Xen to see all 14+1 channels
> in device tree (on the host device tree), we wouldn't want to expose all
> of them to the domains, not even to dom0.
>
> How many channels do we want dom0 to see by the way? For this
> discussion, I'll just assume for now that dom0 only sees 1 channel like
> the domUs.
For dom0 we need only one channel.
>
> Now we have a problem: how do we go about "filtering" the
> "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> below in point b).
>
Xen will not need to filter "arm,scmi-shmem" node. It will just
create shmem node in Domain device-tree. I don't see any problem for xen
configuration here.
What bothers me here is that I set scmi configuration for platform dts,
not for xen or domu dts files.
So for example I have the following structure of the dts files for my
platform (r8a77961-salvator-xs):
* r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
scmi_devid for the devices, that should use scmi.
* r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
is used for system with no hypervisor.
* r8a77961-salvator-xs-xen.dts - dts file for xen which includes
r8a77961-salvator-xs.dts and inherits scmi configuration from it.
* r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
r8a77961-salvator-xs.dts and inherits scmi configuration from it.
In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
I can give you a link to Merge request with this changes if you need it.
For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
nodes will be omitted and new will be generated for the domains.
What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
Just got an idea while writing this: I can create only one
"arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
Then we will have valid configurations for all cases.
This can be a solution. What do you think?
>
> > b) In case if we have all 15 channels, described in partial device-tree,
>
> I think you meant "described in the host device tree", right?
>
Yeah that's what I've meant.
>
> > we should not copy any node to the domain device-tree. I think it will
> > be better to generate arm,scmi-shmem node in the Domain device-tree.
>
> Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> device tree description for the DomU/Dom0 based on the channels
> allocated to the domain.
>
>
> > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > node can't be generated. I prefer it to be copied from the partial
> > device-tree because it includes some platform specific configuration,
> > such as func-id and list of the protocols (for example different
> > platforms may require different list of the protocols). So in this
> > case we will have 1 node copied and 1 node generated.
> >
> > I think even for dom0less we should use arm,scmi-smc node from the
> > device-tree because protocol configuration and funcid is related to the
> > platform.
>
> I am not sure I understood what you wrote. You are saying that the
> "arm,scmi-smc" node includes some platform specific configurations so
> it cannot be automatically generated by Xen (or by the tools) and
> instead it needs to be manually provided as part of the partial dtb for
> the domU. Is that correct?
>
> If so, I would like to understand the reasons behind it. Manual
> device tree editing is problematic.
>
> I looked for "func-id" in
> Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> find any results. Do you have an example of the platform specific
> configuration or protocol configuration that would make it difficult to
> automatically generate the "arm,scmi-smc" node for the domains?
Sorry, I used wrong term (used term from the specification), arm,smc-id
of cause.
>
> Also, is this a problem just for approach #2 or also for approach #1?
> If it is a problem only for approach #2, then let's just go with
> approach #1.
>
We can't copy "arm,scmi-smc" in both approaches. The difference is that
in the first approach we can copy both "arm,scmi-smc" and
"arm,scmi-shmem" nodes while in the second approach we should copy
"arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
arm,scmi-smc node can't be generated because it includes properties and
configurations that depends from platform and should be get from the
device tree.
Here is "arm,scmi-smc" node expample:
firmware {
scmi {
compatible = "arm,scmi-smc"
arm,smc-id = <0x82000002>;
shmem = <&cpu_scp_shm>;
#address-cells = <1>;
#size-cells = <0>;
scmi_power: protocol@11 {
reg = <0x11>;
#power-domain-cells = <1>;
};
scmi_clock: protocol@14 {
...
};
scmi_reset: protocol@16 {
...
};
...
};
};
It has 3 configurable options:
* arm,smc-id parameter, setting func_id for scmi protocol. This id can be
different for different platforms.
For example stm32mp1 architecture use different scm-id for different
agents:
https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h#L39
* shmem which includes phandle to arm,scmi-shmem node. But this is not
a problem and can be updated.
* list of the protocol subnodes. This is also configurable parameter,
not regs or names, but the number of the protocols. For example onle
platform can use power-domains/clock/resets via scmi, when another will
require volage-control and sensor-management to be added.
Xen should know this parameters to be able to generate "arm,scmi-smc" node.
Also we're currently discussing new scmi protocol with ARM: Pinctrl over
SCMI.
It should allow domains to access pinctrl subsystem, placed in Firmware
through SCMI protocol.
scmi_pinctrl node will look like this:
firmware {
scmi {
...
scmi_pinctrl: protocol@18 {
reg = <0x18>;
#pinctrl-cells = <0>;
i2c2_pins: i2c2 {
groups = <74>; /* i2c2_a */
function = <15>; /* i2c2 */
};
irq0_pins: irq0 {
groups = <81>; /* intc_ex_irq0 */
function = <19>; /* intc_ex */
};
avb_pins: avb {
mux {
/* avb_link, avb_mdio, avb_mii */
groups = <17>, <21>, <22>;
function = <1>; /* avb */
};
pins_mdio {
groups = <21>; /* avb_mdio */
drive-strength = <24>;
};
pins_mii_tx {
/* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
pins = <242>, <240>, <236>, <237>, <238>, <239>;
drive-strength = <12>;
};
};
...
};
};
};
So "arm,scmi-smc" node will have even more platform specific settings.
>
> > I prefer the second approach and will try to make it if it's OK to copy
> > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > node.
> >
> > What do you think about that?
>
> From a device tree specification perspective, I think both approaches
> are OK (with a minor comment on the first approach as I wrote above.)
>
> But from a Xen perspective I think it is important that we don't require
> the user to manually provide the SCMI configuration in the partial DTB.
> It would be better if we could generate it automatically from Xen or the
> tools (or even an independent script). Or copy the "arm,scmi-smc" node
> from the host device tree to the domU device tree without modifications.
I think copy "arm,scmi-smc" node is the only option we have.
I'm not sure what do you mean under "host device tree" if you mean Xen
device-tree - then I think it will not cover the case with stm32mp1 I've
mentioned above. I think it will be better to copy "arm,scmi-smc" node
from Domu partial Device-tree to Domu device-tree.
So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
AGENT1 scm-is set in domu device-tree and copied to dom-u.
Do you agree with my points?
>
> So if using approach #1 allows us to automatically generate the
> "arm,scmi-smc" node for the guest, then I think it's best for sure.
>
Summarizing all written above I would focus on the second approach
and put aside the first approach implementation. If you don't mind.
>
> > Also I wanted to mention that I'm not planning to make ARM_SCI support for
> > dom0less in terms of this patch series bacause I can't test
> > dom0less configuration for now. So let me know if some of my
> > functionality breaks dom0less.
>
> That's fine. I don't mean to scope-creep your patch series, which is
> extremely valuable as is.
>
> That said, I would be happy to provide you with a very simple dom0less
> configuration for your platform to enable you to test, or alternatively
> I could write a patch to add dom0less domU support if you are happy to
> help reviewing and testing it.
I was thinking about making dom0less support in the different
patch-series because there are still questions to be discussed.
For example, how arm,scmi-smc node will be generated for DomUs and how
the case, when scmi configuration is different for DomU1 and DomU2 (as
in case of stm32mp1 when smc-id is different) should be handled.
What do you think about continue without dom0less support and discuss
dom0less once we done with the main part?
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-20 2:10 ` Stefano Stabellini
@ 2022-01-20 10:25 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-20 10:25 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Oleksandr, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On Wed, Jan 19, 2022 at 06:10:46PM -0800, Stefano Stabellini wrote:
> On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > On Thu, Jan 06, 2022 at 04:04:34PM +0000, Julien Grall wrote:
> > > On 06/01/2022 15:43, Oleksii Moisieiev wrote:
> > > > On Thu, Jan 06, 2022 at 02:02:10PM +0000, Julien Grall wrote:
> > > > > On 06/01/2022 13:53, Oleksii Moisieiev wrote:
> > > > > > Hi Julien,
> > > > > >
> > > > > > On Mon, Jan 03, 2022 at 01:14:17PM +0000, Julien Grall wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On 24/12/2021 17:02, Oleksii Moisieiev wrote:
> > > > > > > > On Fri, Dec 24, 2021 at 03:42:42PM +0100, Julien Grall wrote:
> > > > > > > > > On 20/12/2021 16:41, Oleksii Moisieiev wrote:
> > > > > > > > > > > 2) What are the expected memory attribute for the regions?
> > > > > > > > > >
> > > > > > > > > > xen uses iommu_permit_access to pass agent page to the guest. So guest can access the page directly.
> > > > > > > > >
> > > > > > > > > I think you misunderstood my comment. Memory can be mapped with various type
> > > > > > > > > (e.g. Device, Memory) and attribute (cacheable, non-cacheable...). What will
> > > > > > > > > the firmware expect? What will the guest OS usually?
> > > > > > > > >
> > > > > > > > > The reason I am asking is the attributes have to matched to avoid any
> > > > > > > > > coherency issues. At the moment, if you use XEN_DOMCTL_memory_mapping, Xen
> > > > > > > > > will configure the stage-2 to use Device nGnRnE. As the result, the result
> > > > > > > > > memory access will be Device nGnRnE which is very strict.
> > > > > > > > > :w
> > > > > > > >
> > > > > > > > Let me share with you the configuration example:
> > > > > > > > scmi expects memory to be configured in the device-tree:
> > > > > > > >
> > > > > > > > cpu_scp_shm: scp-shmem@0xXXXXXXX {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0xXXXXXX 0x0 0x1000>;
> > > > > > > > };
> > > > > > > >
> > > > > > > > where XXXXXX address I allocate in alloc_magic_pages function.
> > > > > > >
> > > > > > > The goal of alloc_magic_pages() is to allocate RAM. However, what you want
> > > > > > > is a guest physical address and then map a part of the shared page.
> > > > > >
> > > > > > Do you mean that I can't use alloc_magic_pages to allocate shared
> > > > > > memory region for SCMI?
> > > > > Correct. alloc_magic_pages() will allocate a RAM page and then assign to the
> > > > > guest. From your description, this is not what you want because you will
> > > > > call XEN_DOMCTL_memory_mapping (and therefore replace the mapping).
> > > > >
> > > >
> > > > Ok thanks, I will refactor this part in v2.
> > > >
> > > > > >
> > > > > > >
> > > > > > > I can see two options here:
> > > > > > > 1) Hardcode the SCMI region in the memory map
> > > > > > > 2) Create a new region in the memory map that can be used for reserving
> > > > > > > memory for mapping.
> > > > > >
> > > > > > Could you please explain what do you mean under the "new region in the
> > > > > > memory map"?
> > > > >
> > > > > I mean reserving some guest physical address that could be used for map host
> > > > > physical address (e.g. SCMI region, GIC CPU interface...).
> > > > >
> > > > > So rather than hardcoding the address, we have something more flexible.
> > > > >
> > > >
> > > > Ok, I will fix that in v2.
> > >
> > > Just for avoidance of doubt. I was clarify option 2 and not requesting to
> > > implement. That said, if you want to implement option 2 I would be happy to
> > > review it.
> > >
> >
> > I'm thinking about the best way to reserve address for the domain.
> > We have xen_pfn_t shared_info_pfn in struct xc_dom_image which is not
> > used for Arm architecture. It can be set from shared_info_arm callback,
> > defined in xg_dom_arm.c.
> > I can use shared_info to store address of the SCMI and then use map_sci_page to
> > call XEN_DOMCTL_memory_mapping.
> >
> > This will allow me to reuse existing functionality and do not allocate
> > extra RAM.
> >
> > What do you think about that?
>
> I cannot speak for Julien but I think he meant something else (Julien
> please correct me if I am wrong.) Exposing addresses via device tree is
> not a problem.
>
> Normally we pick a fixed address for guest resources, for instance
> GUEST_GICD_BASE, see xen/include/public/arch-arm.h. We could do that for
> SCMI as well and it is basically approach 1).
>
> However, it is a bit inflexible and could cause issues with things like
> direct-map (https://urldefense.com/v3/__https://marc.info/?l=xen-devel&m=163997768108997__;!!GF_29dbcQIUBPA!kvNsu9pjqIwZ42N2q6aSQhTT_zA3OCEDkr7DwmAiuldEMwj2UiFReaPI8XlxsG-HOZ6v$ [marc[.]info]). A more
> flexible way would be for the SCMI guest address to be dynamically
> generated somehow.
>
> I am not sure how Julien envisioned the address to be generated exactly.
>
> Thanks to Oleksandr's work we have a way to find large regions of "free"
> address space. It is currently used for grant-table mappings. Maybe we
> could use a subset of it for SCMI? It might be best to wait for Julien's
> answer as he might have a better idea.
Thank you for the answer.
I think it will be best to reserve some space and generate address for
SMCI. Hope Julien will advise how it can be done.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs
2022-01-20 1:53 ` Stefano Stabellini
@ 2022-01-20 10:27 ` Oleksii Moisieiev
0 siblings, 0 replies; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-20 10:27 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Volodymyr Babchuk, Anthony PERARD, Oleksandr,
xen-devel, Wei Liu, Juergen Gross, Andrew Cooper, George Dunlap,
Jan Beulich, Bertrand Marquis
On Wed, Jan 19, 2022 at 05:53:55PM -0800, Stefano Stabellini wrote:
> On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > On Fri, Dec 24, 2021 at 02:30:50PM +0100, Julien Grall wrote:
> > > Hi,
> > >
> > > On 23/12/2021 20:06, Stefano Stabellini wrote:
> > > > On Wed, 22 Dec 2021, Stefano Stabellini wrote:
> > > > > # Future Ideas
> > > > >
> > > > > A great suggestion by Julien is to start supporting the dom0less partial
> > > > > device tree format in xl/libxl as well so that we can have a single
> > > > > "device_tree" option in dom.cfg instead of 4 (device_tree, iomem, irqs,
> > > > > dtdev).
> > > > >
> > > > > Even with that implemented, the user has still to provide a partial dtb,
> > > > > xen,reg and xen,path. I think this is a great step forward and we should
> > > > > do that, if nothing else to make it easier to switch between dom0less
> > > > > and normal domU configurations. But the number of options and
> > > > > information that the user has to provide is still similar to what we
> > > > > have today.
> > > >
> > > > I have just realized that if we start to parse the partial DTB in
> > > > xl/libxl the same way that we do for dom0less guests (parse "xen,path",
> > > > "xen,reg", and "interrupts", making dtdev, irqs and iomem optional)
> > > > actually we can achieve the goal below thanks to the combination:
> > > > "xen,path" + "xen,force-assign-without-iommu".
> > > >
> > > > In other words, with dom0less we already have a way to specify the link
> > > > to the host node even if the device is not a DMA master. We can do that
> > > > by specifying both xen,path and xen,force-assign-without-iommu for a
> > > > device.
> > > >
> > > > This is just FYI. I am not suggesting we should introduce dom0less-style
> > > > partial DTBs in order to get SCMI support in guests (although it would
> > > > be great to have). I think the best way forward for this series is one
> > > > of the combinations below, like a) + d), or a) + c)
> > >
> > > I strongly prefer a) + c) because a warning is easy to miss/ignore. At least
> > > with the extra property the user made an action to think about it and agree
> > > that this is the way do it.
> > >
> > > It is also easier to spot if we ask the user to provide the configuration
> > > file.
> > >
> >
> > Let me share my thoughts about c), which is:
> > c) require force-assign-without-iommu="true" in dom.cfg
> >
> > Adding this parameter to domain config means removing
> > xen,force-assign-without-iommu param from partial DTB.
>
> Why? No I don't think so.
>
>
> > This will affect dom0less configuration, which I can't test for now
> > without extra effort.
>
> We are just talking about adding:
>
> force-assign-without-iommu="true"
>
> to the xl config file. For instance:
>
> dtdev = [ "/amba/serial@ff000000" ]
> iomem = ["0xff000,1"]
> force-assign-without-iommu="true"
>
> It is unrelated to the dom0less partial DTB. There is no need to test
> dom0less to do this.
>
Oh. In this case I would be happy to add it in this patch series.
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-20 10:21 ` Oleksii Moisieiev
@ 2022-01-20 22:29 ` Stefano Stabellini
2022-01-21 15:07 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-20 22:29 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Thu, 20 Jan 2022, Oleksii Moisieiev wrote:
> On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> > On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > Hi Stefano,
> > > > > > >
> > > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > Hi Stefano,
> > > > > > > > >
> > > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > > >
> > > > > > > > > > > The following features are implemented:
> > > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > ---
> > > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > > >
> > > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > > +
> > > > > > > > > > > endmenu
> > > > > > > > > > >
> > > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > > + default n
> > > > > > > > > > > + depends on SCI
> > > > > > > > > > > + ---help---
> > > > > > > > > > > +
> > > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > > + for communication.
> > > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > > obj-y += sci.o
> > > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > > +/*
> > > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > + *
> > > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > > + *
> > > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > > + *
> > > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > > + *
> > > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > > + */
> > > > > > > > > > > +
> > > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > > +
> > > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > > +
> > > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > > +
> > > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > > +
> > > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > > >
> > > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > > >
> > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > > >
> > > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > > channels:
> > > > > > > > >
> > > > > > > > > reserved-memory {
> > > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > > no-map;
> > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > > };
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > > the current scmi channel:
> > > > > > > > >
> > > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > >
> > > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > > assigned to this agent.
> > > > > > > >
> > > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > > >
> > > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > > >
> > > > > > > > reserved-memory {
> > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > ...
> > > > > > > >
> > > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > > "linux,scmi_mem"?
> > > > > > > >
> > > > > > >
> > > > > > > That was my first implementation. But I've met a problem with
> > > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > > there were some if, checking if memory weren't reserved.
> > > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > > shmem page.
> > > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > > which has no compatible string and provides no-map property.
> > > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > > space for the domain.
> > > > > > >
> > > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > > and scmi_devid property to the
> > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > > Those node and property are needed only for Xen and useless for
> > > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > > What do you think about it?
> > > > > >
> > > > > > Reply below
> > > > > >
> > > > > > [...]
> > > > > >
> > > > > >
> > > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > > >
> > > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > > >
> > > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > > work on the Xen code that makes use of it.
> > > > > > > >
> > > > > > > > Does it make sense?
> > > > > > > >
> > > > > > >
> > > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > > >
> > > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > > properties into generic bindings. It is a problem both from a
> > > > > > specification perspective (because it has hard to handle Xen specific
> > > > > > cases in fully generic bindings, especially as those bindings are
> > > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > > >
> > > > > >
> > > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > > driver too. Why the Linux driver doesn't need it?
> > > > > >
> > > > >
> > > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > > On non-virtualized systems - there is no need of this call, because OS
> > > > > is the only one entity, running on the system.
> > > >
> > > > OK. Even if it is only required for virtualized systems, I think that
> > > > scmi_devid is important enough that should be part of the upstream
> > > > binding. I think it is worth starting an email thread on the LKML with
> > > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > > scmi_devid to the binding.
> > > >
> > > >
> > > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > > list of device_ids to dom.cfg, such as:
> > > > > sci_devs = [ 0, 1, 15, 35 ];
> > > > >
> > > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > > We can probably make hypercall taking devid list as input parameter.
> > > > > This will take only 1 hypercall to setup sci permissions.
> > > >
> > > > But how would a user know which are the right SCMI IDs to add to the
> > > > sci_devs list? Would the user have to go and read the reference manual
> > > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > > If that is the case, then I think that it would be better to add
> > > > scmi_devid to device tree.
> > > >
> > > > In general, I think this configuration should happen automatically
> > > > without user intervention. The user should just specify "enable SCMI"
> > > > and it should work.
> > > >
> > > >
> > > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > > way, I don't think we should need linux,scmi_mem.
> > > > >
> > > > > This requires further investigation. I will try to make implementation
> > > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > > reuslts with you.
> > > >
> > > > OK, thanks.
> > >
> > > Hi Stefano,
> > >
> > > As I did some investigation about using reserved-memory area
> > > linux,scmi_mem and now I need your advice.
> > >
> > > I see 2 possible implementations for now:
> > > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > > reserved memory region.
> > > So device-tree will look like this:
> > >
> > > reserved-memory {
> > > /* reserved region for scmi channels*/
> > > scmi_memory: region@53FF0000{
> > > no-map;
> > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > };
> > > };
> > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > memory-region = <&scmi_memory>;
> > > };
> > >
> > > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > > that xen can find reserved memory region without adding additional names
> > > to the device-tree bindings.
> > > memory-region parameter as a reference to reserved memory and region
> > > creation described in:
> > > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> > >
> > > This approach I've implemented already and it works.
> >
> > This approach would require a discussion with the upstream device tree
> > maintainers. Likely, we would need to add a note about the usage of the
> > "memory-region" property to arm,scmi.yaml.
> >
> > Also, I have the feeling that they would ask to add the "memory-region"
> > property directly to the "arm,scmi-smc" node, as an alternative (or
> > in addition) to the existing "shmem" property.
> >
> > That said, from my point of view this approach is also a viable option.
> > I don't see any major problems.
> >
> > The main question (after reading everything else that you wrote below)
> > is whether the "arm,scmi-smc" node in this case could be automatically
> > generated.
> >
>
> arm,scmi-smc node can be generated in both cases. I think I'd leave it
> as backup in case if the second approach will not work.
>
> >
> > > 2) The second approach is the format you suggested:
> > > > > > > > reserved-memory {
> > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > };
> > > > > > > > ...
> > >
> > > This approach has an advantage that xen ARM_SCI driver do not know about
> > > how channels are placed in the reserved memory, but introduces some
> > > disadvantages:
> > > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > > in the device-tree. In current implementation I have separate scmi.dtsi
> > > file which introduces scmi support for both XEN-based and
> > > non-virtualized systems. Having 14 extra channels in the device-tree may
> > > be confusing.
> >
> > I can see that while it would be ideal for Xen to see all 14+1 channels
> > in device tree (on the host device tree), we wouldn't want to expose all
> > of them to the domains, not even to dom0.
> >
> > How many channels do we want dom0 to see by the way? For this
> > discussion, I'll just assume for now that dom0 only sees 1 channel like
> > the domUs.
>
> For dom0 we need only one channel.
>
> >
> > Now we have a problem: how do we go about "filtering" the
> > "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> > below in point b).
> >
>
> Xen will not need to filter "arm,scmi-shmem" node. It will just
> create shmem node in Domain device-tree. I don't see any problem for xen
> configuration here.
> What bothers me here is that I set scmi configuration for platform dts,
> not for xen or domu dts files.
> So for example I have the following structure of the dts files for my
> platform (r8a77961-salvator-xs):
> * r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
> scmi_devid for the devices, that should use scmi.
> * r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
> It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
> is used for system with no hypervisor.
> * r8a77961-salvator-xs-xen.dts - dts file for xen which includes
> r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> * r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
> r8a77961-salvator-xs.dts and inherits scmi configuration from it.
>
> In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
> r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
>
> I can give you a link to Merge request with this changes if you need it.
>
> For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
> nodes will be omitted and new will be generated for the domains.
>
> What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
>
> Just got an idea while writing this: I can create only one
> "arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
> needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
>
> Then we will have valid configurations for all cases.
> This can be a solution. What do you think?
It is good that you brought this up because it helps me explain what I
mean. And of course it is up to you where you place the nodes in the
various dts files at your disposal. Either way it would work but I think
they should belong to r8a77961-salvator-xs.dts.
Generally the platform vendor (e.g. Xilinx) provides a device tree
description of the platform to use including all the available resources
and firmware interfaces. In your case it would be r8a77961-scmi.dtsi +
r8a77961-salvator-xs.dts. This is what I call the "host device tree"
below. Users should be able to boot a fully functional system using the
host device tree pretty much "as is" to run Xen, Linux or any other
software.
Certainly the SCMI device tree description should be part of the host
device tree, so in your case it would be r8a77961-salvator-xs.dts. And
the description should include all 14+1 channels because this is the
generic platform description -- we cannot know for sure how the users
are going to use the system.
This is why r8a77961-salvator-xs-xen.dts should be as small as possible
or ideally inexistent. There shouldn't be a need for a special device
tree modification to allow Xen to run. In reality, even at Xilinx we
have something like r8a77961-salvator-xs-xen.dts, although it is really
small.
But I see that r8a77961-salvator-xs-xen.dts could be viewed as the
device tree additions to run hypervisors and from that point of view it
is more acceptable to place the 14 channels there.
The biggest problem is r8a77961-salvator-xs-domu.dts: who is going to
write it? And how? It wouldn't be provided by the platform vendor, so it
is the user the one that has to find a way to write it.
I know the user already has to write a partial DTB for device
assignment, but any time the process is more complex than "copy the host
device tree node for device XXX to the partial DTB" it is a problem.
Errors are made and the system doesn't work.
I think we don't want to make it even more difficult by having to
manually produce the SCMI domU description too. The SCMI description for
domU could be automatically generated by Xen, or libxl/xl. If that's an
issue, then the SCMI description could be automatically generated by an
external tool but I think it would make things more complex and harder
to maintain.
In short my point of view is:
- r8a77961-scmi.dtsi + r8a77961-salvator-xs.dts should be as generic as
possible so the SCMI nodes should have 14+1 channels
- but putting the 14 channels in r8a77961-salvator-xs-xen.dts is still
OKish
- it is important that r8a77961-salvator-xs-domu.dts is automatically
generated by Xen or libxl or another software tool
> > > b) In case if we have all 15 channels, described in partial device-tree,
> >
> > I think you meant "described in the host device tree", right?
> >
> Yeah that's what I've meant.
> >
> > > we should not copy any node to the domain device-tree. I think it will
> > > be better to generate arm,scmi-shmem node in the Domain device-tree.
> >
> > Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> > device tree description for the DomU/Dom0 based on the channels
> > allocated to the domain.
> >
> >
> > > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > > node can't be generated. I prefer it to be copied from the partial
> > > device-tree because it includes some platform specific configuration,
> > > such as func-id and list of the protocols (for example different
> > > platforms may require different list of the protocols). So in this
> > > case we will have 1 node copied and 1 node generated.
> > >
> > > I think even for dom0less we should use arm,scmi-smc node from the
> > > device-tree because protocol configuration and funcid is related to the
> > > platform.
> >
> > I am not sure I understood what you wrote. You are saying that the
> > "arm,scmi-smc" node includes some platform specific configurations so
> > it cannot be automatically generated by Xen (or by the tools) and
> > instead it needs to be manually provided as part of the partial dtb for
> > the domU. Is that correct?
> >
> > If so, I would like to understand the reasons behind it. Manual
> > device tree editing is problematic.
> >
> > I looked for "func-id" in
> > Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> > find any results. Do you have an example of the platform specific
> > configuration or protocol configuration that would make it difficult to
> > automatically generate the "arm,scmi-smc" node for the domains?
>
> Sorry, I used wrong term (used term from the specification), arm,smc-id
> of cause.
>
> >
> > Also, is this a problem just for approach #2 or also for approach #1?
> > If it is a problem only for approach #2, then let's just go with
> > approach #1.
> >
>
> We can't copy "arm,scmi-smc" in both approaches. The difference is that
> in the first approach we can copy both "arm,scmi-smc" and
> "arm,scmi-shmem" nodes while in the second approach we should copy
> "arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
>
> arm,scmi-smc node can't be generated because it includes properties and
> configurations that depends from platform and should be get from the
> device tree.
> Here is "arm,scmi-smc" node expample:
> firmware {
> scmi {
> compatible = "arm,scmi-smc"
> arm,smc-id = <0x82000002>;
> shmem = <&cpu_scp_shm>;
> #address-cells = <1>;
> #size-cells = <0>;
> scmi_power: protocol@11 {
> reg = <0x11>;
> #power-domain-cells = <1>;
> };
>
> scmi_clock: protocol@14 {
> ...
> };
>
> scmi_reset: protocol@16 {
> ...
> };
> ...
> };
> };
>
> It has 3 configurable options:
> * arm,smc-id parameter, setting func_id for scmi protocol. This id can be
> different for different platforms.
> For example stm32mp1 architecture use different scm-id for different
> agents:
> https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h#L39
>
> * shmem which includes phandle to arm,scmi-shmem node. But this is not
> a problem and can be updated.
>
> * list of the protocol subnodes. This is also configurable parameter,
> not regs or names, but the number of the protocols. For example onle
> platform can use power-domains/clock/resets via scmi, when another will
> require volage-control and sensor-management to be added.
>
> Xen should know this parameters to be able to generate "arm,scmi-smc" node.
>
> Also we're currently discussing new scmi protocol with ARM: Pinctrl over
> SCMI.
>
> It should allow domains to access pinctrl subsystem, placed in Firmware
> through SCMI protocol.
> scmi_pinctrl node will look like this:
>
> firmware {
> scmi {
> ...
> scmi_pinctrl: protocol@18 {
> reg = <0x18>;
> #pinctrl-cells = <0>;
>
> i2c2_pins: i2c2 {
> groups = <74>; /* i2c2_a */
> function = <15>; /* i2c2 */
> };
>
> irq0_pins: irq0 {
> groups = <81>; /* intc_ex_irq0 */
> function = <19>; /* intc_ex */
> };
>
> avb_pins: avb {
> mux {
> /* avb_link, avb_mdio, avb_mii */
> groups = <17>, <21>, <22>;
> function = <1>; /* avb */
> };
>
> pins_mdio {
> groups = <21>; /* avb_mdio */
> drive-strength = <24>;
> };
>
> pins_mii_tx {
> /* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
> PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
> pins = <242>, <240>, <236>, <237>, <238>, <239>;
> drive-strength = <12>;
> };
> };
> ...
> };
> };
> };
>
> So "arm,scmi-smc" node will have even more platform specific settings.
>
> >
> > > I prefer the second approach and will try to make it if it's OK to copy
> > > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > > node.
> > >
> > > What do you think about that?
> >
> > From a device tree specification perspective, I think both approaches
> > are OK (with a minor comment on the first approach as I wrote above.)
> >
> > But from a Xen perspective I think it is important that we don't require
> > the user to manually provide the SCMI configuration in the partial DTB.
> > It would be better if we could generate it automatically from Xen or the
> > tools (or even an independent script). Or copy the "arm,scmi-smc" node
> > from the host device tree to the domU device tree without modifications.
>
> I think copy "arm,scmi-smc" node is the only option we have.
> I'm not sure what do you mean under "host device tree" if you mean Xen
> device-tree - then I think it will not cover the case with stm32mp1 I've
> mentioned above. I think it will be better to copy "arm,scmi-smc" node
> from Domu partial Device-tree to Domu device-tree.
> So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
> AGENT1 scm-is set in domu device-tree and copied to dom-u.
>
> Do you agree with my points?
I think we are saying similar things, but we are getting mixed up with
the terminology. Let's start from the basics :-)
# Host device tree
The device tree given to Xen at boot time. This is the device tree that
Xen parses to discover what's available on the platform. In your case,
it seems to include r8a77961-salvator-xs-xen.dts.
# Partial DTB
(Ignoring Dom0less) this is the small DTB that gets passed to xl with
the "device_tree" option in the xl config file. It is copied verbatim
to the domU device tree by xl/libxl.
# Copy the "arm,scmi-smc" node from host device tree
This means that the domU "arm,scmi-smc" node is an exact copy of the
host device tree SCMI node. I don't think this is actually possible in
most cases because the domU description is typically a bit different
from the host description. For instance, the host description could
include 14+1 channels while the domU description should only include 1
channel.
# Copy the "arm,scmi-smc" node from the partial DTB
This implies that somebody or something create an "arm,scmi-smc" node
for the domU and placed it into the partial DTB. Then, Xen and/or
xl/libxl will copy the node from the partial DTB to the DomU device
tree. The main question in this case is: who is going to write the
partial DTB? We dont want the user (i.e. a person) to have to manually
write the SCMI description for the domU. It should be an automated tools
that does it. At that point, it is easier if it is Xen or xl/libxl.
Alternativaly, we could think of an external tool but I think it would
make things more difficult to maintain.
# Generate the "arm,scmi-smc" node for domUs
When I write "generate the arm,scmi-smc node", I mean that Xen and
libxl/xl will generate the "arm,scmi-smc" node for the domU. Thus, the
node will not be copied from the partial DTB or from the device tree,
instead, it should be created directly by Xen and/or libxl/xl.
However, the domU "arm,scmi-smc" node could still be derived from the
host device tree "arm,scmi-smc" node. In other words, Xen or xl/libxl
would look at the host device tree "arm,scmi-smc" node, copy it to the
domU device tree while making as many changes as necessary.
The DomU "arm,scmi-smc" node doesn't have to be entirely fake and
static. It could be dynamically created to match the host device tree
description. I think this is the best option.
# Conclusion
I am suggesting that Xen and/or libxl automatically produce the
"arm,scmi-smc" node for domUs based on the host device tree description
and based on the channel mapped to the domU. This way, the user (a
person) doesn't have to go and manually edit the domU partial DTB.
> > So if using approach #1 allows us to automatically generate the
> > "arm,scmi-smc" node for the guest, then I think it's best for sure.
> >
>
> Summarizing all written above I would focus on the second approach
> and put aside the first approach implementation. If you don't mind.
Sure, that's fine by me
> > > Also I wanted to mention that I'm not planning to make ARM_SCI support for
> > > dom0less in terms of this patch series bacause I can't test
> > > dom0less configuration for now. So let me know if some of my
> > > functionality breaks dom0less.
> >
> > That's fine. I don't mean to scope-creep your patch series, which is
> > extremely valuable as is.
> >
> > That said, I would be happy to provide you with a very simple dom0less
> > configuration for your platform to enable you to test, or alternatively
> > I could write a patch to add dom0less domU support if you are happy to
> > help reviewing and testing it.
>
> I was thinking about making dom0less support in the different
> patch-series because there are still questions to be discussed.
>
> For example, how arm,scmi-smc node will be generated for DomUs and how
> the case, when scmi configuration is different for DomU1 and DomU2 (as
> in case of stm32mp1 when smc-id is different) should be handled.
>
> What do you think about continue without dom0less support and discuss
> dom0less once we done with the main part?
That's OK, especially if you are happy to work on dom0less support
later.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-20 22:29 ` Stefano Stabellini
@ 2022-01-21 15:07 ` Oleksii Moisieiev
2022-01-21 20:49 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-21 15:07 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
Hi Stefano,
On Thu, Jan 20, 2022 at 02:29:41PM -0800, Stefano Stabellini wrote:
> On Thu, 20 Jan 2022, Oleksii Moisieiev wrote:
> > On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> > > On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > > > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > Hi Stefano,
> > > > > > > >
> > > > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > Hi Stefano,
> > > > > > > > > >
> > > > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > > > >
> > > > > > > > > > > > The following features are implemented:
> > > > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > > > >
> > > > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > ---
> > > > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > > > >
> > > > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > > > +
> > > > > > > > > > > > endmenu
> > > > > > > > > > > >
> > > > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > > > + default n
> > > > > > > > > > > > + depends on SCI
> > > > > > > > > > > > + ---help---
> > > > > > > > > > > > +
> > > > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > > > + for communication.
> > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > > > obj-y += sci.o
> > > > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > > > +/*
> > > > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > > > + *
> > > > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > > > + */
> > > > > > > > > > > > +
> > > > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > > > +
> > > > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > > > +
> > > > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > > > >
> > > > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > > > >
> > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > > > >
> > > > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > > > channels:
> > > > > > > > > >
> > > > > > > > > > reserved-memory {
> > > > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > > > no-map;
> > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > > > };
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > > > the current scmi channel:
> > > > > > > > > >
> > > > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > >
> > > > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > > > assigned to this agent.
> > > > > > > > >
> > > > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > > > >
> > > > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > > > >
> > > > > > > > > reserved-memory {
> > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > > > "linux,scmi_mem"?
> > > > > > > > >
> > > > > > > >
> > > > > > > > That was my first implementation. But I've met a problem with
> > > > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > > > there were some if, checking if memory weren't reserved.
> > > > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > > > shmem page.
> > > > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > > > which has no compatible string and provides no-map property.
> > > > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > > > space for the domain.
> > > > > > > >
> > > > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > > > and scmi_devid property to the
> > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > > > Those node and property are needed only for Xen and useless for
> > > > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > > > What do you think about it?
> > > > > > >
> > > > > > > Reply below
> > > > > > >
> > > > > > > [...]
> > > > > > >
> > > > > > >
> > > > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > > > >
> > > > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > > > >
> > > > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > > > work on the Xen code that makes use of it.
> > > > > > > > >
> > > > > > > > > Does it make sense?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > > > >
> > > > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > > > properties into generic bindings. It is a problem both from a
> > > > > > > specification perspective (because it has hard to handle Xen specific
> > > > > > > cases in fully generic bindings, especially as those bindings are
> > > > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > > > >
> > > > > > >
> > > > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > > > driver too. Why the Linux driver doesn't need it?
> > > > > > >
> > > > > >
> > > > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > > > On non-virtualized systems - there is no need of this call, because OS
> > > > > > is the only one entity, running on the system.
> > > > >
> > > > > OK. Even if it is only required for virtualized systems, I think that
> > > > > scmi_devid is important enough that should be part of the upstream
> > > > > binding. I think it is worth starting an email thread on the LKML with
> > > > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > > > scmi_devid to the binding.
> > > > >
> > > > >
> > > > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > > > list of device_ids to dom.cfg, such as:
> > > > > > sci_devs = [ 0, 1, 15, 35 ];
> > > > > >
> > > > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > > > We can probably make hypercall taking devid list as input parameter.
> > > > > > This will take only 1 hypercall to setup sci permissions.
> > > > >
> > > > > But how would a user know which are the right SCMI IDs to add to the
> > > > > sci_devs list? Would the user have to go and read the reference manual
> > > > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > > > If that is the case, then I think that it would be better to add
> > > > > scmi_devid to device tree.
> > > > >
> > > > > In general, I think this configuration should happen automatically
> > > > > without user intervention. The user should just specify "enable SCMI"
> > > > > and it should work.
> > > > >
> > > > >
> > > > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > > > way, I don't think we should need linux,scmi_mem.
> > > > > >
> > > > > > This requires further investigation. I will try to make implementation
> > > > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > > > reuslts with you.
> > > > >
> > > > > OK, thanks.
> > > >
> > > > Hi Stefano,
> > > >
> > > > As I did some investigation about using reserved-memory area
> > > > linux,scmi_mem and now I need your advice.
> > > >
> > > > I see 2 possible implementations for now:
> > > > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > > > reserved memory region.
> > > > So device-tree will look like this:
> > > >
> > > > reserved-memory {
> > > > /* reserved region for scmi channels*/
> > > > scmi_memory: region@53FF0000{
> > > > no-map;
> > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > };
> > > > };
> > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > compatible = "arm,scmi-shmem";
> > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > memory-region = <&scmi_memory>;
> > > > };
> > > >
> > > > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > > > that xen can find reserved memory region without adding additional names
> > > > to the device-tree bindings.
> > > > memory-region parameter as a reference to reserved memory and region
> > > > creation described in:
> > > > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> > > >
> > > > This approach I've implemented already and it works.
> > >
> > > This approach would require a discussion with the upstream device tree
> > > maintainers. Likely, we would need to add a note about the usage of the
> > > "memory-region" property to arm,scmi.yaml.
> > >
> > > Also, I have the feeling that they would ask to add the "memory-region"
> > > property directly to the "arm,scmi-smc" node, as an alternative (or
> > > in addition) to the existing "shmem" property.
> > >
> > > That said, from my point of view this approach is also a viable option.
> > > I don't see any major problems.
> > >
> > > The main question (after reading everything else that you wrote below)
> > > is whether the "arm,scmi-smc" node in this case could be automatically
> > > generated.
> > >
> >
> > arm,scmi-smc node can be generated in both cases. I think I'd leave it
> > as backup in case if the second approach will not work.
> >
> > >
> > > > 2) The second approach is the format you suggested:
> > > > > > > > > reserved-memory {
> > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > };
> > > > > > > > > ...
> > > >
> > > > This approach has an advantage that xen ARM_SCI driver do not know about
> > > > how channels are placed in the reserved memory, but introduces some
> > > > disadvantages:
> > > > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > > > in the device-tree. In current implementation I have separate scmi.dtsi
> > > > file which introduces scmi support for both XEN-based and
> > > > non-virtualized systems. Having 14 extra channels in the device-tree may
> > > > be confusing.
> > >
> > > I can see that while it would be ideal for Xen to see all 14+1 channels
> > > in device tree (on the host device tree), we wouldn't want to expose all
> > > of them to the domains, not even to dom0.
> > >
> > > How many channels do we want dom0 to see by the way? For this
> > > discussion, I'll just assume for now that dom0 only sees 1 channel like
> > > the domUs.
> >
> > For dom0 we need only one channel.
> >
> > >
> > > Now we have a problem: how do we go about "filtering" the
> > > "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> > > below in point b).
> > >
> >
> > Xen will not need to filter "arm,scmi-shmem" node. It will just
> > create shmem node in Domain device-tree. I don't see any problem for xen
> > configuration here.
> > What bothers me here is that I set scmi configuration for platform dts,
> > not for xen or domu dts files.
> > So for example I have the following structure of the dts files for my
> > platform (r8a77961-salvator-xs):
> > * r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
> > scmi_devid for the devices, that should use scmi.
> > * r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
> > It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
> > is used for system with no hypervisor.
> > * r8a77961-salvator-xs-xen.dts - dts file for xen which includes
> > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > * r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
> > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> >
> > In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
> > r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
> >
> > I can give you a link to Merge request with this changes if you need it.
> >
> > For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
> > nodes will be omitted and new will be generated for the domains.
> >
> > What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
> >
> > Just got an idea while writing this: I can create only one
> > "arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
> > needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
> >
> > Then we will have valid configurations for all cases.
> > This can be a solution. What do you think?
>
> It is good that you brought this up because it helps me explain what I
> mean. And of course it is up to you where you place the nodes in the
> various dts files at your disposal. Either way it would work but I think
> they should belong to r8a77961-salvator-xs.dts.
>
> Generally the platform vendor (e.g. Xilinx) provides a device tree
> description of the platform to use including all the available resources
> and firmware interfaces. In your case it would be r8a77961-scmi.dtsi +
> r8a77961-salvator-xs.dts. This is what I call the "host device tree"
> below. Users should be able to boot a fully functional system using the
> host device tree pretty much "as is" to run Xen, Linux or any other
> software.
>
> Certainly the SCMI device tree description should be part of the host
> device tree, so in your case it would be r8a77961-salvator-xs.dts. And
> the description should include all 14+1 channels because this is the
> generic platform description -- we cannot know for sure how the users
> are going to use the system.
>
> This is why r8a77961-salvator-xs-xen.dts should be as small as possible
> or ideally inexistent. There shouldn't be a need for a special device
> tree modification to allow Xen to run. In reality, even at Xilinx we
> have something like r8a77961-salvator-xs-xen.dts, although it is really
> small.
>
> But I see that r8a77961-salvator-xs-xen.dts could be viewed as the
> device tree additions to run hypervisors and from that point of view it
> is more acceptable to place the 14 channels there.
>
> The biggest problem is r8a77961-salvator-xs-domu.dts: who is going to
> write it? And how? It wouldn't be provided by the platform vendor, so it
> is the user the one that has to find a way to write it.
>
> I know the user already has to write a partial DTB for device
> assignment, but any time the process is more complex than "copy the host
> device tree node for device XXX to the partial DTB" it is a problem.
> Errors are made and the system doesn't work.
>
> I think we don't want to make it even more difficult by having to
> manually produce the SCMI domU description too. The SCMI description for
> domU could be automatically generated by Xen, or libxl/xl. If that's an
> issue, then the SCMI description could be automatically generated by an
> external tool but I think it would make things more complex and harder
> to maintain.
>
> In short my point of view is:
> - r8a77961-scmi.dtsi + r8a77961-salvator-xs.dts should be as generic as
> possible so the SCMI nodes should have 14+1 channels
> - but putting the 14 channels in r8a77961-salvator-xs-xen.dts is still
> OKish
> - it is important that r8a77961-salvator-xs-domu.dts is automatically
> generated by Xen or libxl or another software tool
>
Thank you for the detailed response. I'll put all 14+1 channels to
r8a77961-salvator-xs.dts then.
I've described my thoughts about generation of the arm,scmi-smc node below.
>
> > > > b) In case if we have all 15 channels, described in partial device-tree,
> > >
> > > I think you meant "described in the host device tree", right?
> > >
> > Yeah that's what I've meant.
> > >
> > > > we should not copy any node to the domain device-tree. I think it will
> > > > be better to generate arm,scmi-shmem node in the Domain device-tree.
> > >
> > > Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> > > device tree description for the DomU/Dom0 based on the channels
> > > allocated to the domain.
> > >
> > >
> > > > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > > > node can't be generated. I prefer it to be copied from the partial
> > > > device-tree because it includes some platform specific configuration,
> > > > such as func-id and list of the protocols (for example different
> > > > platforms may require different list of the protocols). So in this
> > > > case we will have 1 node copied and 1 node generated.
> > > >
> > > > I think even for dom0less we should use arm,scmi-smc node from the
> > > > device-tree because protocol configuration and funcid is related to the
> > > > platform.
> > >
> > > I am not sure I understood what you wrote. You are saying that the
> > > "arm,scmi-smc" node includes some platform specific configurations so
> > > it cannot be automatically generated by Xen (or by the tools) and
> > > instead it needs to be manually provided as part of the partial dtb for
> > > the domU. Is that correct?
> > >
> > > If so, I would like to understand the reasons behind it. Manual
> > > device tree editing is problematic.
> > >
> > > I looked for "func-id" in
> > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> > > find any results. Do you have an example of the platform specific
> > > configuration or protocol configuration that would make it difficult to
> > > automatically generate the "arm,scmi-smc" node for the domains?
> >
> > Sorry, I used wrong term (used term from the specification), arm,smc-id
> > of cause.
> >
> > >
> > > Also, is this a problem just for approach #2 or also for approach #1?
> > > If it is a problem only for approach #2, then let's just go with
> > > approach #1.
> > >
> >
> > We can't copy "arm,scmi-smc" in both approaches. The difference is that
> > in the first approach we can copy both "arm,scmi-smc" and
> > "arm,scmi-shmem" nodes while in the second approach we should copy
> > "arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
> >
> > arm,scmi-smc node can't be generated because it includes properties and
> > configurations that depends from platform and should be get from the
> > device tree.
> > Here is "arm,scmi-smc" node expample:
> > firmware {
> > scmi {
> > compatible = "arm,scmi-smc"
> > arm,smc-id = <0x82000002>;
> > shmem = <&cpu_scp_shm>;
> > #address-cells = <1>;
> > #size-cells = <0>;
> > scmi_power: protocol@11 {
> > reg = <0x11>;
> > #power-domain-cells = <1>;
> > };
> >
> > scmi_clock: protocol@14 {
> > ...
> > };
> >
> > scmi_reset: protocol@16 {
> > ...
> > };
> > ...
> > };
> > };
> >
> > It has 3 configurable options:
> > * arm,smc-id parameter, setting func_id for scmi protocol. This id can be
> > different for different platforms.
> > For example stm32mp1 architecture use different scm-id for different
> > agents:
> > https://urldefense.com/v3/__https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h*L39__;Iw!!GF_29dbcQIUBPA!mTRUjtSg19iVaYo3Cgjop5ckPWqKsHVo1EZCEA1zCbod9KpNSXX291A8vAuNdTCr46MA$ [github[.]com]
> >
> > * shmem which includes phandle to arm,scmi-shmem node. But this is not
> > a problem and can be updated.
> >
> > * list of the protocol subnodes. This is also configurable parameter,
> > not regs or names, but the number of the protocols. For example onle
> > platform can use power-domains/clock/resets via scmi, when another will
> > require volage-control and sensor-management to be added.
> >
> > Xen should know this parameters to be able to generate "arm,scmi-smc" node.
> >
> > Also we're currently discussing new scmi protocol with ARM: Pinctrl over
> > SCMI.
> >
> > It should allow domains to access pinctrl subsystem, placed in Firmware
> > through SCMI protocol.
> > scmi_pinctrl node will look like this:
> >
> > firmware {
> > scmi {
> > ...
> > scmi_pinctrl: protocol@18 {
> > reg = <0x18>;
> > #pinctrl-cells = <0>;
> >
> > i2c2_pins: i2c2 {
> > groups = <74>; /* i2c2_a */
> > function = <15>; /* i2c2 */
> > };
> >
> > irq0_pins: irq0 {
> > groups = <81>; /* intc_ex_irq0 */
> > function = <19>; /* intc_ex */
> > };
> >
> > avb_pins: avb {
> > mux {
> > /* avb_link, avb_mdio, avb_mii */
> > groups = <17>, <21>, <22>;
> > function = <1>; /* avb */
> > };
> >
> > pins_mdio {
> > groups = <21>; /* avb_mdio */
> > drive-strength = <24>;
> > };
> >
> > pins_mii_tx {
> > /* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
> > PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
> > pins = <242>, <240>, <236>, <237>, <238>, <239>;
> > drive-strength = <12>;
> > };
> > };
> > ...
> > };
> > };
> > };
> >
> > So "arm,scmi-smc" node will have even more platform specific settings.
> >
> > >
> > > > I prefer the second approach and will try to make it if it's OK to copy
> > > > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > > > node.
> > > >
> > > > What do you think about that?
> > >
> > > From a device tree specification perspective, I think both approaches
> > > are OK (with a minor comment on the first approach as I wrote above.)
> > >
> > > But from a Xen perspective I think it is important that we don't require
> > > the user to manually provide the SCMI configuration in the partial DTB.
> > > It would be better if we could generate it automatically from Xen or the
> > > tools (or even an independent script). Or copy the "arm,scmi-smc" node
> > > from the host device tree to the domU device tree without modifications.
> >
> > I think copy "arm,scmi-smc" node is the only option we have.
> > I'm not sure what do you mean under "host device tree" if you mean Xen
> > device-tree - then I think it will not cover the case with stm32mp1 I've
> > mentioned above. I think it will be better to copy "arm,scmi-smc" node
> > from Domu partial Device-tree to Domu device-tree.
> > So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
> > AGENT1 scm-is set in domu device-tree and copied to dom-u.
> >
> > Do you agree with my points?
>
> I think we are saying similar things, but we are getting mixed up with
> the terminology. Let's start from the basics :-)
>
> # Host device tree
> The device tree given to Xen at boot time. This is the device tree that
> Xen parses to discover what's available on the platform. In your case,
> it seems to include r8a77961-salvator-xs-xen.dts.
>
> # Partial DTB
> (Ignoring Dom0less) this is the small DTB that gets passed to xl with
> the "device_tree" option in the xl config file. It is copied verbatim
> to the domU device tree by xl/libxl.
>
> # Copy the "arm,scmi-smc" node from host device tree
> This means that the domU "arm,scmi-smc" node is an exact copy of the
> host device tree SCMI node. I don't think this is actually possible in
> most cases because the domU description is typically a bit different
> from the host description. For instance, the host description could
> include 14+1 channels while the domU description should only include 1
> channel.
>
> # Copy the "arm,scmi-smc" node from the partial DTB
> This implies that somebody or something create an "arm,scmi-smc" node
> for the domU and placed it into the partial DTB. Then, Xen and/or
> xl/libxl will copy the node from the partial DTB to the DomU device
> tree. The main question in this case is: who is going to write the
> partial DTB? We dont want the user (i.e. a person) to have to manually
> write the SCMI description for the domU. It should be an automated tools
> that does it. At that point, it is easier if it is Xen or xl/libxl.
> Alternativaly, we could think of an external tool but I think it would
> make things more difficult to maintain.
>
> # Generate the "arm,scmi-smc" node for domUs
> When I write "generate the arm,scmi-smc node", I mean that Xen and
> libxl/xl will generate the "arm,scmi-smc" node for the domU. Thus, the
> node will not be copied from the partial DTB or from the device tree,
> instead, it should be created directly by Xen and/or libxl/xl.
>
> However, the domU "arm,scmi-smc" node could still be derived from the
> host device tree "arm,scmi-smc" node. In other words, Xen or xl/libxl
> would look at the host device tree "arm,scmi-smc" node, copy it to the
> domU device tree while making as many changes as necessary.
>
> The DomU "arm,scmi-smc" node doesn't have to be entirely fake and
> static. It could be dynamically created to match the host device tree
> description. I think this is the best option.
>
>
> # Conclusion
> I am suggesting that Xen and/or libxl automatically produce the
> "arm,scmi-smc" node for domUs based on the host device tree description
> and based on the channel mapped to the domU. This way, the user (a
> person) doesn't have to go and manually edit the domU partial DTB.
>
That sounds reasonable. The problem is that arm,scmi-smc node can be
copmlicated and include a lot of configuration. Also for different
mediators this node can be different.
As I inderstand, there is no mechanism for xl to access host device-tree
right now. Correct me if I'm wrong.
I see the following way we can generate arm,scmi-smc node for DomU:
We say that if scmi-smc mediator is enabled - then Dom0 is configured to
use SCMI. This means that Dom0 device-tree will have arm,scmi-smc node
and it can be reached from the userspace.
In this case xl can use infromation from /proc/device-tree/firmware/scmi
to generate arm,scmi-smc node for DomU. But in this case xl should know
the exact path of scmi node.
Or we can generate some special node, called "shared" in Dom0 device-tree
which will include copy of the arm,scmi-smc node, which can be used for domains.
In this case xl can scan /proc/device-tree/shared node and find
arm,scmi-smc copatible node and use it to generate arm,scmi-smc node for
DomU.
Also this can be used for another features in future.
What do you think about this?
>
> > > So if using approach #1 allows us to automatically generate the
> > > "arm,scmi-smc" node for the guest, then I think it's best for sure.
> > >
> >
> > Summarizing all written above I would focus on the second approach
> > and put aside the first approach implementation. If you don't mind.
>
> Sure, that's fine by me
>
>
> > > > Also I wanted to mention that I'm not planning to make ARM_SCI support for
> > > > dom0less in terms of this patch series bacause I can't test
> > > > dom0less configuration for now. So let me know if some of my
> > > > functionality breaks dom0less.
> > >
> > > That's fine. I don't mean to scope-creep your patch series, which is
> > > extremely valuable as is.
> > >
> > > That said, I would be happy to provide you with a very simple dom0less
> > > configuration for your platform to enable you to test, or alternatively
> > > I could write a patch to add dom0less domU support if you are happy to
> > > help reviewing and testing it.
> >
> > I was thinking about making dom0less support in the different
> > patch-series because there are still questions to be discussed.
> >
> > For example, how arm,scmi-smc node will be generated for DomUs and how
> > the case, when scmi configuration is different for DomU1 and DomU2 (as
> > in case of stm32mp1 when smc-id is different) should be handled.
> >
> > What do you think about continue without dom0less support and discuss
> > dom0less once we done with the main part?
>
> That's OK, especially if you are happy to work on dom0less support
> later.
Sure I will be happy to work on dom0less support later.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-21 15:07 ` Oleksii Moisieiev
@ 2022-01-21 20:49 ` Stefano Stabellini
2022-01-24 18:22 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-21 20:49 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Fri, 21 Jan 2022, Oleksii Moisieiev wrote:
> On Thu, Jan 20, 2022 at 02:29:41PM -0800, Stefano Stabellini wrote:
> > On Thu, 20 Jan 2022, Oleksii Moisieiev wrote:
> > > On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> > > > On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > > > > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > > > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > Hi Stefano,
> > > > > > > > >
> > > > > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > Hi Stefano,
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The following features are implemented:
> > > > > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > >
> > > > > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > > > > >
> > > > > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > > > > +
> > > > > > > > > > > > > endmenu
> > > > > > > > > > > > >
> > > > > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > > > > + default n
> > > > > > > > > > > > > + depends on SCI
> > > > > > > > > > > > > + ---help---
> > > > > > > > > > > > > +
> > > > > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > > > > + for communication.
> > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > > > > obj-y += sci.o
> > > > > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > > > > +/*
> > > > > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > + *
> > > > > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > > > > + *
> > > > > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > > > > + *
> > > > > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > > > > + *
> > > > > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > > > > + */
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > > > > >
> > > > > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > > > > >
> > > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > > > > >
> > > > > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > > > > channels:
> > > > > > > > > > >
> > > > > > > > > > > reserved-memory {
> > > > > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > > > > no-map;
> > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > > > > };
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > > > > the current scmi channel:
> > > > > > > > > > >
> > > > > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > >
> > > > > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > > > > assigned to this agent.
> > > > > > > > > >
> > > > > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > > > > >
> > > > > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > > > > >
> > > > > > > > > > reserved-memory {
> > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > ...
> > > > > > > > > >
> > > > > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > > > > "linux,scmi_mem"?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > That was my first implementation. But I've met a problem with
> > > > > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > > > > there were some if, checking if memory weren't reserved.
> > > > > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > > > > shmem page.
> > > > > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > > > > which has no compatible string and provides no-map property.
> > > > > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > > > > space for the domain.
> > > > > > > > >
> > > > > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > > > > and scmi_devid property to the
> > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > > > > Those node and property are needed only for Xen and useless for
> > > > > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > > > > What do you think about it?
> > > > > > > >
> > > > > > > > Reply below
> > > > > > > >
> > > > > > > > [...]
> > > > > > > >
> > > > > > > >
> > > > > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > > > > >
> > > > > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > > > > >
> > > > > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > > > > work on the Xen code that makes use of it.
> > > > > > > > > >
> > > > > > > > > > Does it make sense?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > > > > >
> > > > > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > > > > properties into generic bindings. It is a problem both from a
> > > > > > > > specification perspective (because it has hard to handle Xen specific
> > > > > > > > cases in fully generic bindings, especially as those bindings are
> > > > > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > > > > >
> > > > > > > >
> > > > > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > > > > driver too. Why the Linux driver doesn't need it?
> > > > > > > >
> > > > > > >
> > > > > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > > > > On non-virtualized systems - there is no need of this call, because OS
> > > > > > > is the only one entity, running on the system.
> > > > > >
> > > > > > OK. Even if it is only required for virtualized systems, I think that
> > > > > > scmi_devid is important enough that should be part of the upstream
> > > > > > binding. I think it is worth starting an email thread on the LKML with
> > > > > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > > > > scmi_devid to the binding.
> > > > > >
> > > > > >
> > > > > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > > > > list of device_ids to dom.cfg, such as:
> > > > > > > sci_devs = [ 0, 1, 15, 35 ];
> > > > > > >
> > > > > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > > > > We can probably make hypercall taking devid list as input parameter.
> > > > > > > This will take only 1 hypercall to setup sci permissions.
> > > > > >
> > > > > > But how would a user know which are the right SCMI IDs to add to the
> > > > > > sci_devs list? Would the user have to go and read the reference manual
> > > > > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > > > > If that is the case, then I think that it would be better to add
> > > > > > scmi_devid to device tree.
> > > > > >
> > > > > > In general, I think this configuration should happen automatically
> > > > > > without user intervention. The user should just specify "enable SCMI"
> > > > > > and it should work.
> > > > > >
> > > > > >
> > > > > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > > > > way, I don't think we should need linux,scmi_mem.
> > > > > > >
> > > > > > > This requires further investigation. I will try to make implementation
> > > > > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > > > > reuslts with you.
> > > > > >
> > > > > > OK, thanks.
> > > > >
> > > > > Hi Stefano,
> > > > >
> > > > > As I did some investigation about using reserved-memory area
> > > > > linux,scmi_mem and now I need your advice.
> > > > >
> > > > > I see 2 possible implementations for now:
> > > > > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > > > > reserved memory region.
> > > > > So device-tree will look like this:
> > > > >
> > > > > reserved-memory {
> > > > > /* reserved region for scmi channels*/
> > > > > scmi_memory: region@53FF0000{
> > > > > no-map;
> > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > };
> > > > > };
> > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > compatible = "arm,scmi-shmem";
> > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > memory-region = <&scmi_memory>;
> > > > > };
> > > > >
> > > > > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > > > > that xen can find reserved memory region without adding additional names
> > > > > to the device-tree bindings.
> > > > > memory-region parameter as a reference to reserved memory and region
> > > > > creation described in:
> > > > > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> > > > >
> > > > > This approach I've implemented already and it works.
> > > >
> > > > This approach would require a discussion with the upstream device tree
> > > > maintainers. Likely, we would need to add a note about the usage of the
> > > > "memory-region" property to arm,scmi.yaml.
> > > >
> > > > Also, I have the feeling that they would ask to add the "memory-region"
> > > > property directly to the "arm,scmi-smc" node, as an alternative (or
> > > > in addition) to the existing "shmem" property.
> > > >
> > > > That said, from my point of view this approach is also a viable option.
> > > > I don't see any major problems.
> > > >
> > > > The main question (after reading everything else that you wrote below)
> > > > is whether the "arm,scmi-smc" node in this case could be automatically
> > > > generated.
> > > >
> > >
> > > arm,scmi-smc node can be generated in both cases. I think I'd leave it
> > > as backup in case if the second approach will not work.
> > >
> > > >
> > > > > 2) The second approach is the format you suggested:
> > > > > > > > > > reserved-memory {
> > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > };
> > > > > > > > > > ...
> > > > >
> > > > > This approach has an advantage that xen ARM_SCI driver do not know about
> > > > > how channels are placed in the reserved memory, but introduces some
> > > > > disadvantages:
> > > > > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > > > > in the device-tree. In current implementation I have separate scmi.dtsi
> > > > > file which introduces scmi support for both XEN-based and
> > > > > non-virtualized systems. Having 14 extra channels in the device-tree may
> > > > > be confusing.
> > > >
> > > > I can see that while it would be ideal for Xen to see all 14+1 channels
> > > > in device tree (on the host device tree), we wouldn't want to expose all
> > > > of them to the domains, not even to dom0.
> > > >
> > > > How many channels do we want dom0 to see by the way? For this
> > > > discussion, I'll just assume for now that dom0 only sees 1 channel like
> > > > the domUs.
> > >
> > > For dom0 we need only one channel.
> > >
> > > >
> > > > Now we have a problem: how do we go about "filtering" the
> > > > "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> > > > below in point b).
> > > >
> > >
> > > Xen will not need to filter "arm,scmi-shmem" node. It will just
> > > create shmem node in Domain device-tree. I don't see any problem for xen
> > > configuration here.
> > > What bothers me here is that I set scmi configuration for platform dts,
> > > not for xen or domu dts files.
> > > So for example I have the following structure of the dts files for my
> > > platform (r8a77961-salvator-xs):
> > > * r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
> > > scmi_devid for the devices, that should use scmi.
> > > * r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
> > > It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
> > > is used for system with no hypervisor.
> > > * r8a77961-salvator-xs-xen.dts - dts file for xen which includes
> > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > > * r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
> > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > >
> > > In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
> > > r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
> > >
> > > I can give you a link to Merge request with this changes if you need it.
> > >
> > > For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
> > > nodes will be omitted and new will be generated for the domains.
> > >
> > > What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
> > >
> > > Just got an idea while writing this: I can create only one
> > > "arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
> > > needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
> > >
> > > Then we will have valid configurations for all cases.
> > > This can be a solution. What do you think?
> >
> > It is good that you brought this up because it helps me explain what I
> > mean. And of course it is up to you where you place the nodes in the
> > various dts files at your disposal. Either way it would work but I think
> > they should belong to r8a77961-salvator-xs.dts.
> >
> > Generally the platform vendor (e.g. Xilinx) provides a device tree
> > description of the platform to use including all the available resources
> > and firmware interfaces. In your case it would be r8a77961-scmi.dtsi +
> > r8a77961-salvator-xs.dts. This is what I call the "host device tree"
> > below. Users should be able to boot a fully functional system using the
> > host device tree pretty much "as is" to run Xen, Linux or any other
> > software.
> >
> > Certainly the SCMI device tree description should be part of the host
> > device tree, so in your case it would be r8a77961-salvator-xs.dts. And
> > the description should include all 14+1 channels because this is the
> > generic platform description -- we cannot know for sure how the users
> > are going to use the system.
> >
> > This is why r8a77961-salvator-xs-xen.dts should be as small as possible
> > or ideally inexistent. There shouldn't be a need for a special device
> > tree modification to allow Xen to run. In reality, even at Xilinx we
> > have something like r8a77961-salvator-xs-xen.dts, although it is really
> > small.
> >
> > But I see that r8a77961-salvator-xs-xen.dts could be viewed as the
> > device tree additions to run hypervisors and from that point of view it
> > is more acceptable to place the 14 channels there.
> >
> > The biggest problem is r8a77961-salvator-xs-domu.dts: who is going to
> > write it? And how? It wouldn't be provided by the platform vendor, so it
> > is the user the one that has to find a way to write it.
> >
> > I know the user already has to write a partial DTB for device
> > assignment, but any time the process is more complex than "copy the host
> > device tree node for device XXX to the partial DTB" it is a problem.
> > Errors are made and the system doesn't work.
> >
> > I think we don't want to make it even more difficult by having to
> > manually produce the SCMI domU description too. The SCMI description for
> > domU could be automatically generated by Xen, or libxl/xl. If that's an
> > issue, then the SCMI description could be automatically generated by an
> > external tool but I think it would make things more complex and harder
> > to maintain.
> >
> > In short my point of view is:
> > - r8a77961-scmi.dtsi + r8a77961-salvator-xs.dts should be as generic as
> > possible so the SCMI nodes should have 14+1 channels
> > - but putting the 14 channels in r8a77961-salvator-xs-xen.dts is still
> > OKish
> > - it is important that r8a77961-salvator-xs-domu.dts is automatically
> > generated by Xen or libxl or another software tool
> >
>
> Thank you for the detailed response. I'll put all 14+1 channels to
> r8a77961-salvator-xs.dts then.
> I've described my thoughts about generation of the arm,scmi-smc node below.
>
> >
> > > > > b) In case if we have all 15 channels, described in partial device-tree,
> > > >
> > > > I think you meant "described in the host device tree", right?
> > > >
> > > Yeah that's what I've meant.
> > > >
> > > > > we should not copy any node to the domain device-tree. I think it will
> > > > > be better to generate arm,scmi-shmem node in the Domain device-tree.
> > > >
> > > > Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> > > > device tree description for the DomU/Dom0 based on the channels
> > > > allocated to the domain.
> > > >
> > > >
> > > > > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > > > > node can't be generated. I prefer it to be copied from the partial
> > > > > device-tree because it includes some platform specific configuration,
> > > > > such as func-id and list of the protocols (for example different
> > > > > platforms may require different list of the protocols). So in this
> > > > > case we will have 1 node copied and 1 node generated.
> > > > >
> > > > > I think even for dom0less we should use arm,scmi-smc node from the
> > > > > device-tree because protocol configuration and funcid is related to the
> > > > > platform.
> > > >
> > > > I am not sure I understood what you wrote. You are saying that the
> > > > "arm,scmi-smc" node includes some platform specific configurations so
> > > > it cannot be automatically generated by Xen (or by the tools) and
> > > > instead it needs to be manually provided as part of the partial dtb for
> > > > the domU. Is that correct?
> > > >
> > > > If so, I would like to understand the reasons behind it. Manual
> > > > device tree editing is problematic.
> > > >
> > > > I looked for "func-id" in
> > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> > > > find any results. Do you have an example of the platform specific
> > > > configuration or protocol configuration that would make it difficult to
> > > > automatically generate the "arm,scmi-smc" node for the domains?
> > >
> > > Sorry, I used wrong term (used term from the specification), arm,smc-id
> > > of cause.
> > >
> > > >
> > > > Also, is this a problem just for approach #2 or also for approach #1?
> > > > If it is a problem only for approach #2, then let's just go with
> > > > approach #1.
> > > >
> > >
> > > We can't copy "arm,scmi-smc" in both approaches. The difference is that
> > > in the first approach we can copy both "arm,scmi-smc" and
> > > "arm,scmi-shmem" nodes while in the second approach we should copy
> > > "arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
> > >
> > > arm,scmi-smc node can't be generated because it includes properties and
> > > configurations that depends from platform and should be get from the
> > > device tree.
> > > Here is "arm,scmi-smc" node expample:
> > > firmware {
> > > scmi {
> > > compatible = "arm,scmi-smc"
> > > arm,smc-id = <0x82000002>;
> > > shmem = <&cpu_scp_shm>;
> > > #address-cells = <1>;
> > > #size-cells = <0>;
> > > scmi_power: protocol@11 {
> > > reg = <0x11>;
> > > #power-domain-cells = <1>;
> > > };
> > >
> > > scmi_clock: protocol@14 {
> > > ...
> > > };
> > >
> > > scmi_reset: protocol@16 {
> > > ...
> > > };
> > > ...
> > > };
> > > };
> > >
> > > It has 3 configurable options:
> > > * arm,smc-id parameter, setting func_id for scmi protocol. This id can be
> > > different for different platforms.
> > > For example stm32mp1 architecture use different scm-id for different
> > > agents:
> > > https://urldefense.com/v3/__https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h*L39__;Iw!!GF_29dbcQIUBPA!mTRUjtSg19iVaYo3Cgjop5ckPWqKsHVo1EZCEA1zCbod9KpNSXX291A8vAuNdTCr46MA$ [github[.]com]
> > >
> > > * shmem which includes phandle to arm,scmi-shmem node. But this is not
> > > a problem and can be updated.
> > >
> > > * list of the protocol subnodes. This is also configurable parameter,
> > > not regs or names, but the number of the protocols. For example onle
> > > platform can use power-domains/clock/resets via scmi, when another will
> > > require volage-control and sensor-management to be added.
> > >
> > > Xen should know this parameters to be able to generate "arm,scmi-smc" node.
> > >
> > > Also we're currently discussing new scmi protocol with ARM: Pinctrl over
> > > SCMI.
> > >
> > > It should allow domains to access pinctrl subsystem, placed in Firmware
> > > through SCMI protocol.
> > > scmi_pinctrl node will look like this:
> > >
> > > firmware {
> > > scmi {
> > > ...
> > > scmi_pinctrl: protocol@18 {
> > > reg = <0x18>;
> > > #pinctrl-cells = <0>;
> > >
> > > i2c2_pins: i2c2 {
> > > groups = <74>; /* i2c2_a */
> > > function = <15>; /* i2c2 */
> > > };
> > >
> > > irq0_pins: irq0 {
> > > groups = <81>; /* intc_ex_irq0 */
> > > function = <19>; /* intc_ex */
> > > };
> > >
> > > avb_pins: avb {
> > > mux {
> > > /* avb_link, avb_mdio, avb_mii */
> > > groups = <17>, <21>, <22>;
> > > function = <1>; /* avb */
> > > };
> > >
> > > pins_mdio {
> > > groups = <21>; /* avb_mdio */
> > > drive-strength = <24>;
> > > };
> > >
> > > pins_mii_tx {
> > > /* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
> > > PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
> > > pins = <242>, <240>, <236>, <237>, <238>, <239>;
> > > drive-strength = <12>;
> > > };
> > > };
> > > ...
> > > };
> > > };
> > > };
> > >
> > > So "arm,scmi-smc" node will have even more platform specific settings.
> > >
> > > >
> > > > > I prefer the second approach and will try to make it if it's OK to copy
> > > > > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > > > > node.
> > > > >
> > > > > What do you think about that?
> > > >
> > > > From a device tree specification perspective, I think both approaches
> > > > are OK (with a minor comment on the first approach as I wrote above.)
> > > >
> > > > But from a Xen perspective I think it is important that we don't require
> > > > the user to manually provide the SCMI configuration in the partial DTB.
> > > > It would be better if we could generate it automatically from Xen or the
> > > > tools (or even an independent script). Or copy the "arm,scmi-smc" node
> > > > from the host device tree to the domU device tree without modifications.
> > >
> > > I think copy "arm,scmi-smc" node is the only option we have.
> > > I'm not sure what do you mean under "host device tree" if you mean Xen
> > > device-tree - then I think it will not cover the case with stm32mp1 I've
> > > mentioned above. I think it will be better to copy "arm,scmi-smc" node
> > > from Domu partial Device-tree to Domu device-tree.
> > > So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
> > > AGENT1 scm-is set in domu device-tree and copied to dom-u.
> > >
> > > Do you agree with my points?
> >
> > I think we are saying similar things, but we are getting mixed up with
> > the terminology. Let's start from the basics :-)
> >
> > # Host device tree
> > The device tree given to Xen at boot time. This is the device tree that
> > Xen parses to discover what's available on the platform. In your case,
> > it seems to include r8a77961-salvator-xs-xen.dts.
> >
> > # Partial DTB
> > (Ignoring Dom0less) this is the small DTB that gets passed to xl with
> > the "device_tree" option in the xl config file. It is copied verbatim
> > to the domU device tree by xl/libxl.
> >
> > # Copy the "arm,scmi-smc" node from host device tree
> > This means that the domU "arm,scmi-smc" node is an exact copy of the
> > host device tree SCMI node. I don't think this is actually possible in
> > most cases because the domU description is typically a bit different
> > from the host description. For instance, the host description could
> > include 14+1 channels while the domU description should only include 1
> > channel.
> >
> > # Copy the "arm,scmi-smc" node from the partial DTB
> > This implies that somebody or something create an "arm,scmi-smc" node
> > for the domU and placed it into the partial DTB. Then, Xen and/or
> > xl/libxl will copy the node from the partial DTB to the DomU device
> > tree. The main question in this case is: who is going to write the
> > partial DTB? We dont want the user (i.e. a person) to have to manually
> > write the SCMI description for the domU. It should be an automated tools
> > that does it. At that point, it is easier if it is Xen or xl/libxl.
> > Alternativaly, we could think of an external tool but I think it would
> > make things more difficult to maintain.
> >
> > # Generate the "arm,scmi-smc" node for domUs
> > When I write "generate the arm,scmi-smc node", I mean that Xen and
> > libxl/xl will generate the "arm,scmi-smc" node for the domU. Thus, the
> > node will not be copied from the partial DTB or from the device tree,
> > instead, it should be created directly by Xen and/or libxl/xl.
> >
> > However, the domU "arm,scmi-smc" node could still be derived from the
> > host device tree "arm,scmi-smc" node. In other words, Xen or xl/libxl
> > would look at the host device tree "arm,scmi-smc" node, copy it to the
> > domU device tree while making as many changes as necessary.
> >
> > The DomU "arm,scmi-smc" node doesn't have to be entirely fake and
> > static. It could be dynamically created to match the host device tree
> > description. I think this is the best option.
> >
> >
> > # Conclusion
> > I am suggesting that Xen and/or libxl automatically produce the
> > "arm,scmi-smc" node for domUs based on the host device tree description
> > and based on the channel mapped to the domU. This way, the user (a
> > person) doesn't have to go and manually edit the domU partial DTB.
> >
>
> That sounds reasonable. The problem is that arm,scmi-smc node can be
> copmlicated and include a lot of configuration. Also for different
> mediators this node can be different.
> As I inderstand, there is no mechanism for xl to access host device-tree
> right now. Correct me if I'm wrong.
Yes, you are right. And the lack of a mechanism for xl to access the
host device tree is a problem.
> I see the following way we can generate arm,scmi-smc node for DomU:
> We say that if scmi-smc mediator is enabled - then Dom0 is configured to
> use SCMI. This means that Dom0 device-tree will have arm,scmi-smc node
> and it can be reached from the userspace.
So far so good
> In this case xl can use infromation from /proc/device-tree/firmware/scmi
> to generate arm,scmi-smc node for DomU. But in this case xl should know
> the exact path of scmi node.
>
> Or we can generate some special node, called "shared" in Dom0 device-tree
> which will include copy of the arm,scmi-smc node, which can be used for domains.
> In this case xl can scan /proc/device-tree/shared node and find
> arm,scmi-smc copatible node and use it to generate arm,scmi-smc node for
> DomU.
> Also this can be used for another features in future.
>
> What do you think about this?
Basing the domU SCMI node generation on the dom0 SCMI node is not great,
because it should be based on the host device tree rather than dom0 and
the dom0 SCMI description will be different.
Instead of copying just the host SCMI node somewhere in the dom0 device
tree so that xl/libxl can access it via /proc/device-tree, I think it
would be better to make the full host DTB available to xl/libxl.
There are probably many ways to do this, but there are a couple I can
think of on top of my head:
- introduce a new hypercall to get the full host dtb from Xen
The hypercall would pass an address and a size in guest physical memory
and Xen would copy the host DTB to it.
- introduce something like /proc/device-tree to hypfs
See xen/common/hypfs.c
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-21 20:49 ` Stefano Stabellini
@ 2022-01-24 18:22 ` Oleksii Moisieiev
2022-01-24 19:06 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-24 18:22 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Julien Grall, Volodymyr Babchuk, Bertrand Marquis
On Fri, Jan 21, 2022 at 12:49:55PM -0800, Stefano Stabellini wrote:
> On Fri, 21 Jan 2022, Oleksii Moisieiev wrote:
> > On Thu, Jan 20, 2022 at 02:29:41PM -0800, Stefano Stabellini wrote:
> > > On Thu, 20 Jan 2022, Oleksii Moisieiev wrote:
> > > > On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> > > > > On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > > > > > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > > > > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > Hi Stefano,
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > Hi Stefano,
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The following features are implemented:
> > > > > > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > endmenu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > > > > > + default n
> > > > > > > > > > > > > > + depends on SCI
> > > > > > > > > > > > > > + ---help---
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > > > > > + for communication.
> > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > > > > > obj-y += sci.o
> > > > > > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > > > > > +/*
> > > > > > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > > > > > + */
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > > > > > >
> > > > > > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > > > > > channels:
> > > > > > > > > > > >
> > > > > > > > > > > > reserved-memory {
> > > > > > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > > > > > no-map;
> > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > > > > > the current scmi channel:
> > > > > > > > > > > >
> > > > > > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > >
> > > > > > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > > > > > assigned to this agent.
> > > > > > > > > > >
> > > > > > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > > > > > >
> > > > > > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > > > > > >
> > > > > > > > > > > reserved-memory {
> > > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > ...
> > > > > > > > > > >
> > > > > > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > > > > > "linux,scmi_mem"?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That was my first implementation. But I've met a problem with
> > > > > > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > > > > > there were some if, checking if memory weren't reserved.
> > > > > > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > > > > > shmem page.
> > > > > > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > > > > > which has no compatible string and provides no-map property.
> > > > > > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > > > > > space for the domain.
> > > > > > > > > >
> > > > > > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > > > > > and scmi_devid property to the
> > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > > > > > Those node and property are needed only for Xen and useless for
> > > > > > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > > > > > What do you think about it?
> > > > > > > > >
> > > > > > > > > Reply below
> > > > > > > > >
> > > > > > > > > [...]
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > > > > > >
> > > > > > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > > > > > >
> > > > > > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > > > > > work on the Xen code that makes use of it.
> > > > > > > > > > >
> > > > > > > > > > > Does it make sense?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > > > > > >
> > > > > > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > > > > > properties into generic bindings. It is a problem both from a
> > > > > > > > > specification perspective (because it has hard to handle Xen specific
> > > > > > > > > cases in fully generic bindings, especially as those bindings are
> > > > > > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > > > > > driver too. Why the Linux driver doesn't need it?
> > > > > > > > >
> > > > > > > >
> > > > > > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > > > > > On non-virtualized systems - there is no need of this call, because OS
> > > > > > > > is the only one entity, running on the system.
> > > > > > >
> > > > > > > OK. Even if it is only required for virtualized systems, I think that
> > > > > > > scmi_devid is important enough that should be part of the upstream
> > > > > > > binding. I think it is worth starting an email thread on the LKML with
> > > > > > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > > > > > scmi_devid to the binding.
> > > > > > >
> > > > > > >
> > > > > > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > > > > > list of device_ids to dom.cfg, such as:
> > > > > > > > sci_devs = [ 0, 1, 15, 35 ];
> > > > > > > >
> > > > > > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > > > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > > > > > We can probably make hypercall taking devid list as input parameter.
> > > > > > > > This will take only 1 hypercall to setup sci permissions.
> > > > > > >
> > > > > > > But how would a user know which are the right SCMI IDs to add to the
> > > > > > > sci_devs list? Would the user have to go and read the reference manual
> > > > > > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > > > > > If that is the case, then I think that it would be better to add
> > > > > > > scmi_devid to device tree.
> > > > > > >
> > > > > > > In general, I think this configuration should happen automatically
> > > > > > > without user intervention. The user should just specify "enable SCMI"
> > > > > > > and it should work.
> > > > > > >
> > > > > > >
> > > > > > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > > > > > way, I don't think we should need linux,scmi_mem.
> > > > > > > >
> > > > > > > > This requires further investigation. I will try to make implementation
> > > > > > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > > > > > reuslts with you.
> > > > > > >
> > > > > > > OK, thanks.
> > > > > >
> > > > > > Hi Stefano,
> > > > > >
> > > > > > As I did some investigation about using reserved-memory area
> > > > > > linux,scmi_mem and now I need your advice.
> > > > > >
> > > > > > I see 2 possible implementations for now:
> > > > > > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > > > > > reserved memory region.
> > > > > > So device-tree will look like this:
> > > > > >
> > > > > > reserved-memory {
> > > > > > /* reserved region for scmi channels*/
> > > > > > scmi_memory: region@53FF0000{
> > > > > > no-map;
> > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > };
> > > > > > };
> > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > compatible = "arm,scmi-shmem";
> > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > memory-region = <&scmi_memory>;
> > > > > > };
> > > > > >
> > > > > > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > > > > > that xen can find reserved memory region without adding additional names
> > > > > > to the device-tree bindings.
> > > > > > memory-region parameter as a reference to reserved memory and region
> > > > > > creation described in:
> > > > > > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> > > > > >
> > > > > > This approach I've implemented already and it works.
> > > > >
> > > > > This approach would require a discussion with the upstream device tree
> > > > > maintainers. Likely, we would need to add a note about the usage of the
> > > > > "memory-region" property to arm,scmi.yaml.
> > > > >
> > > > > Also, I have the feeling that they would ask to add the "memory-region"
> > > > > property directly to the "arm,scmi-smc" node, as an alternative (or
> > > > > in addition) to the existing "shmem" property.
> > > > >
> > > > > That said, from my point of view this approach is also a viable option.
> > > > > I don't see any major problems.
> > > > >
> > > > > The main question (after reading everything else that you wrote below)
> > > > > is whether the "arm,scmi-smc" node in this case could be automatically
> > > > > generated.
> > > > >
> > > >
> > > > arm,scmi-smc node can be generated in both cases. I think I'd leave it
> > > > as backup in case if the second approach will not work.
> > > >
> > > > >
> > > > > > 2) The second approach is the format you suggested:
> > > > > > > > > > > reserved-memory {
> > > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > > };
> > > > > > > > > > > ...
> > > > > >
> > > > > > This approach has an advantage that xen ARM_SCI driver do not know about
> > > > > > how channels are placed in the reserved memory, but introduces some
> > > > > > disadvantages:
> > > > > > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > > > > > in the device-tree. In current implementation I have separate scmi.dtsi
> > > > > > file which introduces scmi support for both XEN-based and
> > > > > > non-virtualized systems. Having 14 extra channels in the device-tree may
> > > > > > be confusing.
> > > > >
> > > > > I can see that while it would be ideal for Xen to see all 14+1 channels
> > > > > in device tree (on the host device tree), we wouldn't want to expose all
> > > > > of them to the domains, not even to dom0.
> > > > >
> > > > > How many channels do we want dom0 to see by the way? For this
> > > > > discussion, I'll just assume for now that dom0 only sees 1 channel like
> > > > > the domUs.
> > > >
> > > > For dom0 we need only one channel.
> > > >
> > > > >
> > > > > Now we have a problem: how do we go about "filtering" the
> > > > > "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> > > > > below in point b).
> > > > >
> > > >
> > > > Xen will not need to filter "arm,scmi-shmem" node. It will just
> > > > create shmem node in Domain device-tree. I don't see any problem for xen
> > > > configuration here.
> > > > What bothers me here is that I set scmi configuration for platform dts,
> > > > not for xen or domu dts files.
> > > > So for example I have the following structure of the dts files for my
> > > > platform (r8a77961-salvator-xs):
> > > > * r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
> > > > scmi_devid for the devices, that should use scmi.
> > > > * r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
> > > > It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
> > > > is used for system with no hypervisor.
> > > > * r8a77961-salvator-xs-xen.dts - dts file for xen which includes
> > > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > > > * r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
> > > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > > >
> > > > In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
> > > > r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
> > > >
> > > > I can give you a link to Merge request with this changes if you need it.
> > > >
> > > > For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
> > > > nodes will be omitted and new will be generated for the domains.
> > > >
> > > > What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
> > > >
> > > > Just got an idea while writing this: I can create only one
> > > > "arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
> > > > needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
> > > >
> > > > Then we will have valid configurations for all cases.
> > > > This can be a solution. What do you think?
> > >
> > > It is good that you brought this up because it helps me explain what I
> > > mean. And of course it is up to you where you place the nodes in the
> > > various dts files at your disposal. Either way it would work but I think
> > > they should belong to r8a77961-salvator-xs.dts.
> > >
> > > Generally the platform vendor (e.g. Xilinx) provides a device tree
> > > description of the platform to use including all the available resources
> > > and firmware interfaces. In your case it would be r8a77961-scmi.dtsi +
> > > r8a77961-salvator-xs.dts. This is what I call the "host device tree"
> > > below. Users should be able to boot a fully functional system using the
> > > host device tree pretty much "as is" to run Xen, Linux or any other
> > > software.
> > >
> > > Certainly the SCMI device tree description should be part of the host
> > > device tree, so in your case it would be r8a77961-salvator-xs.dts. And
> > > the description should include all 14+1 channels because this is the
> > > generic platform description -- we cannot know for sure how the users
> > > are going to use the system.
> > >
> > > This is why r8a77961-salvator-xs-xen.dts should be as small as possible
> > > or ideally inexistent. There shouldn't be a need for a special device
> > > tree modification to allow Xen to run. In reality, even at Xilinx we
> > > have something like r8a77961-salvator-xs-xen.dts, although it is really
> > > small.
> > >
> > > But I see that r8a77961-salvator-xs-xen.dts could be viewed as the
> > > device tree additions to run hypervisors and from that point of view it
> > > is more acceptable to place the 14 channels there.
> > >
> > > The biggest problem is r8a77961-salvator-xs-domu.dts: who is going to
> > > write it? And how? It wouldn't be provided by the platform vendor, so it
> > > is the user the one that has to find a way to write it.
> > >
> > > I know the user already has to write a partial DTB for device
> > > assignment, but any time the process is more complex than "copy the host
> > > device tree node for device XXX to the partial DTB" it is a problem.
> > > Errors are made and the system doesn't work.
> > >
> > > I think we don't want to make it even more difficult by having to
> > > manually produce the SCMI domU description too. The SCMI description for
> > > domU could be automatically generated by Xen, or libxl/xl. If that's an
> > > issue, then the SCMI description could be automatically generated by an
> > > external tool but I think it would make things more complex and harder
> > > to maintain.
> > >
> > > In short my point of view is:
> > > - r8a77961-scmi.dtsi + r8a77961-salvator-xs.dts should be as generic as
> > > possible so the SCMI nodes should have 14+1 channels
> > > - but putting the 14 channels in r8a77961-salvator-xs-xen.dts is still
> > > OKish
> > > - it is important that r8a77961-salvator-xs-domu.dts is automatically
> > > generated by Xen or libxl or another software tool
> > >
> >
> > Thank you for the detailed response. I'll put all 14+1 channels to
> > r8a77961-salvator-xs.dts then.
> > I've described my thoughts about generation of the arm,scmi-smc node below.
> >
> > >
> > > > > > b) In case if we have all 15 channels, described in partial device-tree,
> > > > >
> > > > > I think you meant "described in the host device tree", right?
> > > > >
> > > > Yeah that's what I've meant.
> > > > >
> > > > > > we should not copy any node to the domain device-tree. I think it will
> > > > > > be better to generate arm,scmi-shmem node in the Domain device-tree.
> > > > >
> > > > > Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> > > > > device tree description for the DomU/Dom0 based on the channels
> > > > > allocated to the domain.
> > > > >
> > > > >
> > > > > > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > > > > > node can't be generated. I prefer it to be copied from the partial
> > > > > > device-tree because it includes some platform specific configuration,
> > > > > > such as func-id and list of the protocols (for example different
> > > > > > platforms may require different list of the protocols). So in this
> > > > > > case we will have 1 node copied and 1 node generated.
> > > > > >
> > > > > > I think even for dom0less we should use arm,scmi-smc node from the
> > > > > > device-tree because protocol configuration and funcid is related to the
> > > > > > platform.
> > > > >
> > > > > I am not sure I understood what you wrote. You are saying that the
> > > > > "arm,scmi-smc" node includes some platform specific configurations so
> > > > > it cannot be automatically generated by Xen (or by the tools) and
> > > > > instead it needs to be manually provided as part of the partial dtb for
> > > > > the domU. Is that correct?
> > > > >
> > > > > If so, I would like to understand the reasons behind it. Manual
> > > > > device tree editing is problematic.
> > > > >
> > > > > I looked for "func-id" in
> > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> > > > > find any results. Do you have an example of the platform specific
> > > > > configuration or protocol configuration that would make it difficult to
> > > > > automatically generate the "arm,scmi-smc" node for the domains?
> > > >
> > > > Sorry, I used wrong term (used term from the specification), arm,smc-id
> > > > of cause.
> > > >
> > > > >
> > > > > Also, is this a problem just for approach #2 or also for approach #1?
> > > > > If it is a problem only for approach #2, then let's just go with
> > > > > approach #1.
> > > > >
> > > >
> > > > We can't copy "arm,scmi-smc" in both approaches. The difference is that
> > > > in the first approach we can copy both "arm,scmi-smc" and
> > > > "arm,scmi-shmem" nodes while in the second approach we should copy
> > > > "arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
> > > >
> > > > arm,scmi-smc node can't be generated because it includes properties and
> > > > configurations that depends from platform and should be get from the
> > > > device tree.
> > > > Here is "arm,scmi-smc" node expample:
> > > > firmware {
> > > > scmi {
> > > > compatible = "arm,scmi-smc"
> > > > arm,smc-id = <0x82000002>;
> > > > shmem = <&cpu_scp_shm>;
> > > > #address-cells = <1>;
> > > > #size-cells = <0>;
> > > > scmi_power: protocol@11 {
> > > > reg = <0x11>;
> > > > #power-domain-cells = <1>;
> > > > };
> > > >
> > > > scmi_clock: protocol@14 {
> > > > ...
> > > > };
> > > >
> > > > scmi_reset: protocol@16 {
> > > > ...
> > > > };
> > > > ...
> > > > };
> > > > };
> > > >
> > > > It has 3 configurable options:
> > > > * arm,smc-id parameter, setting func_id for scmi protocol. This id can be
> > > > different for different platforms.
> > > > For example stm32mp1 architecture use different scm-id for different
> > > > agents:
> > > > https://urldefense.com/v3/__https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h*L39__;Iw!!GF_29dbcQIUBPA!mTRUjtSg19iVaYo3Cgjop5ckPWqKsHVo1EZCEA1zCbod9KpNSXX291A8vAuNdTCr46MA$ [github[.]com]
> > > >
> > > > * shmem which includes phandle to arm,scmi-shmem node. But this is not
> > > > a problem and can be updated.
> > > >
> > > > * list of the protocol subnodes. This is also configurable parameter,
> > > > not regs or names, but the number of the protocols. For example onle
> > > > platform can use power-domains/clock/resets via scmi, when another will
> > > > require volage-control and sensor-management to be added.
> > > >
> > > > Xen should know this parameters to be able to generate "arm,scmi-smc" node.
> > > >
> > > > Also we're currently discussing new scmi protocol with ARM: Pinctrl over
> > > > SCMI.
> > > >
> > > > It should allow domains to access pinctrl subsystem, placed in Firmware
> > > > through SCMI protocol.
> > > > scmi_pinctrl node will look like this:
> > > >
> > > > firmware {
> > > > scmi {
> > > > ...
> > > > scmi_pinctrl: protocol@18 {
> > > > reg = <0x18>;
> > > > #pinctrl-cells = <0>;
> > > >
> > > > i2c2_pins: i2c2 {
> > > > groups = <74>; /* i2c2_a */
> > > > function = <15>; /* i2c2 */
> > > > };
> > > >
> > > > irq0_pins: irq0 {
> > > > groups = <81>; /* intc_ex_irq0 */
> > > > function = <19>; /* intc_ex */
> > > > };
> > > >
> > > > avb_pins: avb {
> > > > mux {
> > > > /* avb_link, avb_mdio, avb_mii */
> > > > groups = <17>, <21>, <22>;
> > > > function = <1>; /* avb */
> > > > };
> > > >
> > > > pins_mdio {
> > > > groups = <21>; /* avb_mdio */
> > > > drive-strength = <24>;
> > > > };
> > > >
> > > > pins_mii_tx {
> > > > /* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
> > > > PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
> > > > pins = <242>, <240>, <236>, <237>, <238>, <239>;
> > > > drive-strength = <12>;
> > > > };
> > > > };
> > > > ...
> > > > };
> > > > };
> > > > };
> > > >
> > > > So "arm,scmi-smc" node will have even more platform specific settings.
> > > >
> > > > >
> > > > > > I prefer the second approach and will try to make it if it's OK to copy
> > > > > > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > > > > > node.
> > > > > >
> > > > > > What do you think about that?
> > > > >
> > > > > From a device tree specification perspective, I think both approaches
> > > > > are OK (with a minor comment on the first approach as I wrote above.)
> > > > >
> > > > > But from a Xen perspective I think it is important that we don't require
> > > > > the user to manually provide the SCMI configuration in the partial DTB.
> > > > > It would be better if we could generate it automatically from Xen or the
> > > > > tools (or even an independent script). Or copy the "arm,scmi-smc" node
> > > > > from the host device tree to the domU device tree without modifications.
> > > >
> > > > I think copy "arm,scmi-smc" node is the only option we have.
> > > > I'm not sure what do you mean under "host device tree" if you mean Xen
> > > > device-tree - then I think it will not cover the case with stm32mp1 I've
> > > > mentioned above. I think it will be better to copy "arm,scmi-smc" node
> > > > from Domu partial Device-tree to Domu device-tree.
> > > > So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
> > > > AGENT1 scm-is set in domu device-tree and copied to dom-u.
> > > >
> > > > Do you agree with my points?
> > >
> > > I think we are saying similar things, but we are getting mixed up with
> > > the terminology. Let's start from the basics :-)
> > >
> > > # Host device tree
> > > The device tree given to Xen at boot time. This is the device tree that
> > > Xen parses to discover what's available on the platform. In your case,
> > > it seems to include r8a77961-salvator-xs-xen.dts.
> > >
> > > # Partial DTB
> > > (Ignoring Dom0less) this is the small DTB that gets passed to xl with
> > > the "device_tree" option in the xl config file. It is copied verbatim
> > > to the domU device tree by xl/libxl.
> > >
> > > # Copy the "arm,scmi-smc" node from host device tree
> > > This means that the domU "arm,scmi-smc" node is an exact copy of the
> > > host device tree SCMI node. I don't think this is actually possible in
> > > most cases because the domU description is typically a bit different
> > > from the host description. For instance, the host description could
> > > include 14+1 channels while the domU description should only include 1
> > > channel.
> > >
> > > # Copy the "arm,scmi-smc" node from the partial DTB
> > > This implies that somebody or something create an "arm,scmi-smc" node
> > > for the domU and placed it into the partial DTB. Then, Xen and/or
> > > xl/libxl will copy the node from the partial DTB to the DomU device
> > > tree. The main question in this case is: who is going to write the
> > > partial DTB? We dont want the user (i.e. a person) to have to manually
> > > write the SCMI description for the domU. It should be an automated tools
> > > that does it. At that point, it is easier if it is Xen or xl/libxl.
> > > Alternativaly, we could think of an external tool but I think it would
> > > make things more difficult to maintain.
> > >
> > > # Generate the "arm,scmi-smc" node for domUs
> > > When I write "generate the arm,scmi-smc node", I mean that Xen and
> > > libxl/xl will generate the "arm,scmi-smc" node for the domU. Thus, the
> > > node will not be copied from the partial DTB or from the device tree,
> > > instead, it should be created directly by Xen and/or libxl/xl.
> > >
> > > However, the domU "arm,scmi-smc" node could still be derived from the
> > > host device tree "arm,scmi-smc" node. In other words, Xen or xl/libxl
> > > would look at the host device tree "arm,scmi-smc" node, copy it to the
> > > domU device tree while making as many changes as necessary.
> > >
> > > The DomU "arm,scmi-smc" node doesn't have to be entirely fake and
> > > static. It could be dynamically created to match the host device tree
> > > description. I think this is the best option.
> > >
> > >
> > > # Conclusion
> > > I am suggesting that Xen and/or libxl automatically produce the
> > > "arm,scmi-smc" node for domUs based on the host device tree description
> > > and based on the channel mapped to the domU. This way, the user (a
> > > person) doesn't have to go and manually edit the domU partial DTB.
> > >
> >
> > That sounds reasonable. The problem is that arm,scmi-smc node can be
> > copmlicated and include a lot of configuration. Also for different
> > mediators this node can be different.
> > As I inderstand, there is no mechanism for xl to access host device-tree
> > right now. Correct me if I'm wrong.
>
> Yes, you are right. And the lack of a mechanism for xl to access the
> host device tree is a problem.
>
>
> > I see the following way we can generate arm,scmi-smc node for DomU:
> > We say that if scmi-smc mediator is enabled - then Dom0 is configured to
> > use SCMI. This means that Dom0 device-tree will have arm,scmi-smc node
> > and it can be reached from the userspace.
>
> So far so good
>
>
> > In this case xl can use infromation from /proc/device-tree/firmware/scmi
> > to generate arm,scmi-smc node for DomU. But in this case xl should know
> > the exact path of scmi node.
> >
> > Or we can generate some special node, called "shared" in Dom0 device-tree
> > which will include copy of the arm,scmi-smc node, which can be used for domains.
> > In this case xl can scan /proc/device-tree/shared node and find
> > arm,scmi-smc copatible node and use it to generate arm,scmi-smc node for
> > DomU.
> > Also this can be used for another features in future.
> >
> > What do you think about this?
>
> Basing the domU SCMI node generation on the dom0 SCMI node is not great,
> because it should be based on the host device tree rather than dom0 and
> the dom0 SCMI description will be different.
>
> Instead of copying just the host SCMI node somewhere in the dom0 device
> tree so that xl/libxl can access it via /proc/device-tree, I think it
> would be better to make the full host DTB available to xl/libxl.
>
> There are probably many ways to do this, but there are a couple I can
> think of on top of my head:
>
> - introduce a new hypercall to get the full host dtb from Xen
> The hypercall would pass an address and a size in guest physical memory
> and Xen would copy the host DTB to it.
>
> - introduce something like /proc/device-tree to hypfs
> See xen/common/hypfs.c
Hi Stefano,
I've found the discussion, indirectly related to our topic [1], in which
hypervisor <-> toolchain interface topic was raised.
Where Andrii Anisov proposed to introduce another libxl type (f.e. File) which
will read device tree file and operate with a binary blob.
His idea is to introduce new Domctl hypecall, which is able to send device-tree
blob from toolstack to the hypervisor.
I was thinking about using his approach, but in opposite direction.
Such as introduce a set of DOMCTL, which gives ability for xl user to request
Host Device-tree nodes.
I see something like that:
XEN_DOMCTL_host_node_by_path - gets node path as input and returns blob with node;
XEN_DOMCTL_find_host_compatible_node - gets compatible string and "skip" number
for the case if there is more than 1 compatible node and returns blob with node.
Advandages for my sight are:
- I hope Andrii still has some code, which can be reused during the implementation;
- Xen can determine if the requesting node can be exposed to the guest, which
is additional layer of validation;
- no need to transfer all device-tree to the toolstack.
As for the hypfs this sounds like a separate feature to me. I'm not sure if we
need to provide the entire copy of the device-tree. Also there could be some
problems with this solution.
For example, xl tool have to scan whole hypfs tree to find node by phandle or
by compatible string.
Looking forward for your thoughts about that. Also I would be happy if Julien
could share his vision.
[1] https://xen.markmail.org/search/?q=from%3AAndrii+Anisov+device-tree#query:from%3AAndrii%20Anisov%20device-tree+page:2+mid:tvgyp7fzpiyyso3j+state:results
Oleksii
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-24 18:22 ` Oleksii Moisieiev
@ 2022-01-24 19:06 ` Stefano Stabellini
2022-01-24 19:26 ` Julien Grall
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-24 19:06 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, xen-devel, Julien Grall, Volodymyr Babchuk,
Bertrand Marquis
On Mon, 24 Jan 2022, Oleksii Moisieiev wrote:
> On Fri, Jan 21, 2022 at 12:49:55PM -0800, Stefano Stabellini wrote:
> > On Fri, 21 Jan 2022, Oleksii Moisieiev wrote:
> > > On Thu, Jan 20, 2022 at 02:29:41PM -0800, Stefano Stabellini wrote:
> > > > On Thu, 20 Jan 2022, Oleksii Moisieiev wrote:
> > > > > On Wed, Jan 19, 2022 at 05:28:21PM -0800, Stefano Stabellini wrote:
> > > > > > On Wed, 19 Jan 2022, Oleksii Moisieiev wrote:
> > > > > > > On Wed, Dec 22, 2021 at 06:23:24PM -0800, Stefano Stabellini wrote:
> > > > > > > > On Wed, 22 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > On Tue, Dec 21, 2021 at 01:22:50PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > On Tue, 21 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > Hi Stefano,
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 20, 2021 at 04:52:01PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > > On Mon, 20 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > > Hi Stefano,
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Dec 17, 2021 at 06:14:55PM -0800, Stefano Stabellini wrote:
> > > > > > > > > > > > > > On Tue, 14 Dec 2021, Oleksii Moisieiev wrote:
> > > > > > > > > > > > > > > This is the implementation of SCI interface, called SCMI-SMC driver,
> > > > > > > > > > > > > > > which works as the mediator between XEN Domains and Firmware (SCP, ATF etc).
> > > > > > > > > > > > > > > This allows devices from the Domains to work with clocks, resets and
> > > > > > > > > > > > > > > power-domains without access to CPG.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The following features are implemented:
> > > > > > > > > > > > > > > - request SCMI channels from ATF and pass channels to Domains;
> > > > > > > > > > > > > > > - set device permissions for Domains based on the Domain partial
> > > > > > > > > > > > > > > device-tree. Devices with permissions are able to work with clocks,
> > > > > > > > > > > > > > > resets and power-domains via SCMI;
> > > > > > > > > > > > > > > - redirect scmi messages from Domains to ATF.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Signed-off-by: Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > xen/arch/arm/Kconfig | 2 +
> > > > > > > > > > > > > > > xen/arch/arm/sci/Kconfig | 10 +
> > > > > > > > > > > > > > > xen/arch/arm/sci/Makefile | 1 +
> > > > > > > > > > > > > > > xen/arch/arm/sci/scmi_smc.c | 795 ++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > > xen/include/public/arch-arm.h | 1 +
> > > > > > > > > > > > > > > 5 files changed, 809 insertions(+)
> > > > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > > create mode 100644 xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > > index 186e1db389..02d96c6cfc 100644
> > > > > > > > > > > > > > > --- a/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > > +++ b/xen/arch/arm/Kconfig
> > > > > > > > > > > > > > > @@ -114,6 +114,8 @@ config SCI
> > > > > > > > > > > > > > > support. It allows guests to control system resourcess via one of
> > > > > > > > > > > > > > > SCI mediators implemented in XEN.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +source "arch/arm/sci/Kconfig"
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > endmenu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > menu "ARM errata workaround via the alternative framework"
> > > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Kconfig b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > > index 0000000000..9563067ddc
> > > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Kconfig
> > > > > > > > > > > > > > > @@ -0,0 +1,10 @@
> > > > > > > > > > > > > > > +config SCMI_SMC
> > > > > > > > > > > > > > > + bool "Enable SCMI-SMC mediator driver"
> > > > > > > > > > > > > > > + default n
> > > > > > > > > > > > > > > + depends on SCI
> > > > > > > > > > > > > > > + ---help---
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > + Enables mediator in XEN to pass SCMI requests from Domains to ATF.
> > > > > > > > > > > > > > > + This feature allows drivers from Domains to work with System
> > > > > > > > > > > > > > > + Controllers (such as power,resets,clock etc.). SCP is used as transport
> > > > > > > > > > > > > > > + for communication.
> > > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/Makefile b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > > index 837dc7492b..67f2611872 100644
> > > > > > > > > > > > > > > --- a/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/Makefile
> > > > > > > > > > > > > > > @@ -1 +1,2 @@
> > > > > > > > > > > > > > > obj-y += sci.o
> > > > > > > > > > > > > > > +obj-$(CONFIG_SCMI_SMC) += scmi_smc.o
> > > > > > > > > > > > > > > diff --git a/xen/arch/arm/sci/scmi_smc.c b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > > index 0000000000..2eb01ea82d
> > > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > > +++ b/xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > > @@ -0,0 +1,795 @@
> > > > > > > > > > > > > > > +/*
> > > > > > > > > > > > > > > + * xen/arch/arm/sci/scmi_smc.c
> > > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > > + * SCMI mediator driver, using SCP as transport.
> > > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > > + * Oleksii Moisieiev <oleksii_moisieiev@epam.com>
> > > > > > > > > > > > > > > + * Copyright (C) 2021, EPAM Systems.
> > > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > > + * This program is free software; you can redistribute it and/or modify
> > > > > > > > > > > > > > > + * it under the terms of the GNU General Public License as published by
> > > > > > > > > > > > > > > + * the Free Software Foundation; either version 2 of the License, or
> > > > > > > > > > > > > > > + * (at your option) any later version.
> > > > > > > > > > > > > > > + *
> > > > > > > > > > > > > > > + * This program is distributed in the hope that it will be useful,
> > > > > > > > > > > > > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > > > > > > > > > > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > > > > > > > > > > > > > + * GNU General Public License for more details.
> > > > > > > > > > > > > > > + */
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +#include <asm/sci/sci.h>
> > > > > > > > > > > > > > > +#include <asm/smccc.h>
> > > > > > > > > > > > > > > +#include <asm/io.h>
> > > > > > > > > > > > > > > +#include <xen/bitops.h>
> > > > > > > > > > > > > > > +#include <xen/config.h>
> > > > > > > > > > > > > > > +#include <xen/sched.h>
> > > > > > > > > > > > > > > +#include <xen/device_tree.h>
> > > > > > > > > > > > > > > +#include <xen/iocap.h>
> > > > > > > > > > > > > > > +#include <xen/init.h>
> > > > > > > > > > > > > > > +#include <xen/err.h>
> > > > > > > > > > > > > > > +#include <xen/lib.h>
> > > > > > > > > > > > > > > +#include <xen/list.h>
> > > > > > > > > > > > > > > +#include <xen/mm.h>
> > > > > > > > > > > > > > > +#include <xen/string.h>
> > > > > > > > > > > > > > > +#include <xen/time.h>
> > > > > > > > > > > > > > > +#include <xen/vmap.h>
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL 0x10
> > > > > > > > > > > > > > > +#define SCMI_BASE_PROTOCOL_ATTIBUTES 0x1
> > > > > > > > > > > > > > > +#define SCMI_BASE_SET_DEVICE_PERMISSIONS 0x9
> > > > > > > > > > > > > > > +#define SCMI_BASE_RESET_AGENT_CONFIGURATION 0xB
> > > > > > > > > > > > > > > +#define SCMI_BASE_DISCOVER_AGENT 0x7
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +/* SCMI return codes. See section 4.1.4 of SCMI spec (DEN0056C) */
> > > > > > > > > > > > > > > +#define SCMI_SUCCESS 0
> > > > > > > > > > > > > > > +#define SCMI_NOT_SUPPORTED (-1)
> > > > > > > > > > > > > > > +#define SCMI_INVALID_PARAMETERS (-2)
> > > > > > > > > > > > > > > +#define SCMI_DENIED (-3)
> > > > > > > > > > > > > > > +#define SCMI_NOT_FOUND (-4)
> > > > > > > > > > > > > > > +#define SCMI_OUT_OF_RANGE (-5)
> > > > > > > > > > > > > > > +#define SCMI_BUSY (-6)
> > > > > > > > > > > > > > > +#define SCMI_COMMS_ERROR (-7)
> > > > > > > > > > > > > > > +#define SCMI_GENERIC_ERROR (-8)
> > > > > > > > > > > > > > > +#define SCMI_HARDWARE_ERROR (-9)
> > > > > > > > > > > > > > > +#define SCMI_PROTOCOL_ERROR (-10)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +#define DT_MATCH_SCMI_SMC DT_MATCH_COMPATIBLE("arm,scmi-smc")
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +#define SCMI_SMC_ID "arm,smc-id"
> > > > > > > > > > > > > > > +#define SCMI_SHARED_MEMORY "linux,scmi_mem"
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I could find the following SCMI binding in Linux, which describes
> > > > > > > > > > > > > > the arm,scmi-smc compatible and the arm,smc-id property:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > However, linux,scmi_mem is not described. Aren't you supposed to read
> > > > > > > > > > > > > > the "shmem" property instead? And the compatible string used for this
> > > > > > > > > > > > > > seems to be "arm,scmi-shmem".
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > We use linux,scmi_mem node to reserve memory, needed for all
> > > > > > > > > > > > > channels:
> > > > > > > > > > > > >
> > > > > > > > > > > > > reserved-memory {
> > > > > > > > > > > > > /* reserved region for scmi channels*/
> > > > > > > > > > > > > scmi_memory: linux,scmi_mem@53FF0000 {
> > > > > > > > > > > > > no-map;
> > > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > > > > > > > };
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > > arm,scmi-shmem node used in shmem property defines only 1 page needed to
> > > > > > > > > > > > > the current scmi channel:
> > > > > > > > > > > > >
> > > > > > > > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > > > };
> > > > > > > > > > > > >
> > > > > > > > > > > > > For each Domain reg points to unigue page from linux,scmi_mem region,
> > > > > > > > > > > > > assigned to this agent.
> > > > > > > > > > > >
> > > > > > > > > > > > If we were to use "linux,scmi_mem" we would have to introduce it as a
> > > > > > > > > > > > compatible string, not as a node name, and it would need to be described
> > > > > > > > > > > > in Documentation/devicetree/bindings/firmware/arm,scmi.yaml.
> > > > > > > > > > > >
> > > > > > > > > > > > But from your description I don't think it is necessary. We can just use
> > > > > > > > > > > > "arm,scmi-shmem" to describe all the required regions:
> > > > > > > > > > > >
> > > > > > > > > > > > reserved-memory {
> > > > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > ...
> > > > > > > > > > > >
> > > > > > > > > > > > In other words, if all the individual channel pages are described as
> > > > > > > > > > > > "arm,scmi-shmem", why do we also need a single larger region as
> > > > > > > > > > > > "linux,scmi_mem"?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > That was my first implementation. But I've met a problem with
> > > > > > > > > > > scmi driver in kernel. I don't remember the exact place, but I remember
> > > > > > > > > > > there were some if, checking if memory weren't reserved.
> > > > > > > > > > > That's why I ended up splitting nodes reserved memory region and actual
> > > > > > > > > > > shmem page.
> > > > > > > > > > > For linux,scmi_mem node I took format from /reserved-memory/linux,lossy_decompress@54000000,
> > > > > > > > > > > which has no compatible string and provides no-map property.
> > > > > > > > > > > linux,scmi_shmem node is needed to prevent xen from allocating this
> > > > > > > > > > > space for the domain.
> > > > > > > > > > >
> > > > > > > > > > > Very interesting question about should I introduce linux,scmi_mem node
> > > > > > > > > > > and scmi_devid property to the
> > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml?
> > > > > > > > > > > Those node and property are needed only for Xen and useless for
> > > > > > > > > > > non-virtualized systems. I can add this node and property description to
> > > > > > > > > > > arm,scmi.yaml, but leave a note that this is Xen specific params.
> > > > > > > > > > > What do you think about it?
> > > > > > > > > >
> > > > > > > > > > Reply below
> > > > > > > > > >
> > > > > > > > > > [...]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > In general we can't use properties that are not part of the device tree
> > > > > > > > > > > > spec, either https://urldefense.com/v3/__https://www.devicetree.org/specifications/__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXCSnJPN$ [devicetree[.]org] or
> > > > > > > > > > > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings__;!!GF_29dbcQIUBPA!kNodtgmOQBc1iO76_6vTK-O1SoLxee_ChowYQiQYC595rMOsrnmof2zmk7BnhXloYUaj$ [git[.]kernel[.]org]
> > > > > > > > > > > >
> > > > > > > > > > > > "linux,scmi_mem" is currently absent. Are you aware of any upstreaming
> > > > > > > > > > > > activities to get "linux,scmi_mem" upstream under
> > > > > > > > > > > > Documentation/devicetree/bindings in Linux?
> > > > > > > > > > > >
> > > > > > > > > > > > If "linux,scmi_mem" is going upstream in Linux, then we could use it.
> > > > > > > > > > > > Otherwise, first "linux,scmi_mem" needs to be added somewhere under
> > > > > > > > > > > > Documentation/devicetree/bindings (probably
> > > > > > > > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml), then we can
> > > > > > > > > > > > work on the Xen code that makes use of it.
> > > > > > > > > > > >
> > > > > > > > > > > > Does it make sense?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes I agree. I think linux,scmi_mem and scmi_devid should be upstreamed.
> > > > > > > > > > > I will add those properties to arm,scmi.yaml, mark them as related to XEN and send patch.
> > > > > > > > > >
> > > > > > > > > > I didn't realize that linux,scmi_mem and scmi_devid are supposed to be
> > > > > > > > > > Xen specific. In general, it would be best not to introduce Xen specific
> > > > > > > > > > properties into generic bindings. It is a problem both from a
> > > > > > > > > > specification perspective (because it has hard to handle Xen specific
> > > > > > > > > > cases in fully generic bindings, especially as those bindings are
> > > > > > > > > > maintained as part of the Linux kernel) and from a user perspective
> > > > > > > > > > (because now the user has to deal with a Xen-specific dtb, or has to
> > > > > > > > > > modify the host dtb to add Xen-specific information by hand.)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Let me start from scmi_devid. Why would scmi_devid be Xen-specific? It
> > > > > > > > > > looks like a generic property that should be needed for the Linux SCMI
> > > > > > > > > > driver too. Why the Linux driver doesn't need it?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > scmi_devid used during domain build. It passed as input parameter for SCMI_BASE_SET_DEVICE_PERMISSIONS message.
> > > > > > > > > On non-virtualized systems - there is no need of this call, because OS
> > > > > > > > > is the only one entity, running on the system.
> > > > > > > >
> > > > > > > > OK. Even if it is only required for virtualized systems, I think that
> > > > > > > > scmi_devid is important enough that should be part of the upstream
> > > > > > > > binding. I think it is worth starting an email thread on the LKML with
> > > > > > > > Rob Herring and the SCMI maintainers to discuss the addition of
> > > > > > > > scmi_devid to the binding.
> > > > > > > >
> > > > > > > >
> > > > > > > > > I've chatted with Volodymyr_Babchuk and he gave a great idea to add a
> > > > > > > > > list of device_ids to dom.cfg, such as:
> > > > > > > > > sci_devs = [ 0, 1, 15, 35 ];
> > > > > > > > >
> > > > > > > > > Using this approach, we can remove scmi_devid from the device tree and
> > > > > > > > > just pass a list of scmi_devids to XEN using additional hypercall.
> > > > > > > > > We can probably make hypercall taking devid list as input parameter.
> > > > > > > > > This will take only 1 hypercall to setup sci permissions.
> > > > > > > >
> > > > > > > > But how would a user know which are the right SCMI IDs to add to the
> > > > > > > > sci_devs list? Would the user have to go and read the reference manual
> > > > > > > > of the platform to find the SCMI IDs and then write sci_devs by hand?
> > > > > > > > If that is the case, then I think that it would be better to add
> > > > > > > > scmi_devid to device tree.
> > > > > > > >
> > > > > > > > In general, I think this configuration should happen automatically
> > > > > > > > without user intervention. The user should just specify "enable SCMI"
> > > > > > > > and it should work.
> > > > > > > >
> > > > > > > >
> > > > > > > > > > In regards to linux,scmi_mem, I think it would be best to do without it
> > > > > > > > > > and fix the Linux SCMI driver if we need to do so. Xen should be able to
> > > > > > > > > > parse the native "arm,scmi-shmem" nodes and Linux (dom0 or domU) should
> > > > > > > > > > be able to parse the "arm,scmi-shmem" nodes generated by Xen. Either
> > > > > > > > > > way, I don't think we should need linux,scmi_mem.
> > > > > > > > >
> > > > > > > > > This requires further investigation. I will try to make implementation
> > > > > > > > > without linux,scmi_mem, using only arm,scmi-shmem nodes and share
> > > > > > > > > reuslts with you.
> > > > > > > >
> > > > > > > > OK, thanks.
> > > > > > >
> > > > > > > Hi Stefano,
> > > > > > >
> > > > > > > As I did some investigation about using reserved-memory area
> > > > > > > linux,scmi_mem and now I need your advice.
> > > > > > >
> > > > > > > I see 2 possible implementations for now:
> > > > > > > 1) Add memory-region parameter to cpu_scp_shm node which points to the
> > > > > > > reserved memory region.
> > > > > > > So device-tree will look like this:
> > > > > > >
> > > > > > > reserved-memory {
> > > > > > > /* reserved region for scmi channels*/
> > > > > > > scmi_memory: region@53FF0000{
> > > > > > > no-map;
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > > > > > };
> > > > > > > };
> > > > > > > cpu_scp_shm: scp-shmem@0x53FF0000 {
> > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > memory-region = <&scmi_memory>;
> > > > > > > };
> > > > > > >
> > > > > > > So cpu_scp_shm node has a reference to scmi_memory region. This mean
> > > > > > > that xen can find reserved memory region without adding additional names
> > > > > > > to the device-tree bindings.
> > > > > > > memory-region parameter as a reference to reserved memory and region
> > > > > > > creation described in:
> > > > > > > https://urldefense.com/v3/__https://github.com/torvalds/linux/blob/v5.15/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt__;!!GF_29dbcQIUBPA!k6x19x1gYF1CPlgAZj7std3ifqhq-9DXvuF0nwonNPUwMzZpYHYbrRJziJrgdFIOjyan$ [github[.]com]
> > > > > > >
> > > > > > > This approach I've implemented already and it works.
> > > > > >
> > > > > > This approach would require a discussion with the upstream device tree
> > > > > > maintainers. Likely, we would need to add a note about the usage of the
> > > > > > "memory-region" property to arm,scmi.yaml.
> > > > > >
> > > > > > Also, I have the feeling that they would ask to add the "memory-region"
> > > > > > property directly to the "arm,scmi-smc" node, as an alternative (or
> > > > > > in addition) to the existing "shmem" property.
> > > > > >
> > > > > > That said, from my point of view this approach is also a viable option.
> > > > > > I don't see any major problems.
> > > > > >
> > > > > > The main question (after reading everything else that you wrote below)
> > > > > > is whether the "arm,scmi-smc" node in this case could be automatically
> > > > > > generated.
> > > > > >
> > > > >
> > > > > arm,scmi-smc node can be generated in both cases. I think I'd leave it
> > > > > as backup in case if the second approach will not work.
> > > > >
> > > > > >
> > > > > > > 2) The second approach is the format you suggested:
> > > > > > > > > > > > reserved-memory {
> > > > > > > > > > > > scp-shmem@0x53FF0000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF0000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > scp-shmem@0x53FF1000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF1000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > scp-shmem@0x53FF2000 {
> > > > > > > > > > > > compatible = "arm,scmi-shmem";
> > > > > > > > > > > > reg = <0x0 0x53FF2000 0x0 0x1000>;
> > > > > > > > > > > > };
> > > > > > > > > > > > ...
> > > > > > >
> > > > > > > This approach has an advantage that xen ARM_SCI driver do not know about
> > > > > > > how channels are placed in the reserved memory, but introduces some
> > > > > > > disadvantages:
> > > > > > > a) We provide extra 14 (in our case) arm,scmi-shmem nodes which are not used
> > > > > > > in the device-tree. In current implementation I have separate scmi.dtsi
> > > > > > > file which introduces scmi support for both XEN-based and
> > > > > > > non-virtualized systems. Having 14 extra channels in the device-tree may
> > > > > > > be confusing.
> > > > > >
> > > > > > I can see that while it would be ideal for Xen to see all 14+1 channels
> > > > > > in device tree (on the host device tree), we wouldn't want to expose all
> > > > > > of them to the domains, not even to dom0.
> > > > > >
> > > > > > How many channels do we want dom0 to see by the way? For this
> > > > > > discussion, I'll just assume for now that dom0 only sees 1 channel like
> > > > > > the domUs.
> > > > >
> > > > > For dom0 we need only one channel.
> > > > >
> > > > > >
> > > > > > Now we have a problem: how do we go about "filtering" the
> > > > > > "arm,scmi-shmem" device tree nodes? Which is also what you are asking
> > > > > > below in point b).
> > > > > >
> > > > >
> > > > > Xen will not need to filter "arm,scmi-shmem" node. It will just
> > > > > create shmem node in Domain device-tree. I don't see any problem for xen
> > > > > configuration here.
> > > > > What bothers me here is that I set scmi configuration for platform dts,
> > > > > not for xen or domu dts files.
> > > > > So for example I have the following structure of the dts files for my
> > > > > platform (r8a77961-salvator-xs):
> > > > > * r8a77961-scmi.dtsi - this file includes all scmi related nodes and set
> > > > > scmi_devid for the devices, that should use scmi.
> > > > > * r8a77961-salvator-xs.dts - dts file which generates dtb for the platform.
> > > > > It includes r8a77961-scmi.dtsi so I populate scmi to platform dtb, which
> > > > > is used for system with no hypervisor.
> > > > > * r8a77961-salvator-xs-xen.dts - dts file for xen which includes
> > > > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > > > > * r8a77961-salvator-xs-domu.dts - dts file for DomU which includes
> > > > > r8a77961-salvator-xs.dts and inherits scmi configuration from it.
> > > > >
> > > > > In this case r8a77961-salvator-xs.dtb r8a77961-salvator-xs-xen.dtb
> > > > > r8a77961-salvator-xs-domu.dtb files will inherit 14+1 channel.
> > > > >
> > > > > I can give you a link to Merge request with this changes if you need it.
> > > > >
> > > > > For xen and domu dtb it is not a problem because all "arm,scmi-shmem"
> > > > > nodes will be omitted and new will be generated for the domains.
> > > > >
> > > > > What bothers me is that r8a77961-salvator-xs.dtb will have 14 unused channels.
> > > > >
> > > > > Just got an idea while writing this: I can create only one
> > > > > "arm,scmi-shmem" node in r8a77961-scmi.dtsi and add 14 more nodes,
> > > > > needed for xen explicitly in r8a77961-salvator-xs-xen.dts.
> > > > >
> > > > > Then we will have valid configurations for all cases.
> > > > > This can be a solution. What do you think?
> > > >
> > > > It is good that you brought this up because it helps me explain what I
> > > > mean. And of course it is up to you where you place the nodes in the
> > > > various dts files at your disposal. Either way it would work but I think
> > > > they should belong to r8a77961-salvator-xs.dts.
> > > >
> > > > Generally the platform vendor (e.g. Xilinx) provides a device tree
> > > > description of the platform to use including all the available resources
> > > > and firmware interfaces. In your case it would be r8a77961-scmi.dtsi +
> > > > r8a77961-salvator-xs.dts. This is what I call the "host device tree"
> > > > below. Users should be able to boot a fully functional system using the
> > > > host device tree pretty much "as is" to run Xen, Linux or any other
> > > > software.
> > > >
> > > > Certainly the SCMI device tree description should be part of the host
> > > > device tree, so in your case it would be r8a77961-salvator-xs.dts. And
> > > > the description should include all 14+1 channels because this is the
> > > > generic platform description -- we cannot know for sure how the users
> > > > are going to use the system.
> > > >
> > > > This is why r8a77961-salvator-xs-xen.dts should be as small as possible
> > > > or ideally inexistent. There shouldn't be a need for a special device
> > > > tree modification to allow Xen to run. In reality, even at Xilinx we
> > > > have something like r8a77961-salvator-xs-xen.dts, although it is really
> > > > small.
> > > >
> > > > But I see that r8a77961-salvator-xs-xen.dts could be viewed as the
> > > > device tree additions to run hypervisors and from that point of view it
> > > > is more acceptable to place the 14 channels there.
> > > >
> > > > The biggest problem is r8a77961-salvator-xs-domu.dts: who is going to
> > > > write it? And how? It wouldn't be provided by the platform vendor, so it
> > > > is the user the one that has to find a way to write it.
> > > >
> > > > I know the user already has to write a partial DTB for device
> > > > assignment, but any time the process is more complex than "copy the host
> > > > device tree node for device XXX to the partial DTB" it is a problem.
> > > > Errors are made and the system doesn't work.
> > > >
> > > > I think we don't want to make it even more difficult by having to
> > > > manually produce the SCMI domU description too. The SCMI description for
> > > > domU could be automatically generated by Xen, or libxl/xl. If that's an
> > > > issue, then the SCMI description could be automatically generated by an
> > > > external tool but I think it would make things more complex and harder
> > > > to maintain.
> > > >
> > > > In short my point of view is:
> > > > - r8a77961-scmi.dtsi + r8a77961-salvator-xs.dts should be as generic as
> > > > possible so the SCMI nodes should have 14+1 channels
> > > > - but putting the 14 channels in r8a77961-salvator-xs-xen.dts is still
> > > > OKish
> > > > - it is important that r8a77961-salvator-xs-domu.dts is automatically
> > > > generated by Xen or libxl or another software tool
> > > >
> > >
> > > Thank you for the detailed response. I'll put all 14+1 channels to
> > > r8a77961-salvator-xs.dts then.
> > > I've described my thoughts about generation of the arm,scmi-smc node below.
> > >
> > > >
> > > > > > > b) In case if we have all 15 channels, described in partial device-tree,
> > > > > >
> > > > > > I think you meant "described in the host device tree", right?
> > > > > >
> > > > > Yeah that's what I've meant.
> > > > > >
> > > > > > > we should not copy any node to the domain device-tree. I think it will
> > > > > > > be better to generate arm,scmi-shmem node in the Domain device-tree.
> > > > > >
> > > > > > Yes, I think it makes sense for Xen to generate the "arm,scmi-shmem"
> > > > > > device tree description for the DomU/Dom0 based on the channels
> > > > > > allocated to the domain.
> > > > > >
> > > > > >
> > > > > > > The problem is that arm,scmi-smc node, which is using arm,scmi-shmem
> > > > > > > node can't be generated. I prefer it to be copied from the partial
> > > > > > > device-tree because it includes some platform specific configuration,
> > > > > > > such as func-id and list of the protocols (for example different
> > > > > > > platforms may require different list of the protocols). So in this
> > > > > > > case we will have 1 node copied and 1 node generated.
> > > > > > >
> > > > > > > I think even for dom0less we should use arm,scmi-smc node from the
> > > > > > > device-tree because protocol configuration and funcid is related to the
> > > > > > > platform.
> > > > > >
> > > > > > I am not sure I understood what you wrote. You are saying that the
> > > > > > "arm,scmi-smc" node includes some platform specific configurations so
> > > > > > it cannot be automatically generated by Xen (or by the tools) and
> > > > > > instead it needs to be manually provided as part of the partial dtb for
> > > > > > the domU. Is that correct?
> > > > > >
> > > > > > If so, I would like to understand the reasons behind it. Manual
> > > > > > device tree editing is problematic.
> > > > > >
> > > > > > I looked for "func-id" in
> > > > > > Documentation/devicetree/bindings/firmware/arm,scmi.yaml but couldn't
> > > > > > find any results. Do you have an example of the platform specific
> > > > > > configuration or protocol configuration that would make it difficult to
> > > > > > automatically generate the "arm,scmi-smc" node for the domains?
> > > > >
> > > > > Sorry, I used wrong term (used term from the specification), arm,smc-id
> > > > > of cause.
> > > > >
> > > > > >
> > > > > > Also, is this a problem just for approach #2 or also for approach #1?
> > > > > > If it is a problem only for approach #2, then let's just go with
> > > > > > approach #1.
> > > > > >
> > > > >
> > > > > We can't copy "arm,scmi-smc" in both approaches. The difference is that
> > > > > in the first approach we can copy both "arm,scmi-smc" and
> > > > > "arm,scmi-shmem" nodes while in the second approach we should copy
> > > > > "arm,scmi-smc", but we have to generate "arm,scmi-shmem" node.
> > > > >
> > > > > arm,scmi-smc node can't be generated because it includes properties and
> > > > > configurations that depends from platform and should be get from the
> > > > > device tree.
> > > > > Here is "arm,scmi-smc" node expample:
> > > > > firmware {
> > > > > scmi {
> > > > > compatible = "arm,scmi-smc"
> > > > > arm,smc-id = <0x82000002>;
> > > > > shmem = <&cpu_scp_shm>;
> > > > > #address-cells = <1>;
> > > > > #size-cells = <0>;
> > > > > scmi_power: protocol@11 {
> > > > > reg = <0x11>;
> > > > > #power-domain-cells = <1>;
> > > > > };
> > > > >
> > > > > scmi_clock: protocol@14 {
> > > > > ...
> > > > > };
> > > > >
> > > > > scmi_reset: protocol@16 {
> > > > > ...
> > > > > };
> > > > > ...
> > > > > };
> > > > > };
> > > > >
> > > > > It has 3 configurable options:
> > > > > * arm,smc-id parameter, setting func_id for scmi protocol. This id can be
> > > > > different for different platforms.
> > > > > For example stm32mp1 architecture use different scm-id for different
> > > > > agents:
> > > > > https://urldefense.com/v3/__https://github.com/ARM-software/arm-trusted-firmware/blob/0586c41b3f2d52aae847b7212e7b0c7e19197ea2/plat/st/stm32mp1/include/stm32mp1_smc.h*L39__;Iw!!GF_29dbcQIUBPA!mTRUjtSg19iVaYo3Cgjop5ckPWqKsHVo1EZCEA1zCbod9KpNSXX291A8vAuNdTCr46MA$ [github[.]com]
> > > > >
> > > > > * shmem which includes phandle to arm,scmi-shmem node. But this is not
> > > > > a problem and can be updated.
> > > > >
> > > > > * list of the protocol subnodes. This is also configurable parameter,
> > > > > not regs or names, but the number of the protocols. For example onle
> > > > > platform can use power-domains/clock/resets via scmi, when another will
> > > > > require volage-control and sensor-management to be added.
> > > > >
> > > > > Xen should know this parameters to be able to generate "arm,scmi-smc" node.
> > > > >
> > > > > Also we're currently discussing new scmi protocol with ARM: Pinctrl over
> > > > > SCMI.
> > > > >
> > > > > It should allow domains to access pinctrl subsystem, placed in Firmware
> > > > > through SCMI protocol.
> > > > > scmi_pinctrl node will look like this:
> > > > >
> > > > > firmware {
> > > > > scmi {
> > > > > ...
> > > > > scmi_pinctrl: protocol@18 {
> > > > > reg = <0x18>;
> > > > > #pinctrl-cells = <0>;
> > > > >
> > > > > i2c2_pins: i2c2 {
> > > > > groups = <74>; /* i2c2_a */
> > > > > function = <15>; /* i2c2 */
> > > > > };
> > > > >
> > > > > irq0_pins: irq0 {
> > > > > groups = <81>; /* intc_ex_irq0 */
> > > > > function = <19>; /* intc_ex */
> > > > > };
> > > > >
> > > > > avb_pins: avb {
> > > > > mux {
> > > > > /* avb_link, avb_mdio, avb_mii */
> > > > > groups = <17>, <21>, <22>;
> > > > > function = <1>; /* avb */
> > > > > };
> > > > >
> > > > > pins_mdio {
> > > > > groups = <21>; /* avb_mdio */
> > > > > drive-strength = <24>;
> > > > > };
> > > > >
> > > > > pins_mii_tx {
> > > > > /* PIN_AVB_TX_CTL, PIN_AVB_TXC, PIN_AVB_TD0,
> > > > > PIN_AVB_TD1, PIN_AVB_TD2, PIN_AVB_TD3 */
> > > > > pins = <242>, <240>, <236>, <237>, <238>, <239>;
> > > > > drive-strength = <12>;
> > > > > };
> > > > > };
> > > > > ...
> > > > > };
> > > > > };
> > > > > };
> > > > >
> > > > > So "arm,scmi-smc" node will have even more platform specific settings.
> > > > >
> > > > > >
> > > > > > > I prefer the second approach and will try to make it if it's OK to copy
> > > > > > > arm,scmi-smc node from partial Device-tree and generate arm,scmi-shmem
> > > > > > > node.
> > > > > > >
> > > > > > > What do you think about that?
> > > > > >
> > > > > > From a device tree specification perspective, I think both approaches
> > > > > > are OK (with a minor comment on the first approach as I wrote above.)
> > > > > >
> > > > > > But from a Xen perspective I think it is important that we don't require
> > > > > > the user to manually provide the SCMI configuration in the partial DTB.
> > > > > > It would be better if we could generate it automatically from Xen or the
> > > > > > tools (or even an independent script). Or copy the "arm,scmi-smc" node
> > > > > > from the host device tree to the domU device tree without modifications.
> > > > >
> > > > > I think copy "arm,scmi-smc" node is the only option we have.
> > > > > I'm not sure what do you mean under "host device tree" if you mean Xen
> > > > > device-tree - then I think it will not cover the case with stm32mp1 I've
> > > > > mentioned above. I think it will be better to copy "arm,scmi-smc" node
> > > > > from Domu partial Device-tree to Domu device-tree.
> > > > > So AGENT0 smc-id will be set in xen device-tree and copied to dom0 and
> > > > > AGENT1 scm-is set in domu device-tree and copied to dom-u.
> > > > >
> > > > > Do you agree with my points?
> > > >
> > > > I think we are saying similar things, but we are getting mixed up with
> > > > the terminology. Let's start from the basics :-)
> > > >
> > > > # Host device tree
> > > > The device tree given to Xen at boot time. This is the device tree that
> > > > Xen parses to discover what's available on the platform. In your case,
> > > > it seems to include r8a77961-salvator-xs-xen.dts.
> > > >
> > > > # Partial DTB
> > > > (Ignoring Dom0less) this is the small DTB that gets passed to xl with
> > > > the "device_tree" option in the xl config file. It is copied verbatim
> > > > to the domU device tree by xl/libxl.
> > > >
> > > > # Copy the "arm,scmi-smc" node from host device tree
> > > > This means that the domU "arm,scmi-smc" node is an exact copy of the
> > > > host device tree SCMI node. I don't think this is actually possible in
> > > > most cases because the domU description is typically a bit different
> > > > from the host description. For instance, the host description could
> > > > include 14+1 channels while the domU description should only include 1
> > > > channel.
> > > >
> > > > # Copy the "arm,scmi-smc" node from the partial DTB
> > > > This implies that somebody or something create an "arm,scmi-smc" node
> > > > for the domU and placed it into the partial DTB. Then, Xen and/or
> > > > xl/libxl will copy the node from the partial DTB to the DomU device
> > > > tree. The main question in this case is: who is going to write the
> > > > partial DTB? We dont want the user (i.e. a person) to have to manually
> > > > write the SCMI description for the domU. It should be an automated tools
> > > > that does it. At that point, it is easier if it is Xen or xl/libxl.
> > > > Alternativaly, we could think of an external tool but I think it would
> > > > make things more difficult to maintain.
> > > >
> > > > # Generate the "arm,scmi-smc" node for domUs
> > > > When I write "generate the arm,scmi-smc node", I mean that Xen and
> > > > libxl/xl will generate the "arm,scmi-smc" node for the domU. Thus, the
> > > > node will not be copied from the partial DTB or from the device tree,
> > > > instead, it should be created directly by Xen and/or libxl/xl.
> > > >
> > > > However, the domU "arm,scmi-smc" node could still be derived from the
> > > > host device tree "arm,scmi-smc" node. In other words, Xen or xl/libxl
> > > > would look at the host device tree "arm,scmi-smc" node, copy it to the
> > > > domU device tree while making as many changes as necessary.
> > > >
> > > > The DomU "arm,scmi-smc" node doesn't have to be entirely fake and
> > > > static. It could be dynamically created to match the host device tree
> > > > description. I think this is the best option.
> > > >
> > > >
> > > > # Conclusion
> > > > I am suggesting that Xen and/or libxl automatically produce the
> > > > "arm,scmi-smc" node for domUs based on the host device tree description
> > > > and based on the channel mapped to the domU. This way, the user (a
> > > > person) doesn't have to go and manually edit the domU partial DTB.
> > > >
> > >
> > > That sounds reasonable. The problem is that arm,scmi-smc node can be
> > > copmlicated and include a lot of configuration. Also for different
> > > mediators this node can be different.
> > > As I inderstand, there is no mechanism for xl to access host device-tree
> > > right now. Correct me if I'm wrong.
> >
> > Yes, you are right. And the lack of a mechanism for xl to access the
> > host device tree is a problem.
> >
> >
> > > I see the following way we can generate arm,scmi-smc node for DomU:
> > > We say that if scmi-smc mediator is enabled - then Dom0 is configured to
> > > use SCMI. This means that Dom0 device-tree will have arm,scmi-smc node
> > > and it can be reached from the userspace.
> >
> > So far so good
> >
> >
> > > In this case xl can use infromation from /proc/device-tree/firmware/scmi
> > > to generate arm,scmi-smc node for DomU. But in this case xl should know
> > > the exact path of scmi node.
> > >
> > > Or we can generate some special node, called "shared" in Dom0 device-tree
> > > which will include copy of the arm,scmi-smc node, which can be used for domains.
> > > In this case xl can scan /proc/device-tree/shared node and find
> > > arm,scmi-smc copatible node and use it to generate arm,scmi-smc node for
> > > DomU.
> > > Also this can be used for another features in future.
> > >
> > > What do you think about this?
> >
> > Basing the domU SCMI node generation on the dom0 SCMI node is not great,
> > because it should be based on the host device tree rather than dom0 and
> > the dom0 SCMI description will be different.
> >
> > Instead of copying just the host SCMI node somewhere in the dom0 device
> > tree so that xl/libxl can access it via /proc/device-tree, I think it
> > would be better to make the full host DTB available to xl/libxl.
> >
> > There are probably many ways to do this, but there are a couple I can
> > think of on top of my head:
> >
> > - introduce a new hypercall to get the full host dtb from Xen
> > The hypercall would pass an address and a size in guest physical memory
> > and Xen would copy the host DTB to it.
> >
> > - introduce something like /proc/device-tree to hypfs
> > See xen/common/hypfs.c
>
>
> Hi Stefano,
>
> I've found the discussion, indirectly related to our topic [1], in which
> hypervisor <-> toolchain interface topic was raised.
>
> Where Andrii Anisov proposed to introduce another libxl type (f.e. File) which
> will read device tree file and operate with a binary blob.
>
> His idea is to introduce new Domctl hypecall, which is able to send device-tree
> blob from toolstack to the hypervisor.
> I was thinking about using his approach, but in opposite direction.
> Such as introduce a set of DOMCTL, which gives ability for xl user to request
> Host Device-tree nodes.
> I see something like that:
> XEN_DOMCTL_host_node_by_path - gets node path as input and returns blob with node;
> XEN_DOMCTL_find_host_compatible_node - gets compatible string and "skip" number
> for the case if there is more than 1 compatible node and returns blob with node.
>
> Advandages for my sight are:
> - I hope Andrii still has some code, which can be reused during the implementation;
> - Xen can determine if the requesting node can be exposed to the guest, which
> is additional layer of validation;
> - no need to transfer all device-tree to the toolstack.
>
> As for the hypfs this sounds like a separate feature to me. I'm not sure if we
> need to provide the entire copy of the device-tree. Also there could be some
> problems with this solution.
> For example, xl tool have to scan whole hypfs tree to find node by phandle or
> by compatible string.
>
> Looking forward for your thoughts about that. Also I would be happy if Julien
> could share his vision.
>
> [1] https://xen.markmail.org/search/?q=from%3AAndrii+Anisov+device-tree#query:from%3AAndrii%20Anisov%20device-tree+page:2+mid:tvgyp7fzpiyyso3j+state:results
It looks like XEN_DOMCTL_host_node_by_path and
XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
think that a single hypercall that retrieves the entire host DTB would
be easier to implement and more robust in the long term.
hypfs has the advantage that it would create an interface more similar
to the one people are already used to on Linux systems
(/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
which intuitively I think it would be slower. Also the feature might be
harder to implement but I am not sure.
I don't have a strong preference and this is not a stable interface (we
don't have to be extra paranoid about forward and backward
compatibility). So I am fine either way. Let's see what the others think
as well.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-24 19:06 ` Stefano Stabellini
@ 2022-01-24 19:26 ` Julien Grall
2022-01-24 22:14 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Julien Grall @ 2022-01-24 19:26 UTC (permalink / raw)
To: Stefano Stabellini, Oleksii Moisieiev
Cc: xen-devel, Volodymyr Babchuk, Bertrand Marquis
Hi,
On 24/01/2022 19:06, Stefano Stabellini wrote:
> It looks like XEN_DOMCTL_host_node_by_path and
> XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> think that a single hypercall that retrieves the entire host DTB would
> be easier to implement
DOMCTL should only be used to handle per-domain information. If we want
to create a new sub-hypercall of either __HYPERVISOR_platform_op or
__HYPERVISOR_sysctl_op (not sure which one).
AFAICT, both are versioned.
> and more robust in the long term. >
> hypfs has the advantage that it would create an interface more similar
> to the one people are already used to on Linux systems
> (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> which intuitively I think it would be slower.
Even if you have the binary blob, you would still have to scan the
device-tree. That said, it is probably going to be potentially a bit
faster because you have less hypercall.
However, here this is a trade-off between memory use and speed. If you
want speed, then you may have to transfer up to 2MB every time. So the
question is do we care more about speed or memory usage?
> Also the feature might be
> harder to implement but I am not sure.
>
> I don't have a strong preference and this is not a stable interface (we
> don't have to be extra paranoid about forward and backward
> compatibility). So I am fine either way. Let's see what the others think
> as well.
My preference would be to use hypfs as this is cleaner than exposing a blob.
However, are we sure we can simply copy the content of the host
Device-Tree to the guest Device-Tree for SCMI? For instance, I know that
for device passthrough there are some property that needs to be altered
for some devices. Hence, why it is not present. Although, I vaguely
recalled to have written a PoC, not sure if it was posted on the ML.
Cheeers,
--
Julien Grall
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-24 19:26 ` Julien Grall
@ 2022-01-24 22:14 ` Stefano Stabellini
2022-01-25 14:35 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-24 22:14 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Oleksii Moisieiev, xen-devel,
Volodymyr Babchuk, Bertrand Marquis
On Mon, 24 Jan 2022, Julien Grall wrote:
> On 24/01/2022 19:06, Stefano Stabellini wrote:
> > It looks like XEN_DOMCTL_host_node_by_path and
> > XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> > think that a single hypercall that retrieves the entire host DTB would
> > be easier to implement
>
> DOMCTL should only be used to handle per-domain information. If we want to
> create a new sub-hypercall of either __HYPERVISOR_platform_op or
> __HYPERVISOR_sysctl_op (not sure which one).
>
> AFAICT, both are versioned.
>
> > and more robust in the long term. >
> > hypfs has the advantage that it would create an interface more similar
> > to the one people are already used to on Linux systems
> > (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> > which intuitively I think it would be slower.
>
> Even if you have the binary blob, you would still have to scan the
> device-tree. That said, it is probably going to be potentially a bit faster
> because you have less hypercall.
>
> However, here this is a trade-off between memory use and speed. If you want
> speed, then you may have to transfer up to 2MB every time. So the question is
> do we care more about speed or memory usage?
>
> > Also the feature might be
> > harder to implement but I am not sure.
> >
> > I don't have a strong preference and this is not a stable interface (we
> > don't have to be extra paranoid about forward and backward
> > compatibility). So I am fine either way. Let's see what the others think
> > as well.
>
> My preference would be to use hypfs as this is cleaner than exposing a blob.
That's also fine by me. Probably the hypfs implementation shouldn't be
much more difficult than something like
XEN_DOMCTL_host_node_by_path/XEN_DOMCTL_find_host_compatible_node.
> However, are we sure we can simply copy the content of the host Device-Tree to
> the guest Device-Tree for SCMI? For instance, I know that for device
> passthrough there are some property that needs to be altered for some devices.
> Hence, why it is not present. Although, I vaguely recalled to have written a
> PoC, not sure if it was posted on the ML.
The SCMI node cannot be copied "as is" from host to guest. It needs a
couple of changes but they seem feasible as they are limited to the
channels exposed to the guest. (The generic device passthrough case is a
lot more difficult.)
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-24 22:14 ` Stefano Stabellini
@ 2022-01-25 14:35 ` Oleksii Moisieiev
2022-01-25 21:19 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-25 14:35 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On Mon, Jan 24, 2022 at 02:14:43PM -0800, Stefano Stabellini wrote:
> On Mon, 24 Jan 2022, Julien Grall wrote:
> > On 24/01/2022 19:06, Stefano Stabellini wrote:
> > > It looks like XEN_DOMCTL_host_node_by_path and
> > > XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> > > think that a single hypercall that retrieves the entire host DTB would
> > > be easier to implement
> >
> > DOMCTL should only be used to handle per-domain information. If we want to
> > create a new sub-hypercall of either __HYPERVISOR_platform_op or
> > __HYPERVISOR_sysctl_op (not sure which one).
> >
> > AFAICT, both are versioned.
> >
> > > and more robust in the long term. >
> > > hypfs has the advantage that it would create an interface more similar
> > > to the one people are already used to on Linux systems
> > > (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> > > which intuitively I think it would be slower.
> >
> > Even if you have the binary blob, you would still have to scan the
> > device-tree. That said, it is probably going to be potentially a bit faster
> > because you have less hypercall.
> >
> > However, here this is a trade-off between memory use and speed. If you want
> > speed, then you may have to transfer up to 2MB every time. So the question is
> > do we care more about speed or memory usage?
> >
> > > Also the feature might be
> > > harder to implement but I am not sure.
> > >
> > > I don't have a strong preference and this is not a stable interface (we
> > > don't have to be extra paranoid about forward and backward
> > > compatibility). So I am fine either way. Let's see what the others think
> > > as well.
> >
> > My preference would be to use hypfs as this is cleaner than exposing a blob.
>
> That's also fine by me. Probably the hypfs implementation shouldn't be
> much more difficult than something like
> XEN_DOMCTL_host_node_by_path/XEN_DOMCTL_find_host_compatible_node.
>
>
> > However, are we sure we can simply copy the content of the host Device-Tree to
> > the guest Device-Tree for SCMI? For instance, I know that for device
> > passthrough there are some property that needs to be altered for some devices.
> > Hence, why it is not present. Although, I vaguely recalled to have written a
> > PoC, not sure if it was posted on the ML.
>
> The SCMI node cannot be copied "as is" from host to guest. It needs a
> couple of changes but they seem feasible as they are limited to the
> channels exposed to the guest. (The generic device passthrough case is a
> lot more difficult.)
Hi Stefano,
What I'm thinking about is do we actually need to create SCMI node in DomU device-tree?
I have this question is because we don't need SCMI node to be present in DomU
device-tree if it has no passed-through devices, which are using scmi.
So if we don't have passed-through devices or do not provide DomU partial device-tree
in config, then there is no need to create SCMI node.
For now I see the following possible domu configurations:
1) If DomU has a lot of passed-through devices and it's easier to inherit
host device-tree and disable not passed-through devices.
Partial device tree will looks like this:
#include "r8a77961-salvator-xs.dts" //include host device tree
/
{
soc {
...
}
};
// Disable non passed-through devices
&hscif {
status = "disabled";
};
In this case DomU partial device-tree will inherit arm,scmi-smc and
arm,scmi-shmem nodes and all clock/reset/power-domains which are using scmi.
All this nodes can be copied to DomU device-tree from partial device-tree.
2) DomU has few passed-through devices, so it's easier to add the device nodes
to the passthrough node of DomU partial device-tree.
DomU partial device-tree will look like this:
{
scmi_shmem: scp-shmem@0x53FF0000 {
compatible = "arm,scmi-shmem";
reg = <0x0 0x53FF0000 0x0 0x10000>;
};
scmi {
arm,smc-id = <....>;
compatible = "arm,scmi-smc";
shmem = <&scmi_shmem>;
scmi_clock: protocol@14 {
...
};
scmi_reset: protocol@16 {
...
};
};
passthrough {
hscif0: serial@e6540000 {
compatible = "renesas,hscif-r8a77961";
scmi_devid = <5>;
clocks = <&scmi_clock 5>;
resets = <&scmi_reset 5>;
...
};
};
};
As you can see in this case we have to manually copy arm,scmi-shmem and
arm,scmi-smc nodes with hscif0 node or the device-tree compilation will fail.
We can use 0x53FF0000, provided in arm,scmi-shmem node and map domain channel
to this address and copy scmi related nodes to the DomU device-tree.
This is useful when we need to expose only certain protocols to the DomU.
Also it's easy to modify DomU scmi node, as we need for stm32mp1 for example
when different smc-id should be set for DomU.
3) DomU doesn't have any passthrough nodes, which are using scmi.
In this case we don't want SCMI nodes to be in the DomU device-tree.
I see only one use-case when we may need scmi nodes to be generated by xl in
DomU device-tree:
Xen generates psci node to handle cpu_on and cpu_off.
According to the Section 4.3.2.5 of the DEN0056C [1]:
> For these power domains, this protocol can be used to implement PSCI CPU_SUSPEND, CPU_ON, CPU_FREEZE, CPU_DEFAULT_SUSPEND and CPU_OFF functions.
So in theory psci node can use scmi to control cpu state. But this is not our
use-case because we don't want to give DomU ability to stop physical CPU.
Xen can't intercept and handle CPU_ON and CPU_OFF requests when mailbox transport
is used for SCMI communication.
[1] "SCMI Specification DEN0056C," [Online]. Available: https://developer.arm.com/documentation/den0056/latest
Best regards,
Oleksii.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-25 14:35 ` Oleksii Moisieiev
@ 2022-01-25 21:19 ` Stefano Stabellini
2022-01-27 18:11 ` Oleksii Moisieiev
0 siblings, 1 reply; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-25 21:19 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, Julien Grall, xen-devel, Volodymyr Babchuk,
Bertrand Marquis
On Tue, 25 Jan 2022, Oleksii Moisieiev wrote:
> On Mon, Jan 24, 2022 at 02:14:43PM -0800, Stefano Stabellini wrote:
> > On Mon, 24 Jan 2022, Julien Grall wrote:
> > > On 24/01/2022 19:06, Stefano Stabellini wrote:
> > > > It looks like XEN_DOMCTL_host_node_by_path and
> > > > XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> > > > think that a single hypercall that retrieves the entire host DTB would
> > > > be easier to implement
> > >
> > > DOMCTL should only be used to handle per-domain information. If we want to
> > > create a new sub-hypercall of either __HYPERVISOR_platform_op or
> > > __HYPERVISOR_sysctl_op (not sure which one).
> > >
> > > AFAICT, both are versioned.
> > >
> > > > and more robust in the long term. >
> > > > hypfs has the advantage that it would create an interface more similar
> > > > to the one people are already used to on Linux systems
> > > > (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> > > > which intuitively I think it would be slower.
> > >
> > > Even if you have the binary blob, you would still have to scan the
> > > device-tree. That said, it is probably going to be potentially a bit faster
> > > because you have less hypercall.
> > >
> > > However, here this is a trade-off between memory use and speed. If you want
> > > speed, then you may have to transfer up to 2MB every time. So the question is
> > > do we care more about speed or memory usage?
> > >
> > > > Also the feature might be
> > > > harder to implement but I am not sure.
> > > >
> > > > I don't have a strong preference and this is not a stable interface (we
> > > > don't have to be extra paranoid about forward and backward
> > > > compatibility). So I am fine either way. Let's see what the others think
> > > > as well.
> > >
> > > My preference would be to use hypfs as this is cleaner than exposing a blob.
> >
> > That's also fine by me. Probably the hypfs implementation shouldn't be
> > much more difficult than something like
> > XEN_DOMCTL_host_node_by_path/XEN_DOMCTL_find_host_compatible_node.
> >
> >
> > > However, are we sure we can simply copy the content of the host Device-Tree to
> > > the guest Device-Tree for SCMI? For instance, I know that for device
> > > passthrough there are some property that needs to be altered for some devices.
> > > Hence, why it is not present. Although, I vaguely recalled to have written a
> > > PoC, not sure if it was posted on the ML.
> >
> > The SCMI node cannot be copied "as is" from host to guest. It needs a
> > couple of changes but they seem feasible as they are limited to the
> > channels exposed to the guest. (The generic device passthrough case is a
> > lot more difficult.)
>
>
> Hi Stefano,
>
> What I'm thinking about is do we actually need to create SCMI node in DomU device-tree?
> I have this question is because we don't need SCMI node to be present in DomU
> device-tree if it has no passed-through devices, which are using scmi.
> So if we don't have passed-through devices or do not provide DomU partial device-tree
> in config, then there is no need to create SCMI node.
>
> For now I see the following possible domu configurations:
> 1) If DomU has a lot of passed-through devices and it's easier to inherit
> host device-tree and disable not passed-through devices.
> Partial device tree will looks like this:
>
> #include "r8a77961-salvator-xs.dts" //include host device tree
>
> /
> {
> soc {
> ...
> }
>
> };
>
> // Disable non passed-through devices
> &hscif {
> status = "disabled";
> };
>
> In this case DomU partial device-tree will inherit arm,scmi-smc and
> arm,scmi-shmem nodes and all clock/reset/power-domains which are using scmi.
> All this nodes can be copied to DomU device-tree from partial device-tree.
This is an almost dom0 configuration. For this kind of use-cases, I
think it is enough to handle dom0 automatically correctly. I wouldn't
ask for anything more than that.
> 2) DomU has few passed-through devices, so it's easier to add the device nodes
> to the passthrough node of DomU partial device-tree.
> DomU partial device-tree will look like this:
> {
> scmi_shmem: scp-shmem@0x53FF0000 {
> compatible = "arm,scmi-shmem";
> reg = <0x0 0x53FF0000 0x0 0x10000>;
> };
> scmi {
> arm,smc-id = <....>;
> compatible = "arm,scmi-smc";
> shmem = <&scmi_shmem>;
> scmi_clock: protocol@14 {
> ...
> };
> scmi_reset: protocol@16 {
> ...
> };
> };
> passthrough {
> hscif0: serial@e6540000 {
> compatible = "renesas,hscif-r8a77961";
> scmi_devid = <5>;
> clocks = <&scmi_clock 5>;
> resets = <&scmi_reset 5>;
> ...
> };
> };
> };
>
> As you can see in this case we have to manually copy arm,scmi-shmem and
> arm,scmi-smc nodes with hscif0 node or the device-tree compilation will fail.
> We can use 0x53FF0000, provided in arm,scmi-shmem node and map domain channel
> to this address and copy scmi related nodes to the DomU device-tree.
> This is useful when we need to expose only certain protocols to the DomU.
> Also it's easy to modify DomU scmi node, as we need for stm32mp1 for example
> when different smc-id should be set for DomU.
I think this is the most interesting case that should be automated and
not require manual intervention. Let me explain why.
Currently we require partial device trees to be manually written because
there is no easy way to automatically generate them. (I have some ideas
on how to automatically generate partial device trees but that is a
separate discussion.)
Unfortunately, it has become increasingly clear that it is too difficult
for users (even advanced users!) to come up with the appropriate partial
device trees. Thus, I am reluctant to introduce more things that rely on
the user having to manually specify partial device tree information.
This is why I would like the SCMI nodes to be automatically added for
domUs.
Of course, if a user provides the scmi and scmi_shmem nodes in the
partial device tree we could just use them. But ideally we should also
be able to automatically generated them based on the host device tree
nodes, so that the user only needs to provide serial@e6540000 (in your
example, scmi_devid would need to be populated too) and the rest would
be done automatically as we do today for the gic and vuart nodes.
At the same time I don't want to scope-creep this patch series too much
and I don't mean to ask you to add a huge new infrastructure to Xen and
the Xen tools just to get SCMI support in. I would rather have a
not-great automatic generation of the domU SCMI nodes than nothing (e.g.
using your suggested XEN_DOMCTL_host_node_by_path and
XEN_DOMCTL_find_host_compatible_node hypercalls althought they would
need to be platform_op as Julien suggested).
It looks like the generation of scmi_shmem and scmi should be easy
enough that could be handled without difficulty in xl/libxl. But if that
turns out to be too difficult, we could have a small independent
bash/python script that from the host device tree generates the partial
device tree with the SCMI nodes. From Xen point of view we are would
still be using the partial device tree, but the partial device tree
itself would be generated instead of manually written. As this workflow
requires a separate tool I think it is a worse option than the one
above. Still better than nothing though.
> 3) DomU doesn't have any passthrough nodes, which are using scmi.
> In this case we don't want SCMI nodes to be in the DomU device-tree.
>
> I see only one use-case when we may need scmi nodes to be generated by xl in
> DomU device-tree:
> Xen generates psci node to handle cpu_on and cpu_off.
> According to the Section 4.3.2.5 of the DEN0056C [1]:
> > For these power domains, this protocol can be used to implement PSCI CPU_SUSPEND, CPU_ON, CPU_FREEZE, CPU_DEFAULT_SUSPEND and CPU_OFF functions.
>
> So in theory psci node can use scmi to control cpu state. But this is not our
> use-case because we don't want to give DomU ability to stop physical CPU.
> Xen can't intercept and handle CPU_ON and CPU_OFF requests when mailbox transport
> is used for SCMI communication.
>
> [1] "SCMI Specification DEN0056C," [Online]. Available: https://developer.arm.com/documentation/den0056/latest
I agree with you on this one; I am not worried about this case.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-25 21:19 ` Stefano Stabellini
@ 2022-01-27 18:11 ` Oleksii Moisieiev
2022-01-27 21:18 ` Stefano Stabellini
0 siblings, 1 reply; 95+ messages in thread
From: Oleksii Moisieiev @ 2022-01-27 18:11 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, xen-devel, Volodymyr Babchuk, Bertrand Marquis
On Tue, Jan 25, 2022 at 01:19:46PM -0800, Stefano Stabellini wrote:
> On Tue, 25 Jan 2022, Oleksii Moisieiev wrote:
> > On Mon, Jan 24, 2022 at 02:14:43PM -0800, Stefano Stabellini wrote:
> > > On Mon, 24 Jan 2022, Julien Grall wrote:
> > > > On 24/01/2022 19:06, Stefano Stabellini wrote:
> > > > > It looks like XEN_DOMCTL_host_node_by_path and
> > > > > XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> > > > > think that a single hypercall that retrieves the entire host DTB would
> > > > > be easier to implement
> > > >
> > > > DOMCTL should only be used to handle per-domain information. If we want to
> > > > create a new sub-hypercall of either __HYPERVISOR_platform_op or
> > > > __HYPERVISOR_sysctl_op (not sure which one).
> > > >
> > > > AFAICT, both are versioned.
> > > >
> > > > > and more robust in the long term. >
> > > > > hypfs has the advantage that it would create an interface more similar
> > > > > to the one people are already used to on Linux systems
> > > > > (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> > > > > which intuitively I think it would be slower.
> > > >
> > > > Even if you have the binary blob, you would still have to scan the
> > > > device-tree. That said, it is probably going to be potentially a bit faster
> > > > because you have less hypercall.
> > > >
> > > > However, here this is a trade-off between memory use and speed. If you want
> > > > speed, then you may have to transfer up to 2MB every time. So the question is
> > > > do we care more about speed or memory usage?
> > > >
> > > > > Also the feature might be
> > > > > harder to implement but I am not sure.
> > > > >
> > > > > I don't have a strong preference and this is not a stable interface (we
> > > > > don't have to be extra paranoid about forward and backward
> > > > > compatibility). So I am fine either way. Let's see what the others think
> > > > > as well.
> > > >
> > > > My preference would be to use hypfs as this is cleaner than exposing a blob.
> > >
> > > That's also fine by me. Probably the hypfs implementation shouldn't be
> > > much more difficult than something like
> > > XEN_DOMCTL_host_node_by_path/XEN_DOMCTL_find_host_compatible_node.
> > >
> > >
> > > > However, are we sure we can simply copy the content of the host Device-Tree to
> > > > the guest Device-Tree for SCMI? For instance, I know that for device
> > > > passthrough there are some property that needs to be altered for some devices.
> > > > Hence, why it is not present. Although, I vaguely recalled to have written a
> > > > PoC, not sure if it was posted on the ML.
> > >
> > > The SCMI node cannot be copied "as is" from host to guest. It needs a
> > > couple of changes but they seem feasible as they are limited to the
> > > channels exposed to the guest. (The generic device passthrough case is a
> > > lot more difficult.)
> >
> >
> > Hi Stefano,
> >
> > What I'm thinking about is do we actually need to create SCMI node in DomU device-tree?
> > I have this question is because we don't need SCMI node to be present in DomU
> > device-tree if it has no passed-through devices, which are using scmi.
> > So if we don't have passed-through devices or do not provide DomU partial device-tree
> > in config, then there is no need to create SCMI node.
> >
> > For now I see the following possible domu configurations:
> > 1) If DomU has a lot of passed-through devices and it's easier to inherit
> > host device-tree and disable not passed-through devices.
> > Partial device tree will looks like this:
> >
> > #include "r8a77961-salvator-xs.dts" //include host device tree
> >
> > /
> > {
> > soc {
> > ...
> > }
> >
> > };
> >
> > // Disable non passed-through devices
> > &hscif {
> > status = "disabled";
> > };
> >
> > In this case DomU partial device-tree will inherit arm,scmi-smc and
> > arm,scmi-shmem nodes and all clock/reset/power-domains which are using scmi.
> > All this nodes can be copied to DomU device-tree from partial device-tree.
>
> This is an almost dom0 configuration. For this kind of use-cases, I
> think it is enough to handle dom0 automatically correctly. I wouldn't
> ask for anything more than that.
>
>
> > 2) DomU has few passed-through devices, so it's easier to add the device nodes
> > to the passthrough node of DomU partial device-tree.
> > DomU partial device-tree will look like this:
> > {
> > scmi_shmem: scp-shmem@0x53FF0000 {
> > compatible = "arm,scmi-shmem";
> > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > };
> > scmi {
> > arm,smc-id = <....>;
> > compatible = "arm,scmi-smc";
> > shmem = <&scmi_shmem>;
> > scmi_clock: protocol@14 {
> > ...
> > };
> > scmi_reset: protocol@16 {
> > ...
> > };
> > };
> > passthrough {
> > hscif0: serial@e6540000 {
> > compatible = "renesas,hscif-r8a77961";
> > scmi_devid = <5>;
> > clocks = <&scmi_clock 5>;
> > resets = <&scmi_reset 5>;
> > ...
> > };
> > };
> > };
> >
> > As you can see in this case we have to manually copy arm,scmi-shmem and
> > arm,scmi-smc nodes with hscif0 node or the device-tree compilation will fail.
> > We can use 0x53FF0000, provided in arm,scmi-shmem node and map domain channel
> > to this address and copy scmi related nodes to the DomU device-tree.
> > This is useful when we need to expose only certain protocols to the DomU.
> > Also it's easy to modify DomU scmi node, as we need for stm32mp1 for example
> > when different smc-id should be set for DomU.
>
> I think this is the most interesting case that should be automated and
> not require manual intervention. Let me explain why.
>
> Currently we require partial device trees to be manually written because
> there is no easy way to automatically generate them. (I have some ideas
> on how to automatically generate partial device trees but that is a
> separate discussion.)
>
> Unfortunately, it has become increasingly clear that it is too difficult
> for users (even advanced users!) to come up with the appropriate partial
> device trees. Thus, I am reluctant to introduce more things that rely on
> the user having to manually specify partial device tree information.
> This is why I would like the SCMI nodes to be automatically added for
> domUs.
>
> Of course, if a user provides the scmi and scmi_shmem nodes in the
> partial device tree we could just use them. But ideally we should also
> be able to automatically generated them based on the host device tree
> nodes, so that the user only needs to provide serial@e6540000 (in your
> example, scmi_devid would need to be populated too) and the rest would
> be done automatically as we do today for the gic and vuart nodes.
>
> At the same time I don't want to scope-creep this patch series too much
> and I don't mean to ask you to add a huge new infrastructure to Xen and
> the Xen tools just to get SCMI support in. I would rather have a
> not-great automatic generation of the domU SCMI nodes than nothing (e.g.
> using your suggested XEN_DOMCTL_host_node_by_path and
> XEN_DOMCTL_find_host_compatible_node hypercalls althought they would
> need to be platform_op as Julien suggested).
>
> It looks like the generation of scmi_shmem and scmi should be easy
> enough that could be handled without difficulty in xl/libxl. But if that
> turns out to be too difficult, we could have a small independent
> bash/python script that from the host device tree generates the partial
> device tree with the SCMI nodes. From Xen point of view we are would
> still be using the partial device tree, but the partial device tree
> itself would be generated instead of manually written. As this workflow
> requires a separate tool I think it is a worse option than the one
> above. Still better than nothing though.
>
Hi Stefano,
Thank you for the detail answer. I went through hypfs and will try to
export host device_tree. Then xl will be able to use hypfs data to
generate arm,scmi-smc node if arm,scmi-smc node was not provided in
DomU partial device-tree. Unfortunately, some changes should be done
to hypfs because it seems not ready to handle nested dynamic dirs.
I'll see if I can update hypfs without breaking the original
functionality. If not, I will have to create all hypfs dir structure on
start. For now I'm working on making dynamically created hypfs tree
structure, based on host device-tree.
Best regards,
Oleksii.
>
> > 3) DomU doesn't have any passthrough nodes, which are using scmi.
> > In this case we don't want SCMI nodes to be in the DomU device-tree.
> >
> > I see only one use-case when we may need scmi nodes to be generated by xl in
> > DomU device-tree:
> > Xen generates psci node to handle cpu_on and cpu_off.
> > According to the Section 4.3.2.5 of the DEN0056C [1]:
> > > For these power domains, this protocol can be used to implement PSCI CPU_SUSPEND, CPU_ON, CPU_FREEZE, CPU_DEFAULT_SUSPEND and CPU_OFF functions.
> >
> > So in theory psci node can use scmi to control cpu state. But this is not our
> > use-case because we don't want to give DomU ability to stop physical CPU.
> > Xen can't intercept and handle CPU_ON and CPU_OFF requests when mailbox transport
> > is used for SCMI communication.
> >
> > [1] "SCMI Specification DEN0056C," [Online]. Available: https://urldefense.com/v3/__https://developer.arm.com/documentation/den0056/latest__;!!GF_29dbcQIUBPA!k5oB4BpbIN-hU5jrWvNy9FLXi3Kavu3qTr5lESjK8NnlS261E0Nuqg2_pUQWxb2hDdLa$ [developer[.]arm[.]com]
>
> I agree with you on this one; I am not worried about this case.
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver
2022-01-27 18:11 ` Oleksii Moisieiev
@ 2022-01-27 21:18 ` Stefano Stabellini
0 siblings, 0 replies; 95+ messages in thread
From: Stefano Stabellini @ 2022-01-27 21:18 UTC (permalink / raw)
To: Oleksii Moisieiev
Cc: Stefano Stabellini, Julien Grall, xen-devel, Volodymyr Babchuk,
Bertrand Marquis
On Thu, 27 Jan 2022, Oleksii Moisieiev wrote:
> On Tue, Jan 25, 2022 at 01:19:46PM -0800, Stefano Stabellini wrote:
> > On Tue, 25 Jan 2022, Oleksii Moisieiev wrote:
> > > On Mon, Jan 24, 2022 at 02:14:43PM -0800, Stefano Stabellini wrote:
> > > > On Mon, 24 Jan 2022, Julien Grall wrote:
> > > > > On 24/01/2022 19:06, Stefano Stabellini wrote:
> > > > > > It looks like XEN_DOMCTL_host_node_by_path and
> > > > > > XEN_DOMCTL_find_host_compatible_node would also solve the problem but I
> > > > > > think that a single hypercall that retrieves the entire host DTB would
> > > > > > be easier to implement
> > > > >
> > > > > DOMCTL should only be used to handle per-domain information. If we want to
> > > > > create a new sub-hypercall of either __HYPERVISOR_platform_op or
> > > > > __HYPERVISOR_sysctl_op (not sure which one).
> > > > >
> > > > > AFAICT, both are versioned.
> > > > >
> > > > > > and more robust in the long term. >
> > > > > > hypfs has the advantage that it would create an interface more similar
> > > > > > to the one people are already used to on Linux systems
> > > > > > (/proc/device-tree). xl/libxl would have to scan the whole hypfs tree,
> > > > > > which intuitively I think it would be slower.
> > > > >
> > > > > Even if you have the binary blob, you would still have to scan the
> > > > > device-tree. That said, it is probably going to be potentially a bit faster
> > > > > because you have less hypercall.
> > > > >
> > > > > However, here this is a trade-off between memory use and speed. If you want
> > > > > speed, then you may have to transfer up to 2MB every time. So the question is
> > > > > do we care more about speed or memory usage?
> > > > >
> > > > > > Also the feature might be
> > > > > > harder to implement but I am not sure.
> > > > > >
> > > > > > I don't have a strong preference and this is not a stable interface (we
> > > > > > don't have to be extra paranoid about forward and backward
> > > > > > compatibility). So I am fine either way. Let's see what the others think
> > > > > > as well.
> > > > >
> > > > > My preference would be to use hypfs as this is cleaner than exposing a blob.
> > > >
> > > > That's also fine by me. Probably the hypfs implementation shouldn't be
> > > > much more difficult than something like
> > > > XEN_DOMCTL_host_node_by_path/XEN_DOMCTL_find_host_compatible_node.
> > > >
> > > >
> > > > > However, are we sure we can simply copy the content of the host Device-Tree to
> > > > > the guest Device-Tree for SCMI? For instance, I know that for device
> > > > > passthrough there are some property that needs to be altered for some devices.
> > > > > Hence, why it is not present. Although, I vaguely recalled to have written a
> > > > > PoC, not sure if it was posted on the ML.
> > > >
> > > > The SCMI node cannot be copied "as is" from host to guest. It needs a
> > > > couple of changes but they seem feasible as they are limited to the
> > > > channels exposed to the guest. (The generic device passthrough case is a
> > > > lot more difficult.)
> > >
> > >
> > > Hi Stefano,
> > >
> > > What I'm thinking about is do we actually need to create SCMI node in DomU device-tree?
> > > I have this question is because we don't need SCMI node to be present in DomU
> > > device-tree if it has no passed-through devices, which are using scmi.
> > > So if we don't have passed-through devices or do not provide DomU partial device-tree
> > > in config, then there is no need to create SCMI node.
> > >
> > > For now I see the following possible domu configurations:
> > > 1) If DomU has a lot of passed-through devices and it's easier to inherit
> > > host device-tree and disable not passed-through devices.
> > > Partial device tree will looks like this:
> > >
> > > #include "r8a77961-salvator-xs.dts" //include host device tree
> > >
> > > /
> > > {
> > > soc {
> > > ...
> > > }
> > >
> > > };
> > >
> > > // Disable non passed-through devices
> > > &hscif {
> > > status = "disabled";
> > > };
> > >
> > > In this case DomU partial device-tree will inherit arm,scmi-smc and
> > > arm,scmi-shmem nodes and all clock/reset/power-domains which are using scmi.
> > > All this nodes can be copied to DomU device-tree from partial device-tree.
> >
> > This is an almost dom0 configuration. For this kind of use-cases, I
> > think it is enough to handle dom0 automatically correctly. I wouldn't
> > ask for anything more than that.
> >
> >
> > > 2) DomU has few passed-through devices, so it's easier to add the device nodes
> > > to the passthrough node of DomU partial device-tree.
> > > DomU partial device-tree will look like this:
> > > {
> > > scmi_shmem: scp-shmem@0x53FF0000 {
> > > compatible = "arm,scmi-shmem";
> > > reg = <0x0 0x53FF0000 0x0 0x10000>;
> > > };
> > > scmi {
> > > arm,smc-id = <....>;
> > > compatible = "arm,scmi-smc";
> > > shmem = <&scmi_shmem>;
> > > scmi_clock: protocol@14 {
> > > ...
> > > };
> > > scmi_reset: protocol@16 {
> > > ...
> > > };
> > > };
> > > passthrough {
> > > hscif0: serial@e6540000 {
> > > compatible = "renesas,hscif-r8a77961";
> > > scmi_devid = <5>;
> > > clocks = <&scmi_clock 5>;
> > > resets = <&scmi_reset 5>;
> > > ...
> > > };
> > > };
> > > };
> > >
> > > As you can see in this case we have to manually copy arm,scmi-shmem and
> > > arm,scmi-smc nodes with hscif0 node or the device-tree compilation will fail.
> > > We can use 0x53FF0000, provided in arm,scmi-shmem node and map domain channel
> > > to this address and copy scmi related nodes to the DomU device-tree.
> > > This is useful when we need to expose only certain protocols to the DomU.
> > > Also it's easy to modify DomU scmi node, as we need for stm32mp1 for example
> > > when different smc-id should be set for DomU.
> >
> > I think this is the most interesting case that should be automated and
> > not require manual intervention. Let me explain why.
> >
> > Currently we require partial device trees to be manually written because
> > there is no easy way to automatically generate them. (I have some ideas
> > on how to automatically generate partial device trees but that is a
> > separate discussion.)
> >
> > Unfortunately, it has become increasingly clear that it is too difficult
> > for users (even advanced users!) to come up with the appropriate partial
> > device trees. Thus, I am reluctant to introduce more things that rely on
> > the user having to manually specify partial device tree information.
> > This is why I would like the SCMI nodes to be automatically added for
> > domUs.
> >
> > Of course, if a user provides the scmi and scmi_shmem nodes in the
> > partial device tree we could just use them. But ideally we should also
> > be able to automatically generated them based on the host device tree
> > nodes, so that the user only needs to provide serial@e6540000 (in your
> > example, scmi_devid would need to be populated too) and the rest would
> > be done automatically as we do today for the gic and vuart nodes.
> >
> > At the same time I don't want to scope-creep this patch series too much
> > and I don't mean to ask you to add a huge new infrastructure to Xen and
> > the Xen tools just to get SCMI support in. I would rather have a
> > not-great automatic generation of the domU SCMI nodes than nothing (e.g.
> > using your suggested XEN_DOMCTL_host_node_by_path and
> > XEN_DOMCTL_find_host_compatible_node hypercalls althought they would
> > need to be platform_op as Julien suggested).
> >
> > It looks like the generation of scmi_shmem and scmi should be easy
> > enough that could be handled without difficulty in xl/libxl. But if that
> > turns out to be too difficult, we could have a small independent
> > bash/python script that from the host device tree generates the partial
> > device tree with the SCMI nodes. From Xen point of view we are would
> > still be using the partial device tree, but the partial device tree
> > itself would be generated instead of manually written. As this workflow
> > requires a separate tool I think it is a worse option than the one
> > above. Still better than nothing though.
> >
> Hi Stefano,
>
> Thank you for the detail answer. I went through hypfs and will try to
> export host device_tree. Then xl will be able to use hypfs data to
> generate arm,scmi-smc node if arm,scmi-smc node was not provided in
> DomU partial device-tree. Unfortunately, some changes should be done
> to hypfs because it seems not ready to handle nested dynamic dirs.
>
> I'll see if I can update hypfs without breaking the original
> functionality. If not, I will have to create all hypfs dir structure on
> start. For now I'm working on making dynamically created hypfs tree
> structure, based on host device-tree.
That's fantastic, thank you Oleksii! I think it is going to be super
userful!
^ permalink raw reply [flat|nested] 95+ messages in thread
end of thread, other threads:[~2022-01-27 21:19 UTC | newest]
Thread overview: 95+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-14 9:34 [RFC v1 0/5] Introduce SCI-mediator feature Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 1/5] xen/arm: add support for Renesas R-Car Gen3 platform Oleksii Moisieiev
2021-12-15 6:38 ` Oleksandr Andrushchenko
2021-12-15 20:08 ` Oleksii Moisieiev
2021-12-15 9:39 ` Julien Grall
2021-12-17 10:48 ` Oleksii Moisieiev
2021-12-15 9:57 ` Oleksandr Tyshchenko
2021-12-17 10:52 ` Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 2/5] xen/arm: add generic SCI mediator framework Oleksii Moisieiev
2021-12-17 2:45 ` Stefano Stabellini
2021-12-17 12:50 ` Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 3/5] xen/arm: introduce SCMI-SMC mediator driver Oleksii Moisieiev
2021-12-17 11:35 ` Oleksandr
2021-12-17 13:23 ` Oleksii Moisieiev
2021-12-17 13:37 ` Julien Grall
2021-12-17 13:58 ` Oleksii Moisieiev
2021-12-17 16:38 ` Julien Grall
2021-12-20 15:41 ` Oleksii Moisieiev
2021-12-24 14:42 ` Julien Grall
2021-12-24 17:02 ` Oleksii Moisieiev
2022-01-03 13:14 ` Julien Grall
2022-01-06 13:53 ` Oleksii Moisieiev
2022-01-06 14:02 ` Julien Grall
2022-01-06 15:43 ` Oleksii Moisieiev
2022-01-06 16:04 ` Julien Grall
2022-01-06 16:28 ` Oleksii Moisieiev
2022-01-19 10:37 ` Oleksii Moisieiev
2022-01-20 2:10 ` Stefano Stabellini
2022-01-20 10:25 ` Oleksii Moisieiev
2021-12-18 2:14 ` Stefano Stabellini
2021-12-20 18:12 ` Oleksii Moisieiev
2021-12-21 0:52 ` Stefano Stabellini
2021-12-21 20:03 ` Oleksii Moisieiev
2021-12-21 21:22 ` Stefano Stabellini
2021-12-22 11:04 ` Oleksii Moisieiev
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 18:45 ` Volodymyr Babchuk
2021-12-23 19:06 ` Oleksii Moisieiev
2021-12-24 0:16 ` Stefano Stabellini
2021-12-24 13:29 ` Julien Grall
2021-12-24 13:59 ` Oleksii Moisieiev
2021-12-24 14:28 ` Julien Grall
2021-12-24 16:49 ` Oleksii Moisieiev
2022-01-03 14:23 ` Julien Grall
2022-01-06 15:19 ` Oleksii Moisieiev
2021-12-24 14:07 ` Oleksii Moisieiev
2022-01-19 12:04 ` Oleksii Moisieiev
2022-01-20 1:28 ` Stefano Stabellini
2022-01-20 10:21 ` Oleksii Moisieiev
2022-01-20 22:29 ` Stefano Stabellini
2022-01-21 15:07 ` Oleksii Moisieiev
2022-01-21 20:49 ` Stefano Stabellini
2022-01-24 18:22 ` Oleksii Moisieiev
2022-01-24 19:06 ` Stefano Stabellini
2022-01-24 19:26 ` Julien Grall
2022-01-24 22:14 ` Stefano Stabellini
2022-01-25 14:35 ` Oleksii Moisieiev
2022-01-25 21:19 ` Stefano Stabellini
2022-01-27 18:11 ` Oleksii Moisieiev
2022-01-27 21:18 ` Stefano Stabellini
2021-12-14 9:34 ` [RFC v1 4/5] tools/arm: add "scmi_smc" option to xl.cfg Oleksii Moisieiev
2021-12-15 21:51 ` Oleksandr
2021-12-17 11:00 ` Oleksii Moisieiev
2021-12-21 0:54 ` Stefano Stabellini
2021-12-22 10:24 ` Oleksii Moisieiev
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 19:13 ` Oleksii Moisieiev
2021-12-21 13:27 ` Anthony PERARD
2021-12-22 12:20 ` Oleksii Moisieiev
2021-12-14 9:34 ` [RFC v1 5/5] xen/arm: add SCI mediator support for DomUs Oleksii Moisieiev
2021-12-14 9:41 ` Jan Beulich
2021-12-16 17:36 ` Oleksii Moisieiev
2021-12-17 7:12 ` Jan Beulich
2021-12-17 7:16 ` Jan Beulich
2021-12-17 13:40 ` Oleksii Moisieiev
2021-12-16 0:04 ` Oleksandr
2021-12-17 12:15 ` Oleksii Moisieiev
2021-12-21 14:45 ` Anthony PERARD
2021-12-21 21:39 ` Stefano Stabellini
2021-12-22 9:24 ` Julien Grall
2021-12-22 11:17 ` Volodymyr Babchuk
2021-12-22 11:30 ` Julien Grall
2021-12-22 12:34 ` Volodymyr Babchuk
2021-12-22 13:49 ` Julien Grall
2021-12-23 2:23 ` Stefano Stabellini
2021-12-23 19:06 ` Stefano Stabellini
2021-12-24 13:30 ` Julien Grall
2022-01-19 9:40 ` Oleksii Moisieiev
2022-01-20 1:53 ` Stefano Stabellini
2022-01-20 10:27 ` Oleksii Moisieiev
2021-12-23 19:11 ` Oleksii Moisieiev
2021-12-21 1:37 ` Stefano Stabellini
2021-12-22 13:41 ` Oleksii Moisieiev
2021-12-16 0:33 ` [RFC v1 0/5] Introduce SCI-mediator feature Oleksandr
2021-12-17 12:24 ` Oleksii Moisieiev
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.