[XEN PATCH v7 00/20] Xen FF-A mediator

[XEN PATCH v7 00/20] Xen FF-A mediator

[Jens Wiklander]

Hi,

This patch sets add an FF-A [1] mediator to the TEE mediator framework
already present in Xen.  The FF-A mediator implements the subset of the
FF-A 1.1 specification needed to communicate with OP-TEE using FF-A as
transport mechanism instead of SMC/HVC as with the TEE mediator. It allows
a similar design in OP-TEE as with the TEE mediator where OP-TEE presents
one virtual partition of itself to each guest in Xen.

The FF-A mediator is generic in the sense it has nothing OP-TEE specific
except that only the subset needed for OP-TEE is implemented so far. The
hooks needed to inform OP-TEE that a guest is created or destroyed are part
of the FF-A specification.

It should be possible to extend the FF-A mediator to implement a larger
portion of the FF-A 1.1 specification without breaking with the way OP-TEE
is communicated with here. So it should be possible to support any TEE or
Secure Partition using FF-A as transport with this mediator.

The patches are also available at https://github.com/jenswi-linaro/xen
branch "xen_ffa_v7".

With help from Bertrand I've intregrated this in a test setup with OP-TEE.
Please check prerequisites at
https://optee.readthedocs.io/en/latest/building/prerequisites.html

My setup is duplicated using:
repo init -u https://github.com/jenswi-linaro/manifest.git -m qemu_v8.xml \
        -b qemu_xen_ffa
repo sync -j8
cd build
make -j8 toolchains
make -j8 all
make run-only

Test in dom0 with for instance:
xtest 1004

at the prompt.

To start up a domu and connect to it do:
cd /mnt/host/build/qemu_v8/xen
xl create guest_ffa.cfg
xl console domu

Then test as usual with "xtest 1004".

The setup uses the branch "ffa" from https://github.com/jenswi-linaro/xen.
That's currently the same as the "xen_ffa_v7" branch, but the "ffa" branch
may change later as I update for a new version of the patch set.

[1] https://developer.arm.com/documentation/den0077/latest

Thanks,
Jens

v6->v7:
* Split some of the larger patches into smaller patches for easier review.
  For instance, the v6 patch "xen/arm: add a primitive FF-A mediator" has
  been replaced with:
  - "xen/arm: add a primitive FF-A mediator"
  - "tools: add Arm FF-A mediator"
  - "docs: add Arm FF-A mediator"
  - "xen/arm: ffa: add remaining SMC function IDs"
* Some small fixes in the error path for handle_mem_share()
* Switched to SPDX for license in new files.
* Fixed comment style issues in
  "xen/arm: smccc: add support for SMCCCv1.2 extended input/output registers"
* Made FFA support UNSUPPORTED in "xen/arm: add a primitive FF-A mediator"
* Replaced ffa_get_call_count() with FFA_NR_FUNCS
* Update the FFA_MAX_SHM_PAGE_COUNT with a formula instead of a value.
* Replaced XEN_ARM_FLAGS_FFA with XEN_DOMCTL_CONFIG_TEE_FFA to minimize impact
  on struct xen_arch_domainconfig. This works because the FF-A mediator and
  the OP-TEE mediator will not be used at the same time in by a guest.
* Replaced "ffa" boolean in the guest config with a new "ffa" value to the
  enumeration "tee_type".
* Integrated the FF-A mediator in the TEE mediator framework instead of
  being its own.
* Rebased on staging as of 2023-02-16

v5->v6:
* Updated "xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h"
  commit message and moved the patch right before the patch which needs it.
  Applied Michal Orzel's R-B tag.
* Renamed the guest configuration option "ffa_enabled" to "ffa" and
  updated the description.
* More tools update in "xen/arm: add a primitive FF-A mediator" with the "ffa"
  option, including golang and ocaml.
* Update ffa_domain_init() to return an error if communication with
  the SPMC can't be established.
* Factored out a ffa_domain_destroy() from ffa_relinquish_resources().
* Added ffa_get_call_count() to give an accurate number of FF-A function,
  updated in each patch as new FF-A functions are added.
* Added a flags field in struct xen_arch_domainconfig that replaces the
  ffa_enabled field.
* Made check_mandatory_feature() __init
* Replaced a few printk() calls with gprintk() where needed.
* Rebased on staging as of 2022-09-14

V4->v5:
* Added "xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h"
* Added documentation for the "ffa_enabled" guest config flag
* Changed to GPL license for xen/arch/arm/ffa.c
* Added __read_mostly and const where applicable
* Added more describing comments in the code
* Moved list of shared memory object ("ffa_mem_list") into the guest context
  as they are guest specific
* Simplified a few of the simple wrapper functions for SMC to SPMC
* Added a BUILD_BUG_ON(PAGE_SIZE != FFA_PAGE_SIZE) since the mediator
  currently depends on the page size to be same as FFA_PAGE_SIZE (4k).
* Added max number of shared memory object per guest and max number of
  size of each shared memory object
* Added helper macros to calculate offsets of different FF-A data structures
  in the communication buffer instead of relying on pointer arithmetic
* Addressed style issues and other comments
* Broke the commit "xen/arm: add FF-A mediator" into multiple parts, trying
  to add a few features at a time as requested
* Added a missing call to rxtx_unmap() in ffa_relinquish_resources()
* Assignment of "ffa_enabled" is kept as is until I have something definitive
  on the type etc.
* Tested with CONFIG_DEBUG=y

v3->v4:
* Missed v3 and sent a v4 instead by mistake.

v2->v3:
* Generates offsets into struct arm_smccc_1_2_regs with asm-offsets.c in
  order to avoid hard coded offsets in the assembly function
  arm_smccc_1_2_smc()
* Adds an entry in SUPPORT.md on the FF-A status
* Adds a configuration variable "ffa_enabled" to tell if FF-A should be
  enabled for a particular domu guest
* Moves the ffa_frag_list for fragmented memory share requests into
  struct ffa_ctx instead to keep it per guest in order to avoid mixups
  and simplify locking
* Adds a spinlock to struct ffa_ctx for per guest locking
* Addressing style issues and suggestions
* Uses FFA_FEATURES to check that all the needed features are available
  before initializing the mediator
* Rebased on staging as of 2022-06-20

v1->v2:
* Rebased on staging to resolve some merge conflicts as requested

Jens Wiklander (20):
  xen/arm: smccc: add support for SMCCCv1.2 extended input/output
    registers
  xen/arm: tee: add a primitive FF-A mediator
  tools: add Arm FF-A mediator
  docs: add Arm FF-A mediator
  xen/arm: ffa: add remaining SMC function IDs
  xen/arm: ffa: add flags for FFA_PARTITION_INFO_GET
  xen/arm: ffa: add defines for framework direct request/response
    messages
  xen/arm: ffa: note dependency on 4k pages
  xen/arm: ffa: add support for FFA_ID_GET
  xen/arm: ffa: add direct request support
  xen/arm: ffa: map SPMC rx/tx buffers
  xen/arm: ffa: send guest events to Secure Partitions
  xen/arm: ffa: support mapping guest RX/TX buffers
  xen/arm: ffa: support guest FFA_PARTITION_INFO_GET
  xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h
  xen/arm: ffa: add defines for sharing memory
  xen/arm: ffa: add ABI structs for sharing memory
  xen/arm: ffa: support sharing memory
  xen/arm: ffa: add support to reclaim shared memory
  xen/arm: ffa: support sharing large memory ranges

 SUPPORT.md                         |    7 +
 docs/man/xl.cfg.5.pod.in           |   15 +
 tools/libs/light/libxl_arm.c       |    3 +
 tools/libs/light/libxl_types.idl   |    3 +-
 xen/arch/arm/arm64/asm-offsets.c   |    9 +
 xen/arch/arm/arm64/smc.S           |   42 +
 xen/arch/arm/include/asm/domain.h  |    2 +-
 xen/arch/arm/include/asm/psci.h    |    4 +
 xen/arch/arm/include/asm/regs.h    |   12 +
 xen/arch/arm/include/asm/smccc.h   |   40 +
 xen/arch/arm/include/asm/tee/ffa.h |   35 +
 xen/arch/arm/tee/Kconfig           |   11 +
 xen/arch/arm/tee/Makefile          |    1 +
 xen/arch/arm/tee/ffa.c             | 1839 ++++++++++++++++++++++++++++
 xen/arch/arm/tee/optee.c           |   11 -
 xen/arch/arm/vsmc.c                |   19 +-
 xen/include/public/arch-arm.h      |    1 +
 17 files changed, 2037 insertions(+), 17 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
 create mode 100644 xen/arch/arm/tee/ffa.c

-- 
2.34.1

[XEN PATCH v7 01/20] xen/arm: smccc: add support for SMCCCv1.2 extended input/output registers

[Jens Wiklander]

SMCCC v1.2 [1] AArch64 allows x0-x17 to be used as both parameter
registers and result registers for the SMC and HVC instructions.

Arm Firmware Framework for Armv8-A specification makes use of x0-x7 as
parameter and result registers.

Let us add new interface to support this extended set of input/output
registers.

This is based on 3fdc0cb59d97 ("arm64: smccc: Add support for SMCCCv1.2
extended input/output registers") by Sudeep Holla from the Linux kernel

The SMCCC version reported to the VM is bumped to 1.2 in order to support
handling FF-A messages.

[1] https://developer.arm.com/documentation/den0028/c/?lang=en

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/arm64/asm-offsets.c |  9 +++++++
 xen/arch/arm/arm64/smc.S         | 42 ++++++++++++++++++++++++++++++++
 xen/arch/arm/include/asm/smccc.h | 40 ++++++++++++++++++++++++++++++
 xen/arch/arm/vsmc.c              |  2 +-
 4 files changed, 92 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/arm64/asm-offsets.c b/xen/arch/arm/arm64/asm-offsets.c
index 7226cd9b2eb0..7adb67a1b81a 100644
--- a/xen/arch/arm/arm64/asm-offsets.c
+++ b/xen/arch/arm/arm64/asm-offsets.c
@@ -57,6 +57,15 @@ void __dummy__(void)
    BLANK();
    OFFSET(SMCCC_RES_a0, struct arm_smccc_res, a0);
    OFFSET(SMCCC_RES_a2, struct arm_smccc_res, a2);
+   OFFSET(ARM_SMCCC_1_2_REGS_X0_OFFS, struct arm_smccc_1_2_regs, a0);
+   OFFSET(ARM_SMCCC_1_2_REGS_X2_OFFS, struct arm_smccc_1_2_regs, a2);
+   OFFSET(ARM_SMCCC_1_2_REGS_X4_OFFS, struct arm_smccc_1_2_regs, a4);
+   OFFSET(ARM_SMCCC_1_2_REGS_X6_OFFS, struct arm_smccc_1_2_regs, a6);
+   OFFSET(ARM_SMCCC_1_2_REGS_X8_OFFS, struct arm_smccc_1_2_regs, a8);
+   OFFSET(ARM_SMCCC_1_2_REGS_X10_OFFS, struct arm_smccc_1_2_regs, a10);
+   OFFSET(ARM_SMCCC_1_2_REGS_X12_OFFS, struct arm_smccc_1_2_regs, a12);
+   OFFSET(ARM_SMCCC_1_2_REGS_X14_OFFS, struct arm_smccc_1_2_regs, a14);
+   OFFSET(ARM_SMCCC_1_2_REGS_X16_OFFS, struct arm_smccc_1_2_regs, a16);
 }
 
 /*
diff --git a/xen/arch/arm/arm64/smc.S b/xen/arch/arm/arm64/smc.S
index 91bae62dd4d2..fc6b676e2ee3 100644
--- a/xen/arch/arm/arm64/smc.S
+++ b/xen/arch/arm/arm64/smc.S
@@ -27,3 +27,45 @@ ENTRY(__arm_smccc_1_0_smc)
         stp     x2, x3, [x4, #SMCCC_RES_a2]
 1:
         ret
+
+/*
+ * void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
+ *                        struct arm_smccc_1_2_regs *res)
+ */
+ENTRY(arm_smccc_1_2_smc)
+    /* Save `res` and free a GPR that won't be clobbered by SMC call */
+    stp     x1, x19, [sp, #-16]!
+
+    /* Ensure `args` won't be clobbered while loading regs in next step */
+    mov	x19, x0
+
+    /* Load the registers x0 - x17 from the struct arm_smccc_1_2_regs */
+    ldp	x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
+    ldp	x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
+    ldp	x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
+    ldp	x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
+    ldp	x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
+    ldp	x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
+    ldp	x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
+    ldp	x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
+    ldp	x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
+
+    smc #0
+
+    /* Load the `res` from the stack */
+    ldr	x19, [sp]
+
+    /* Store the registers x0 - x17 into the result structure */
+    stp	x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
+    stp	x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
+    stp	x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
+    stp	x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
+    stp	x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
+    stp	x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
+    stp	x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
+    stp	x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
+    stp	x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
+
+    /* Restore original x19 */
+    ldp     xzr, x19, [sp], #16
+    ret
diff --git a/xen/arch/arm/include/asm/smccc.h b/xen/arch/arm/include/asm/smccc.h
index b3dbeecc90ad..1adcd37443c7 100644
--- a/xen/arch/arm/include/asm/smccc.h
+++ b/xen/arch/arm/include/asm/smccc.h
@@ -33,6 +33,7 @@
 
 #define ARM_SMCCC_VERSION_1_0   SMCCC_VERSION(1, 0)
 #define ARM_SMCCC_VERSION_1_1   SMCCC_VERSION(1, 1)
+#define ARM_SMCCC_VERSION_1_2   SMCCC_VERSION(1, 2)
 
 /*
  * This file provides common defines for ARM SMC Calling Convention as
@@ -265,6 +266,45 @@ void __arm_smccc_1_0_smc(register_t a0, register_t a1, register_t a2,
         else                                                    \
             arm_smccc_1_0_smc(__VA_ARGS__);                     \
     } while ( 0 )
+
+/*
+ * struct arm_smccc_1_2_regs - Arguments for or Results from SMC call
+ * @a0-a17 argument values from registers 0 to 17
+ */
+struct arm_smccc_1_2_regs {
+    unsigned long a0;
+    unsigned long a1;
+    unsigned long a2;
+    unsigned long a3;
+    unsigned long a4;
+    unsigned long a5;
+    unsigned long a6;
+    unsigned long a7;
+    unsigned long a8;
+    unsigned long a9;
+    unsigned long a10;
+    unsigned long a11;
+    unsigned long a12;
+    unsigned long a13;
+    unsigned long a14;
+    unsigned long a15;
+    unsigned long a16;
+    unsigned long a17;
+};
+
+/*
+ * arm_smccc_1_2_smc() - make SMC calls
+ * @args: arguments passed via struct arm_smccc_1_2_regs
+ * @res: result values via struct arm_smccc_1_2_regs
+ *
+ * This function is used to make SMC calls following SMC Calling Convention
+ * v1.2 or above. The content of the supplied param are copied from the
+ * structure to registers prior to the SMC instruction. The return values
+ * are updated with the content from registers on return from the SMC
+ * instruction.
+ */
+void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
+                       struct arm_smccc_1_2_regs *res);
 #endif /* CONFIG_ARM_64 */
 
 #endif /* __ASSEMBLY__ */
diff --git a/xen/arch/arm/vsmc.c b/xen/arch/arm/vsmc.c
index 7335276f3fa1..cd68fa80e98a 100644
--- a/xen/arch/arm/vsmc.c
+++ b/xen/arch/arm/vsmc.c
@@ -85,7 +85,7 @@ static bool handle_arch(struct cpu_user_regs *regs)
     switch ( fid )
     {
     case ARM_SMCCC_VERSION_FID:
-        set_user_reg(regs, 0, ARM_SMCCC_VERSION_1_1);
+        set_user_reg(regs, 0, ARM_SMCCC_VERSION_1_2);
         return true;
 
     case ARM_SMCCC_ARCH_FEATURES_FID:
-- 
2.34.1

[XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Jens Wiklander]

Adds a FF-A version 1.1 [1] mediator to communicate with a Secure
Partition in secure world.

This commit brings in only the parts needed to negotiate FF-A version
number with guest and SPMC.

[1] https://developer.arm.com/documentation/den0077/e
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/include/asm/domain.h  |   2 +-
 xen/arch/arm/include/asm/psci.h    |   4 +
 xen/arch/arm/include/asm/tee/ffa.h |  35 +++++
 xen/arch/arm/tee/Kconfig           |  11 ++
 xen/arch/arm/tee/Makefile          |   1 +
 xen/arch/arm/tee/ffa.c             | 217 +++++++++++++++++++++++++++++
 xen/arch/arm/vsmc.c                |  17 ++-
 xen/include/public/arch-arm.h      |   1 +
 8 files changed, 284 insertions(+), 4 deletions(-)
 create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
 create mode 100644 xen/arch/arm/tee/ffa.c

diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
index 0e310601e846..754daa8efa04 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -110,7 +110,7 @@ struct arch_domain
     struct vpl011 vpl011;
 #endif
 
-#ifdef CONFIG_TEE
+#if defined(CONFIG_TEE) || defined(CONFIG_FFA)
     void *tee;
 #endif
 
diff --git a/xen/arch/arm/include/asm/psci.h b/xen/arch/arm/include/asm/psci.h
index 832f77afff3a..4780972621bb 100644
--- a/xen/arch/arm/include/asm/psci.h
+++ b/xen/arch/arm/include/asm/psci.h
@@ -24,6 +24,10 @@ void call_psci_cpu_off(void);
 void call_psci_system_off(void);
 void call_psci_system_reset(void);
 
+/* Range of allocated PSCI function numbers */
+#define	PSCI_FNUM_MIN_VALUE                 _AC(0,U)
+#define	PSCI_FNUM_MAX_VALUE                 _AC(0x1f,U)
+
 /* PSCI v0.2 interface */
 #define PSCI_0_2_FN32(nr) ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,             \
                                              ARM_SMCCC_CONV_32,               \
diff --git a/xen/arch/arm/include/asm/tee/ffa.h b/xen/arch/arm/include/asm/tee/ffa.h
new file mode 100644
index 000000000000..94960100718e
--- /dev/null
+++ b/xen/arch/arm/include/asm/tee/ffa.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * xen/arch/arm/include/asm/tee/ffa.h
+ *
+ * Arm Firmware Framework for ARMv8-A(FFA) mediator
+ *
+ * Copyright (C) 2023  Linaro Limited
+ */
+
+#ifndef __ASM_ARM_TEE_FFA_H__
+#define __ASM_ARM_TEE_FFA_H__
+
+#include <xen/const.h>
+#include <xen/kconfig.h>
+
+#include <asm/smccc.h>
+#include <asm/types.h>
+
+#define FFA_FNUM_MIN_VALUE              _AC(0x60,U)
+#define FFA_FNUM_MAX_VALUE              _AC(0x86,U)
+
+static inline bool is_ffa_fid(uint32_t fid)
+{
+    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
+
+    return fn >= FFA_FNUM_MIN_VALUE && fn <= FFA_FNUM_MAX_VALUE;
+}
+
+#ifdef CONFIG_FFA
+#define FFA_NR_FUNCS    (9 + (IS_ENABLED(CONFIG_ARM_64) ? 3 : 0) )
+#else
+#define FFA_NR_FUNCS    0
+#endif
+
+#endif /*__ASM_ARM_TEE_FFA_H__*/
diff --git a/xen/arch/arm/tee/Kconfig b/xen/arch/arm/tee/Kconfig
index 392169b2559d..923f08ba8cb7 100644
--- a/xen/arch/arm/tee/Kconfig
+++ b/xen/arch/arm/tee/Kconfig
@@ -8,3 +8,14 @@ config OPTEE
 	  virtualization-enabled OP-TEE present. You can learn more
 	  about virtualization for OP-TEE at
 	  https://optee.readthedocs.io/architecture/virtualization.html
+
+config FFA
+	bool "Enable FF-A mediator support (UNSUPPORTED)" if UNSUPPORTED
+	default n
+	depends on ARM_64
+	help
+	  This option enables a minimal FF-A mediator. The mediator is
+	  generic as it follows the FF-A specification [1], but it only
+	  implements a small subset of the specification.
+
+	  [1] https://developer.arm.com/documentation/den0077/latest
diff --git a/xen/arch/arm/tee/Makefile b/xen/arch/arm/tee/Makefile
index 982c87968447..58a1015e40e0 100644
--- a/xen/arch/arm/tee/Makefile
+++ b/xen/arch/arm/tee/Makefile
@@ -1,2 +1,3 @@
+obj-$(CONFIG_FFA) += ffa.o
 obj-y += tee.o
 obj-$(CONFIG_OPTEE) += optee.o
diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
new file mode 100644
index 000000000000..824153c9303a
--- /dev/null
+++ b/xen/arch/arm/tee/ffa.c
@@ -0,0 +1,217 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * xen/arch/arm/tee/ffa.c
+ *
+ * Arm Firmware Framework for ARMv8-A (FF-A) mediator
+ *
+ * Copyright (C) 2023  Linaro Limited
+ */
+
+#include <xen/domain_page.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <xen/types.h>
+#include <xen/sizes.h>
+#include <xen/bitops.h>
+
+#include <asm/smccc.h>
+#include <asm/event.h>
+#include <asm/tee/tee.h>
+#include <asm/tee/ffa.h>
+#include <asm/regs.h>
+
+/* Error codes */
+#define FFA_RET_OK                      0
+#define FFA_RET_NOT_SUPPORTED           -1
+#define FFA_RET_INVALID_PARAMETERS      -2
+#define FFA_RET_NO_MEMORY               -3
+#define FFA_RET_BUSY                    -4
+#define FFA_RET_INTERRUPTED             -5
+#define FFA_RET_DENIED                  -6
+#define FFA_RET_RETRY                   -7
+#define FFA_RET_ABORTED                 -8
+
+/* FFA_VERSION helpers */
+#define FFA_VERSION_MAJOR_SHIFT         16U
+#define FFA_VERSION_MAJOR_MASK          0x7FFFU
+#define FFA_VERSION_MINOR_SHIFT         0U
+#define FFA_VERSION_MINOR_MASK          0xFFFFU
+#define MAKE_FFA_VERSION(major, minor)  \
+        ((((major) & FFA_VERSION_MAJOR_MASK) << FFA_VERSION_MAJOR_SHIFT) | \
+         ((minor) & FFA_VERSION_MINOR_MASK))
+
+#define FFA_MIN_VERSION         MAKE_FFA_VERSION(1, 0)
+#define FFA_VERSION_1_0         MAKE_FFA_VERSION(1, 0)
+#define FFA_VERSION_1_1         MAKE_FFA_VERSION(1, 1)
+
+/*
+ * This is the version we want to use in communication with guests and SPs.
+ * During negotiation with a guest or a SP we may need to lower it for
+ * that particular guest or SP.
+ */
+#define FFA_MY_VERSION_MAJOR    1U
+#define FFA_MY_VERSION_MINOR    1U
+#define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
+                                                 FFA_MY_VERSION_MINOR)
+
+/* Function IDs */
+#define FFA_ERROR                       0x84000060U
+#define FFA_SUCCESS_32                  0x84000061U
+#define FFA_VERSION                     0x84000063U
+
+struct ffa_ctx {
+    uint32_t guest_vers;
+};
+
+/* Negotiated FF-A version to use with the SPMC */
+static uint32_t ffa_version __ro_after_init;
+
+static bool ffa_get_version(uint32_t *vers)
+{
+    const struct arm_smccc_1_2_regs arg = {
+        .a0 = FFA_VERSION,
+        .a1 = FFA_MY_VERSION,
+    };
+    struct arm_smccc_1_2_regs resp;
+
+    arm_smccc_1_2_smc(&arg, &resp);
+    if ( resp.a0 == FFA_RET_NOT_SUPPORTED )
+    {
+        gprintk(XENLOG_ERR, "ffa: FFA_VERSION returned not supported\n");
+        return false;
+    }
+
+    *vers = resp.a0;
+
+    return true;
+}
+
+static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
+                     register_t v2, register_t v3, register_t v4, register_t v5,
+                     register_t v6, register_t v7)
+{
+        set_user_reg(regs, 0, v0);
+        set_user_reg(regs, 1, v1);
+        set_user_reg(regs, 2, v2);
+        set_user_reg(regs, 3, v3);
+        set_user_reg(regs, 4, v4);
+        set_user_reg(regs, 5, v5);
+        set_user_reg(regs, 6, v6);
+        set_user_reg(regs, 7, v7);
+}
+
+static void handle_version(struct cpu_user_regs *regs)
+{
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    uint32_t vers = get_user_reg(regs, 1);
+
+    if ( vers < FFA_VERSION_1_1 )
+        vers = FFA_VERSION_1_0;
+    else
+        vers = FFA_VERSION_1_1;
+
+    ctx->guest_vers = vers;
+    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
+}
+
+static bool ffa_handle_call(struct cpu_user_regs *regs)
+{
+    uint32_t fid = get_user_reg(regs, 0);
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+
+    if ( !ctx )
+        return false;
+
+    switch ( fid )
+    {
+    case FFA_VERSION:
+        handle_version(regs);
+        return true;
+
+    default:
+        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
+        return false;
+    }
+}
+
+static int ffa_domain_init(struct domain *d)
+{
+    struct ffa_ctx *ctx;
+
+    if ( !ffa_version )
+        return -ENODEV;
+
+    ctx = xzalloc(struct ffa_ctx);
+    if ( !ctx )
+        return -ENOMEM;
+
+    d->arch.tee = ctx;
+
+    return 0;
+}
+
+/* This function is supposed to undo what ffa_domain_init() has done */
+static int ffa_relinquish_resources(struct domain *d)
+{
+    struct ffa_ctx *ctx = d->arch.tee;
+
+    if ( !ctx )
+        return 0;
+
+    XFREE(d->arch.tee);
+
+    return 0;
+}
+
+static bool ffa_probe(void)
+{
+    uint32_t vers;
+    unsigned int major_vers;
+    unsigned int minor_vers;
+
+    /*
+     * psci_init_smccc() updates this value with what's reported by EL-3
+     * or secure world.
+     */
+    if ( smccc_ver < ARM_SMCCC_VERSION_1_2 )
+    {
+        printk(XENLOG_ERR
+               "ffa: unsupported SMCCC version %#x (need at least %#x)\n",
+               smccc_ver, ARM_SMCCC_VERSION_1_2);
+        return false;
+    }
+
+    if ( !ffa_get_version(&vers) )
+        return false;
+
+    if ( vers < FFA_MIN_VERSION || vers > FFA_MY_VERSION )
+    {
+        printk(XENLOG_ERR "ffa: Incompatible version %#x found\n", vers);
+        return false;
+    }
+
+    major_vers = (vers >> FFA_VERSION_MAJOR_SHIFT) & FFA_VERSION_MAJOR_MASK;
+    minor_vers = vers & FFA_VERSION_MINOR_MASK;
+    printk(XENLOG_INFO "ARM FF-A Mediator version %u.%u\n",
+           FFA_MY_VERSION_MAJOR, FFA_MY_VERSION_MINOR);
+    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
+           major_vers, minor_vers);
+
+    ffa_version = vers;
+
+    return true;
+}
+
+static const struct tee_mediator_ops ffa_ops =
+{
+    .probe = ffa_probe,
+    .domain_init = ffa_domain_init,
+    .relinquish_resources = ffa_relinquish_resources,
+    .handle_call = ffa_handle_call,
+};
+
+REGISTER_TEE_MEDIATOR(ffa, "FF-A", XEN_DOMCTL_CONFIG_TEE_FFA, &ffa_ops);
diff --git a/xen/arch/arm/vsmc.c b/xen/arch/arm/vsmc.c
index cd68fa80e98a..7f2f5eb9ce3d 100644
--- a/xen/arch/arm/vsmc.c
+++ b/xen/arch/arm/vsmc.c
@@ -15,6 +15,7 @@
 #include <asm/monitor.h>
 #include <asm/regs.h>
 #include <asm/smccc.h>
+#include <asm/tee/ffa.h>
 #include <asm/tee/tee.h>
 #include <asm/traps.h>
 #include <asm/vpsci.h>
@@ -24,7 +25,7 @@
 #define XEN_SMCCC_FUNCTION_COUNT 3
 
 /* Number of functions currently supported by Standard Service Service Calls. */
-#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS)
+#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS + FFA_NR_FUNCS)
 
 static bool fill_uid(struct cpu_user_regs *regs, xen_uuid_t uuid)
 {
@@ -188,13 +189,23 @@ static bool handle_existing_apis(struct cpu_user_regs *regs)
     return do_vpsci_0_1_call(regs, fid);
 }
 
+static bool is_psci_fid(uint32_t fid)
+{
+    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
+
+    return fn >= PSCI_FNUM_MIN_VALUE && fn <= PSCI_FNUM_MAX_VALUE;
+}
+
 /* PSCI 0.2 interface and other Standard Secure Calls */
 static bool handle_sssc(struct cpu_user_regs *regs)
 {
     uint32_t fid = (uint32_t)get_user_reg(regs, 0);
 
-    if ( do_vpsci_0_2_call(regs, fid) )
-        return true;
+    if ( is_psci_fid(fid) )
+        return do_vpsci_0_2_call(regs, fid);
+
+    if ( is_ffa_fid(fid) )
+        return tee_handle_call(regs);
 
     switch ( fid )
     {
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 1528ced5097a..92aff923056a 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -296,6 +296,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
 
 #define XEN_DOMCTL_CONFIG_TEE_NONE      0
 #define XEN_DOMCTL_CONFIG_TEE_OPTEE     1
+#define XEN_DOMCTL_CONFIG_TEE_FFA       2
 
 struct xen_arch_domainconfig {
     /* IN/OUT */
-- 
2.34.1

[XEN PATCH v7 03/20] tools: add Arm FF-A mediator

[Jens Wiklander]

Adds a new "ffa" value to the Enumeration "tee_type" to indicate if a
guest is trusted to use FF-A.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 tools/libs/light/libxl_arm.c     | 3 +++
 tools/libs/light/libxl_types.idl | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index ddc7b2a15975..601890dda1ce 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -205,6 +205,9 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
     case LIBXL_TEE_TYPE_OPTEE:
         config->arch.tee_type = XEN_DOMCTL_CONFIG_TEE_OPTEE;
         break;
+    case LIBXL_TEE_TYPE_FFA:
+        config->arch.tee_type = XEN_DOMCTL_CONFIG_TEE_FFA;
+        break;
     default:
         LOG(ERROR, "Unknown TEE type %d",
             d_config->b_info.tee);
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index 0cfad8508dbd..64fb570bc19a 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -494,7 +494,8 @@ libxl_gic_version = Enumeration("gic_version", [
 
 libxl_tee_type = Enumeration("tee_type", [
     (0, "none"),
-    (1, "optee")
+    (1, "optee"),
+    (2, "ffa")
     ], init_val = "LIBXL_TEE_TYPE_NONE")
 
 libxl_rdm_reserve = Struct("rdm_reserve", [
-- 
2.34.1

[XEN PATCH v7 04/20] docs: add Arm FF-A mediator

[Jens Wiklander]

Describes a FF-A version 1.1 [1] mediator to communicate with a Secure
Partition in secure world.

[1] https://developer.arm.com/documentation/den0077/latest
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 SUPPORT.md               |  7 +++++++
 docs/man/xl.cfg.5.pod.in | 15 +++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/SUPPORT.md b/SUPPORT.md
index aa1940e55f09..5e0595419684 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -818,6 +818,13 @@ that covers the DMA of the device to be passed through.
 
 No support for QEMU backends in a 16K or 64K domain.
 
+### ARM: Firmware Framework for Arm A-profile (FF-A) Mediator
+
+    Status, Arm64: Tech Preview
+
+There are still some code paths where a vCPU may hog a pCPU longer than
+necessary. The FF-A mediator is not yet implemented for Arm32.
+
 ### ARM: Guest Device Tree support
 
     Status: Supported
diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index 024bceeb61b2..ca4fc3e67b0c 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1645,6 +1645,21 @@ in OP-TEE.
 
 This feature is a B<technology preview>.
 
+=item B<ffa>
+
+B<Arm only.> Allow a guest to communicate via FF-A with Secure Partitions
+(SP), default false.
+
+Currently is only a small subset of the FF-A specification supported. Just
+enough to communicate with OP-TEE. In general only direct messaging and
+sharing memory with one SP. More advanced use cases where memory might be
+shared or donated to multple SPs is not supported.
+
+See L<https://developer.arm.com/documentation/den0077/latest> for more
+informantion about FF-A.
+
+This feature is a B<technology preview>.
+
 =back
 
 =back
-- 
2.34.1

[XEN PATCH v7 05/20] xen/arm: ffa: add remaining SMC function IDs

[Jens Wiklander]

Adds the remaining SMC function IDs from FF-A 1.1 specification.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 824153c9303a..aa6cdbe0a4f9 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -59,7 +59,41 @@
 /* Function IDs */
 #define FFA_ERROR                       0x84000060U
 #define FFA_SUCCESS_32                  0x84000061U
+#define FFA_SUCCESS_64                  0xC4000061U
+#define FFA_INTERRUPT                   0x84000062U
 #define FFA_VERSION                     0x84000063U
+#define FFA_FEATURES                    0x84000064U
+#define FFA_RX_ACQUIRE                  0x84000084U
+#define FFA_RX_RELEASE                  0x84000065U
+#define FFA_RXTX_MAP_32                 0x84000066U
+#define FFA_RXTX_MAP_64                 0xC4000066U
+#define FFA_RXTX_UNMAP                  0x84000067U
+#define FFA_PARTITION_INFO_GET          0x84000068U
+#define FFA_ID_GET                      0x84000069U
+#define FFA_SPM_ID_GET                  0x84000085U
+#define FFA_MSG_WAIT                    0x8400006BU
+#define FFA_MSG_YIELD                   0x8400006CU
+#define FFA_MSG_RUN                     0x8400006DU
+#define FFA_MSG_SEND2                   0x84000086U
+#define FFA_MSG_SEND_DIRECT_REQ_32      0x8400006FU
+#define FFA_MSG_SEND_DIRECT_REQ_64      0xC400006FU
+#define FFA_MSG_SEND_DIRECT_RESP_32     0x84000070U
+#define FFA_MSG_SEND_DIRECT_RESP_64     0xC4000070U
+#define FFA_MEM_DONATE_32               0x84000071U
+#define FFA_MEM_DONATE_64               0xC4000071U
+#define FFA_MEM_LEND_32                 0x84000072U
+#define FFA_MEM_LEND_64                 0xC4000072U
+#define FFA_MEM_SHARE_32                0x84000073U
+#define FFA_MEM_SHARE_64                0xC4000073U
+#define FFA_MEM_RETRIEVE_REQ_32         0x84000074U
+#define FFA_MEM_RETRIEVE_REQ_64         0xC4000074U
+#define FFA_MEM_RETRIEVE_RESP           0x84000075U
+#define FFA_MEM_RELINQUISH              0x84000076U
+#define FFA_MEM_RECLAIM                 0x84000077U
+#define FFA_MEM_FRAG_RX                 0x8400007AU
+#define FFA_MEM_FRAG_TX                 0x8400007BU
+#define FFA_MSG_SEND                    0x8400006EU
+#define FFA_MSG_POLL                    0x8400006AU
 
 struct ffa_ctx {
     uint32_t guest_vers;
-- 
2.34.1

[XEN PATCH v7 06/20] xen/arm: ffa: add flags for FFA_PARTITION_INFO_GET

[Jens Wiklander]

Defines flags used for the function FFA_PARTITION_INFO_GET.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index aa6cdbe0a4f9..f4562ed2defc 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -56,6 +56,32 @@
 #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
                                                  FFA_MY_VERSION_MINOR)
 
+/*
+ * Flags used for the FFA_PARTITION_INFO_GET return message:
+ * BIT(0): Supports receipt of direct requests
+ * BIT(1): Can send direct requests
+ * BIT(2): Can send and receive indirect messages
+ * BIT(3): Supports receipt of notifications
+ * BIT(4-5): Partition ID is a PE endpoint ID
+ */
+#define FFA_PART_PROP_DIRECT_REQ_RECV   BIT(0, U)
+#define FFA_PART_PROP_DIRECT_REQ_SEND   BIT(1, U)
+#define FFA_PART_PROP_INDIRECT_MSGS     BIT(2, U)
+#define FFA_PART_PROP_RECV_NOTIF        BIT(3, U)
+#define FFA_PART_PROP_IS_PE_ID          (0U << 4)
+#define FFA_PART_PROP_IS_SEPID_INDEP    (1U << 4)
+#define FFA_PART_PROP_IS_SEPID_DEP      (2U << 4)
+#define FFA_PART_PROP_IS_AUX_ID         (3U << 4)
+#define FFA_PART_PROP_NOTIF_CREATED     BIT(6, U)
+#define FFA_PART_PROP_NOTIF_DESTROYED   BIT(7, U)
+#define FFA_PART_PROP_AARCH64_STATE     BIT(8, U)
+
+/*
+ * Flag used as parameter to FFA_PARTITION_INFO_GET to return partition
+ * count only.
+ */
+#define FFA_PARTITION_INFO_GET_COUNT_FLAG BIT(0, U)
+
 /* Function IDs */
 #define FFA_ERROR                       0x84000060U
 #define FFA_SUCCESS_32                  0x84000061U
-- 
2.34.1

[XEN PATCH v7 07/20] xen/arm: ffa: add defines for framework direct request/response messages

[Jens Wiklander]

Adds defines for framework direct request/response messages.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index f4562ed2defc..d04bac9cc47f 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -56,6 +56,15 @@
 #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
                                                  FFA_MY_VERSION_MINOR)
 
+/* Framework direct request/response */
+#define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
+#define FFA_MSG_TYPE_MASK               0xFFU;
+#define FFA_MSG_PSCI                    0x0U
+#define FFA_MSG_SEND_VM_CREATED         0x4U
+#define FFA_MSG_RESP_VM_CREATED         0x5U
+#define FFA_MSG_SEND_VM_DESTROYED       0x6U
+#define FFA_MSG_RESP_VM_DESTROYED       0x7U
+
 /*
  * Flags used for the FFA_PARTITION_INFO_GET return message:
  * BIT(0): Supports receipt of direct requests
-- 
2.34.1

[XEN PATCH v7 08/20] xen/arm: ffa: note dependency on 4k pages

[Jens Wiklander]

Adds a BUILD_BUG_ON() to assert the dependency on 4k pages in the FF-A
mediator.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index d04bac9cc47f..8b0b80ce1ff5 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -56,6 +56,16 @@
 #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
                                                  FFA_MY_VERSION_MINOR)
 
+/*
+ * The FF-A specification explicitly works with 4K pages as a measure of
+ * memory size, for example, FFA_RXTX_MAP takes one parameter "RX/TX page
+ * count" which is the number of contiguous 4K pages allocated. Xen may use
+ * a different page size depending on the configuration to avoid confusion
+ * with PAGE_SIZE use a special define when it's a page size as in the FF-A
+ * specification.
+ */
+#define FFA_PAGE_SIZE                   SZ_4K
+
 /* Framework direct request/response */
 #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
 #define FFA_MSG_TYPE_MASK               0xFFU;
@@ -242,6 +252,17 @@ static bool ffa_probe(void)
     unsigned int major_vers;
     unsigned int minor_vers;
 
+    /*
+     * FF-A often works in units of 4K pages and currently it's assumed
+     * that we can map memory using that granularity. See also the comment
+     * above the FFA_PAGE_SIZE define.
+     *
+     * It is possible to support a PAGE_SIZE larger than 4K in Xen, but
+     * until that is fully handled in this code make sure that we only use
+     * 4K page sizes.
+     */
+    BUILD_BUG_ON(PAGE_SIZE != FFA_PAGE_SIZE);
+
     /*
      * psci_init_smccc() updates this value with what's reported by EL-3
      * or secure world.
-- 
2.34.1

[XEN PATCH v7 09/20] xen/arm: ffa: add support for FFA_ID_GET

[Jens Wiklander]

Adds support for the FF-A function FFA_ID_GET to return the ID of the
calling client.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 8b0b80ce1ff5..463fd7730573 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -167,6 +167,12 @@ static bool ffa_get_version(uint32_t *vers)
     return true;
 }
 
+static uint16_t get_vm_id(const struct domain *d)
+{
+    /* +1 since 0 is reserved for the hypervisor in FF-A */
+    return d->domain_id + 1;
+}
+
 static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
                      register_t v2, register_t v3, register_t v4, register_t v5,
                      register_t v6, register_t v7)
@@ -181,6 +187,12 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
         set_user_reg(regs, 7, v7);
 }
 
+static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
+                             uint32_t w3)
+{
+    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
+}
+
 static void handle_version(struct cpu_user_regs *regs)
 {
     struct domain *d = current->domain;
@@ -210,6 +222,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
     case FFA_VERSION:
         handle_version(regs);
         return true;
+    case FFA_ID_GET:
+        set_regs_success(regs, get_vm_id(d), 0);
+        return true;
 
     default:
         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
@@ -221,7 +236,11 @@ static int ffa_domain_init(struct domain *d)
 {
     struct ffa_ctx *ctx;
 
-    if ( !ffa_version )
+     /*
+      * We can't use that last possible domain ID or get_vm_id() would cause
+      * an overflow.
+      */
+    if ( !ffa_version || d->domain_id == UINT16_MAX)
         return -ENODEV;
 
     ctx = xzalloc(struct ffa_ctx);
-- 
2.34.1

[XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Jens Wiklander]

Adds support for sending a FF-A direct request. Checks that the SP also
supports handling a 32-bit direct request. 64-bit direct requests are
not used by the mediator itself so there is not need to check for that.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 463fd7730573..a5d8a12635b6 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -142,6 +142,7 @@
 
 struct ffa_ctx {
     uint32_t guest_vers;
+    bool interrupted;
 };
 
 /* Negotiated FF-A version to use with the SPMC */
@@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
     return true;
 }
 
+static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
+{
+    switch ( resp->a0 )
+    {
+    case FFA_ERROR:
+        if ( resp->a2 )
+            return resp->a2;
+        else
+            return FFA_RET_NOT_SUPPORTED;
+    case FFA_SUCCESS_32:
+    case FFA_SUCCESS_64:
+        return FFA_RET_OK;
+    default:
+        return FFA_RET_NOT_SUPPORTED;
+    }
+}
+
+static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
+                               register_t a3, register_t a4)
+{
+    const struct arm_smccc_1_2_regs arg = {
+        .a0 = fid,
+        .a1 = a1,
+        .a2 = a2,
+        .a3 = a3,
+        .a4 = a4,
+    };
+    struct arm_smccc_1_2_regs resp;
+
+    arm_smccc_1_2_smc(&arg, &resp);
+
+    return get_ffa_ret_code(&resp);
+}
+
+static int32_t ffa_features(uint32_t id)
+{
+    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
+}
+
+static bool check_mandatory_feature(uint32_t id)
+{
+    uint32_t ret = ffa_features(id);
+
+    if (ret)
+        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
+
+    return !ret;
+}
+
 static uint16_t get_vm_id(const struct domain *d)
 {
     /* +1 since 0 is reserved for the hypervisor in FF-A */
@@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
 }
 
+static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
+{
+    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
+    struct arm_smccc_1_2_regs resp = { };
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    uint32_t src_dst;
+    uint64_t mask;
+
+    if ( smccc_is_conv_64(fid) )
+        mask = GENMASK_ULL(63, 0);
+    else
+        mask = GENMASK_ULL(31, 0);
+
+    src_dst = get_user_reg(regs, 1);
+    if ( (src_dst >> 16) != get_vm_id(d) )
+    {
+        resp.a0 = FFA_ERROR;
+        resp.a2 = FFA_RET_INVALID_PARAMETERS;
+        goto out;
+    }
+
+    arg.a1 = src_dst;
+    arg.a2 = get_user_reg(regs, 2) & mask;
+    arg.a3 = get_user_reg(regs, 3) & mask;
+    arg.a4 = get_user_reg(regs, 4) & mask;
+    arg.a5 = get_user_reg(regs, 5) & mask;
+    arg.a6 = get_user_reg(regs, 6) & mask;
+    arg.a7 = get_user_reg(regs, 7) & mask;
+
+    while ( true )
+    {
+        arm_smccc_1_2_smc(&arg, &resp);
+
+        switch ( resp.a0 )
+        {
+        case FFA_INTERRUPT:
+            ctx->interrupted = true;
+            goto out;
+        case FFA_ERROR:
+        case FFA_SUCCESS_32:
+        case FFA_SUCCESS_64:
+        case FFA_MSG_SEND_DIRECT_RESP_32:
+        case FFA_MSG_SEND_DIRECT_RESP_64:
+            goto out;
+        default:
+            /* Bad fid, report back. */
+            memset(&arg, 0, sizeof(arg));
+            arg.a0 = FFA_ERROR;
+            arg.a1 = src_dst;
+            arg.a2 = FFA_RET_NOT_SUPPORTED;
+            continue;
+        }
+    }
+
+out:
+    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
+             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
+}
+
 static bool ffa_handle_call(struct cpu_user_regs *regs)
 {
     uint32_t fid = get_user_reg(regs, 0);
@@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
     case FFA_ID_GET:
         set_regs_success(regs, get_vm_id(d), 0);
         return true;
+    case FFA_MSG_SEND_DIRECT_REQ_32:
+#ifdef CONFIG_ARM_64
+    case FFA_MSG_SEND_DIRECT_REQ_64:
+#endif
+        handle_msg_send_direct_req(regs, fid);
+        return true;
 
     default:
         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
@@ -310,6 +426,9 @@ static bool ffa_probe(void)
     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
            major_vers, minor_vers);
 
+    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
+        return false;
+
     ffa_version = vers;
 
     return true;
-- 
2.34.1

[XEN PATCH v7 11/20] xen/arm: ffa: map SPMC rx/tx buffers

[Jens Wiklander]

When initializing the FF-A mediator map the RX and TX buffers shared with
the SPMC.

These buffer are later used to to transmit data that cannot be passed in
registers only.

Adds a check that the SP supports the needed FF-A features
FFA_RXTX_MAP_64 / FFA_RXTX_MAP_32 and FFA_RXTX_UNMAP. In 64-bit mode we
must use FFA_RXTX_MAP_64 since registers are used to transmit the
physical addresses of the RX/TX buffers.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 57 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index a5d8a12635b6..07dd5c36d54b 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -148,6 +148,15 @@ struct ffa_ctx {
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
+/*
+ * Our rx/tx buffers shared with the SPMC.
+ *
+ * ffa_page_count is the number of pages used in each of these buffers.
+ */
+static void *ffa_rx __read_mostly;
+static void *ffa_tx __read_mostly;
+static unsigned int ffa_page_count __read_mostly;
+
 static bool ffa_get_version(uint32_t *vers)
 {
     const struct arm_smccc_1_2_regs arg = {
@@ -217,6 +226,17 @@ static bool check_mandatory_feature(uint32_t id)
     return !ret;
 }
 
+static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
+                            uint32_t page_count)
+{
+    uint32_t fid = FFA_RXTX_MAP_32;
+
+    if ( IS_ENABLED(CONFIG_ARM_64) )
+        fid = FFA_RXTX_MAP_64;
+
+    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
+}
+
 static uint16_t get_vm_id(const struct domain *d)
 {
     /* +1 since 0 is reserved for the hypervisor in FF-A */
@@ -384,6 +404,7 @@ static int ffa_relinquish_resources(struct domain *d)
 static bool ffa_probe(void)
 {
     uint32_t vers;
+    int e;
     unsigned int major_vers;
     unsigned int minor_vers;
 
@@ -426,12 +447,46 @@ static bool ffa_probe(void)
     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
            major_vers, minor_vers);
 
-    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
+    if (
+#ifdef CONFIG_ARM_64
+         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
+#endif
+#ifdef CONFIG_ARM_32
+         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
+#endif
+         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
+         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
         return false;
 
+    ffa_rx = alloc_xenheap_pages(0, 0);
+    if ( !ffa_rx )
+        return false;
+
+    ffa_tx = alloc_xenheap_pages(0, 0);
+    if ( !ffa_tx )
+        goto err_free_ffa_rx;
+
+    e = ffa_rxtx_map(__pa(ffa_tx), __pa(ffa_rx), 1);
+    if ( e )
+    {
+        printk(XENLOG_ERR "ffa: Failed to map rxtx: error %d\n", e);
+        goto err_free_ffa_tx;
+    }
+    ffa_page_count = 1;
     ffa_version = vers;
 
     return true;
+
+err_free_ffa_tx:
+    free_xenheap_pages(ffa_tx, 0);
+    ffa_tx = NULL;
+err_free_ffa_rx:
+    free_xenheap_pages(ffa_rx, 0);
+    ffa_rx = NULL;
+    ffa_page_count = 0;
+    ffa_version = 0;
+
+    return false;
 }
 
 static const struct tee_mediator_ops ffa_ops =
-- 
2.34.1

[XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Jens Wiklander]

The FF-A specification defines framework messages sent as direct
requests when certain events occurs. For instance when a VM (guest) is
created or destroyed. Only SPs which have subscribed to these events
will receive them. An SP can subscribe to these messages in its
partition properties.

Adds a check that the SP supports the needed FF-A features
FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.

The partition properties of each SP is retrieved with
FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
caller (us), so once we're done with the buffer it must be released
using FFA_RX_RELEASE before another call can be made.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 190 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 07dd5c36d54b..f1b014b6c7f4 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -140,6 +140,14 @@
 #define FFA_MSG_SEND                    0x8400006EU
 #define FFA_MSG_POLL                    0x8400006AU
 
+/* Partition information descriptor */
+struct ffa_partition_info_1_1 {
+    uint16_t id;
+    uint16_t execution_context;
+    uint32_t partition_properties;
+    uint8_t uuid[16];
+};
+
 struct ffa_ctx {
     uint32_t guest_vers;
     bool interrupted;
@@ -148,6 +156,12 @@ struct ffa_ctx {
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
+/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
+static uint16_t *subscr_vm_created __read_mostly;
+static unsigned int subscr_vm_created_count __read_mostly;
+static uint16_t *subscr_vm_destroyed __read_mostly;
+static unsigned int subscr_vm_destroyed_count __read_mostly;
+
 /*
  * Our rx/tx buffers shared with the SPMC.
  *
@@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
     return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
 }
 
+static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
+                                      uint32_t w4, uint32_t w5,
+                                      uint32_t *count)
+{
+    const struct arm_smccc_1_2_regs arg = {
+        .a0 = FFA_PARTITION_INFO_GET,
+        .a1 = w1,
+        .a2 = w2,
+        .a3 = w3,
+        .a4 = w4,
+        .a5 = w5,
+    };
+    struct arm_smccc_1_2_regs resp;
+    uint32_t ret;
+
+    arm_smccc_1_2_smc(&arg, &resp);
+
+    ret = get_ffa_ret_code(&resp);
+    if ( !ret )
+        *count = resp.a2;
+
+    return ret;
+}
+
+static int32_t ffa_rx_release(void)
+{
+    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
+}
+
+static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
+                                      uint8_t msg)
+{
+    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
+    int32_t res;
+
+    if ( msg == FFA_MSG_SEND_VM_CREATED )
+        exp_resp |= FFA_MSG_RESP_VM_CREATED;
+    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
+        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
+    else
+        return FFA_RET_INVALID_PARAMETERS;
+
+    do {
+        const struct arm_smccc_1_2_regs arg = {
+            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
+            .a1 = sp_id,
+            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
+            .a5 = vm_id,
+        };
+        struct arm_smccc_1_2_regs resp;
+
+        arm_smccc_1_2_smc(&arg, &resp);
+        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
+        {
+            /*
+             * This is an invalid response, likely due to some error in the
+             * implementation of the ABI.
+             */
+            return FFA_RET_INVALID_PARAMETERS;
+        }
+        res = resp.a3;
+    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
+
+    return res;
+}
+
 static uint16_t get_vm_id(const struct domain *d)
 {
     /* +1 since 0 is reserved for the hypervisor in FF-A */
@@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
 static int ffa_domain_init(struct domain *d)
 {
     struct ffa_ctx *ctx;
+    unsigned int n;
+    unsigned int m;
+    unsigned int c_pos;
+    int32_t res;
 
      /*
       * We can't use that last possible domain ID or get_vm_id() would cause
@@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
     if ( !ctx )
         return -ENOMEM;
 
+    for ( n = 0; n < subscr_vm_created_count; n++ )
+    {
+        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
+                                     FFA_MSG_SEND_VM_CREATED);
+        if ( res )
+        {
+            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
+                   get_vm_id(d), subscr_vm_created[n], res);
+            c_pos = n;
+            goto err;
+        }
+    }
+
     d->arch.tee = ctx;
 
     return 0;
+
+err:
+    /* Undo any already sent vm created messaged */
+    for ( n = 0; n < c_pos; n++ )
+        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
+            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
+                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
+                                       FFA_MSG_SEND_VM_DESTROYED);
+
+    return -ENOMEM;
 }
 
 /* This function is supposed to undo what ffa_domain_init() has done */
 static int ffa_relinquish_resources(struct domain *d)
 {
     struct ffa_ctx *ctx = d->arch.tee;
+    unsigned int n;
+    int32_t res;
 
     if ( !ctx )
         return 0;
 
+    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
+    {
+        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
+                                     FFA_MSG_SEND_VM_DESTROYED);
+
+        if ( res )
+            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
+                   get_vm_id(d), subscr_vm_destroyed[n], res);
+    }
+
     XFREE(d->arch.tee);
 
     return 0;
 }
 
+static bool init_subscribers(void)
+{
+    struct ffa_partition_info_1_1 *fpi;
+    bool ret = false;
+    uint32_t count;
+    int e;
+    uint32_t n;
+    uint32_t c_pos;
+    uint32_t d_pos;
+
+    if ( ffa_version < FFA_VERSION_1_1 )
+        return true;
+
+    e = ffa_partition_info_get(0, 0, 0, 0, 0, &count);
+    if ( e )
+    {
+        printk(XENLOG_ERR "ffa: Failed to get list of SPs: %d\n", e);
+        goto out;
+    }
+
+    fpi = ffa_rx;
+    subscr_vm_created_count = 0;
+    subscr_vm_destroyed_count = 0;
+    for ( n = 0; n < count; n++ )
+    {
+        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED)
+            subscr_vm_created_count++;
+        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED)
+            subscr_vm_destroyed_count++;
+    }
+
+    if ( subscr_vm_created_count )
+        subscr_vm_created = xzalloc_array(uint16_t, subscr_vm_created_count);
+    if ( subscr_vm_destroyed_count )
+        subscr_vm_destroyed = xzalloc_array(uint16_t,
+                                            subscr_vm_destroyed_count);
+    if ( (subscr_vm_created_count && !subscr_vm_created) ||
+         (subscr_vm_destroyed_count && !subscr_vm_destroyed) )
+    {
+        printk(XENLOG_ERR "ffa: Failed to allocate subscription lists\n");
+        subscr_vm_created_count = 0;
+        subscr_vm_destroyed_count = 0;
+        XFREE(subscr_vm_created);
+        XFREE(subscr_vm_destroyed);
+        goto out;
+    }
+
+    for ( c_pos = 0, d_pos = 0, n = 0; n < count; n++ )
+    {
+        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED )
+            subscr_vm_created[c_pos++] = fpi[n].id;
+        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED )
+            subscr_vm_destroyed[d_pos++] = fpi[n].id;
+    }
+
+    ret = true;
+out:
+    ffa_rx_release();
+
+    return ret;
+}
+
 static bool ffa_probe(void)
 {
     uint32_t vers;
@@ -447,7 +628,8 @@ static bool ffa_probe(void)
     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
            major_vers, minor_vers);
 
-    if (
+    if ( !check_mandatory_feature(FFA_PARTITION_INFO_GET) ||
+         !check_mandatory_feature(FFA_RX_RELEASE) ||
 #ifdef CONFIG_ARM_64
          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
 #endif
@@ -475,6 +657,9 @@ static bool ffa_probe(void)
     ffa_page_count = 1;
     ffa_version = vers;
 
+    if ( !init_subscribers() )
+        goto err_free_ffa_tx;
+
     return true;
 
 err_free_ffa_tx:
@@ -485,6 +670,10 @@ err_free_ffa_rx:
     ffa_rx = NULL;
     ffa_page_count = 0;
     ffa_version = 0;
+    XFREE(subscr_vm_created);
+    subscr_vm_created_count = 0;
+    XFREE(subscr_vm_destroyed);
+    subscr_vm_destroyed_count = 0;
 
     return false;
 }
-- 
2.34.1

[XEN PATCH v7 13/20] xen/arm: ffa: support mapping guest RX/TX buffers

[Jens Wiklander]

Adds support in the mediator to map and unmap the RX and TX buffers
provided by the guest using the two FF-A functions FFA_RXTX_MAP and
FFA_RXTX_UNMAP.

These buffer are later used to to transmit data that cannot be passed in
registers only.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 127 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index f1b014b6c7f4..953b6dfd5eca 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -149,10 +149,17 @@ struct ffa_partition_info_1_1 {
 };
 
 struct ffa_ctx {
+    void *rx;
+    const void *tx;
+    struct page_info *rx_pg;
+    struct page_info *tx_pg;
+    unsigned int page_count;
     uint32_t guest_vers;
+    bool tx_is_mine;
     bool interrupted;
 };
 
+
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
@@ -337,6 +344,11 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
         set_user_reg(regs, 7, v7);
 }
 
+static void set_regs_error(struct cpu_user_regs *regs, uint32_t error_code)
+{
+    set_regs(regs, FFA_ERROR, 0, error_code, 0, 0, 0, 0, 0);
+}
+
 static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
                              uint32_t w3)
 {
@@ -358,6 +370,99 @@ static void handle_version(struct cpu_user_regs *regs)
     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
 }
 
+static uint32_t handle_rxtx_map(uint32_t fid, register_t tx_addr,
+                                register_t rx_addr, uint32_t page_count)
+{
+    uint32_t ret = FFA_RET_INVALID_PARAMETERS;
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    struct page_info *tx_pg;
+    struct page_info *rx_pg;
+    p2m_type_t t;
+    void *rx;
+    void *tx;
+
+    if ( !smccc_is_conv_64(fid) )
+    {
+        tx_addr &= UINT32_MAX;
+        rx_addr &= UINT32_MAX;
+    }
+
+    /* For now to keep things simple, only deal with a single page */
+    if ( page_count != 1 )
+        return FFA_RET_NOT_SUPPORTED;
+
+    /* Already mapped */
+    if ( ctx->rx )
+        return FFA_RET_DENIED;
+
+    tx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(tx_addr)), &t, P2M_ALLOC);
+    if ( !tx_pg )
+        return FFA_RET_INVALID_PARAMETERS;
+    /* Only normal RAM for now */
+    if ( !p2m_is_ram(t) )
+        goto err_put_tx_pg;
+
+    rx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(rx_addr)), &t, P2M_ALLOC);
+    if ( !tx_pg )
+        goto err_put_tx_pg;
+    /* Only normal RAM for now */
+    if ( !p2m_is_ram(t) )
+        goto err_put_rx_pg;
+
+    tx = __map_domain_page_global(tx_pg);
+    if ( !tx )
+        goto err_put_rx_pg;
+
+    rx = __map_domain_page_global(rx_pg);
+    if ( !rx )
+        goto err_unmap_tx;
+
+    ctx->rx = rx;
+    ctx->tx = tx;
+    ctx->rx_pg = rx_pg;
+    ctx->tx_pg = tx_pg;
+    ctx->page_count = 1;
+    ctx->tx_is_mine = true;
+    return FFA_RET_OK;
+
+err_unmap_tx:
+    unmap_domain_page_global(tx);
+err_put_rx_pg:
+    put_page(rx_pg);
+err_put_tx_pg:
+    put_page(tx_pg);
+
+    return ret;
+}
+
+static void rxtx_unmap(struct ffa_ctx *ctx)
+{
+    unmap_domain_page_global(ctx->rx);
+    unmap_domain_page_global(ctx->tx);
+    put_page(ctx->rx_pg);
+    put_page(ctx->tx_pg);
+    ctx->rx = NULL;
+    ctx->tx = NULL;
+    ctx->rx_pg = NULL;
+    ctx->tx_pg = NULL;
+    ctx->page_count = 0;
+    ctx->tx_is_mine = false;
+}
+
+static uint32_t handle_rxtx_unmap(void)
+{
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+
+    if ( !ctx->rx )
+        return FFA_RET_INVALID_PARAMETERS;
+
+    rxtx_unmap(ctx);
+
+    return FFA_RET_OK;
+}
+
 static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
 {
     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
@@ -423,6 +528,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
     uint32_t fid = get_user_reg(regs, 0);
     struct domain *d = current->domain;
     struct ffa_ctx *ctx = d->arch.tee;
+    int e;
 
     if ( !ctx )
         return false;
@@ -435,6 +541,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
     case FFA_ID_GET:
         set_regs_success(regs, get_vm_id(d), 0);
         return true;
+    case FFA_RXTX_MAP_32:
+#ifdef CONFIG_ARM_64
+    case FFA_RXTX_MAP_64:
+#endif
+        e = handle_rxtx_map(fid, get_user_reg(regs, 1), get_user_reg(regs, 2),
+                            get_user_reg(regs, 3));
+        if ( e )
+            set_regs_error(regs, e);
+        else
+            set_regs_success(regs, 0, 0);
+        return true;
+    case FFA_RXTX_UNMAP:
+        e = handle_rxtx_unmap();
+        if ( e )
+            set_regs_error(regs, e);
+        else
+            set_regs_success(regs, 0, 0);
+        return true;
     case FFA_MSG_SEND_DIRECT_REQ_32:
 #ifdef CONFIG_ARM_64
     case FFA_MSG_SEND_DIRECT_REQ_64:
@@ -515,6 +639,9 @@ static int ffa_relinquish_resources(struct domain *d)
                    get_vm_id(d), subscr_vm_destroyed[n], res);
     }
 
+    if ( ctx->rx )
+        rxtx_unmap(ctx);
+
     XFREE(d->arch.tee);
 
     return 0;
-- 
2.34.1

[XEN PATCH v7 14/20] xen/arm: ffa: support guest FFA_PARTITION_INFO_GET

[Jens Wiklander]

Adds support in the mediator to handle FFA_PARTITION_INFO_GET requests
from a guest. The requests are forwarded to the SPMC and the response is
translated according to the FF-A version in use by the guest.

Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
caller (the guest in this case), so once it is done with the buffer it
must be released using FFA_RX_RELEASE before another call can be made.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 126 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 124 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 953b6dfd5eca..3571817c0bcd 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -141,6 +141,12 @@
 #define FFA_MSG_POLL                    0x8400006AU
 
 /* Partition information descriptor */
+struct ffa_partition_info_1_0 {
+    uint16_t id;
+    uint16_t execution_context;
+    uint32_t partition_properties;
+};
+
 struct ffa_partition_info_1_1 {
     uint16_t id;
     uint16_t execution_context;
@@ -157,9 +163,8 @@ struct ffa_ctx {
     uint32_t guest_vers;
     bool tx_is_mine;
     bool interrupted;
+    spinlock_t lock;
 };
-
-
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
@@ -173,10 +178,16 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
  * Our rx/tx buffers shared with the SPMC.
  *
  * ffa_page_count is the number of pages used in each of these buffers.
+ *
+ * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
+ * Note that the SPMC is also tracking the ownership of our RX buffer so
+ * for calls which uses our RX buffer to deliver a result we must call
+ * ffa_rx_release() to let the SPMC know that we're done with the buffer.
  */
 static void *ffa_rx __read_mostly;
 static void *ffa_tx __read_mostly;
 static unsigned int ffa_page_count __read_mostly;
+static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
 
 static bool ffa_get_version(uint32_t *vers)
 {
@@ -463,6 +474,98 @@ static uint32_t handle_rxtx_unmap(void)
     return FFA_RET_OK;
 }
 
+static uint32_t handle_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
+                                          uint32_t w4, uint32_t w5,
+                                          uint32_t *count)
+{
+    bool query_count_only = w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG;
+    uint32_t w5_mask = 0;
+    uint32_t ret = FFA_RET_DENIED;
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+
+    /*
+     * FF-A v1.0 has w5 MBZ while v1.1 allows
+     * FFA_PARTITION_INFO_GET_COUNT_FLAG to be non-zero.
+     */
+    if ( ctx->guest_vers == FFA_VERSION_1_1 )
+        w5_mask = FFA_PARTITION_INFO_GET_COUNT_FLAG;
+    if ( w5 & ~w5_mask )
+        return FFA_RET_INVALID_PARAMETERS;
+
+    if ( query_count_only )
+        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
+
+    if ( !ffa_page_count )
+        return FFA_RET_DENIED;
+
+    spin_lock(&ctx->lock);
+    spin_lock(&ffa_rx_buffer_lock);
+    if ( !ctx->page_count || !ctx->tx_is_mine )
+        goto out;
+    ret = ffa_partition_info_get(w1, w2, w3, w4, w5, count);
+    if ( ret )
+        goto out;
+
+    if ( ctx->guest_vers == FFA_VERSION_1_0 )
+    {
+        size_t n;
+        struct ffa_partition_info_1_1 *src = ffa_rx;
+        struct ffa_partition_info_1_0 *dst = ctx->rx;
+
+        if ( ctx->page_count * FFA_PAGE_SIZE < *count * sizeof(*dst) )
+        {
+            ret = FFA_RET_NO_MEMORY;
+            goto out_rx_release;
+        }
+
+        for ( n = 0; n < *count; n++ )
+        {
+            dst[n].id = src[n].id;
+            dst[n].execution_context = src[n].execution_context;
+            dst[n].partition_properties = src[n].partition_properties;
+        }
+    }
+    else
+    {
+        size_t sz = *count * sizeof(struct ffa_partition_info_1_1);
+
+        if ( ctx->page_count * FFA_PAGE_SIZE < sz )
+        {
+            ret = FFA_RET_NO_MEMORY;
+            goto out_rx_release;
+        }
+
+
+        memcpy(ctx->rx, ffa_rx, sz);
+    }
+    ctx->tx_is_mine = false;
+out_rx_release:
+    ffa_rx_release();
+out:
+    spin_unlock(&ffa_rx_buffer_lock);
+    spin_unlock(&ctx->lock);
+
+    return ret;
+}
+
+static uint32_t handle_rx_release(void)
+{
+    uint32_t ret = FFA_RET_DENIED;
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+
+    spin_lock(&ctx->lock);
+    if ( !ctx->page_count || ctx->tx_is_mine )
+        goto out;
+    ret = FFA_RET_OK;
+    ctx->tx_is_mine = true;
+out:
+    spin_unlock(&ctx->lock);
+
+    return ret;
+}
+
 static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
 {
     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
@@ -528,6 +631,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
     uint32_t fid = get_user_reg(regs, 0);
     struct domain *d = current->domain;
     struct ffa_ctx *ctx = d->arch.tee;
+    uint32_t count;
     int e;
 
     if ( !ctx )
@@ -559,6 +663,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
         else
             set_regs_success(regs, 0, 0);
         return true;
+    case FFA_PARTITION_INFO_GET:
+        e = handle_partition_info_get(get_user_reg(regs, 1),
+                                      get_user_reg(regs, 2),
+                                      get_user_reg(regs, 3),
+                                      get_user_reg(regs, 4),
+                                      get_user_reg(regs, 5), &count);
+        if ( e )
+            set_regs_error(regs, e);
+        else
+            set_regs_success(regs, count, 0);
+        return true;
+    case FFA_RX_RELEASE:
+        e = handle_rx_release();
+        if ( e )
+            set_regs_error(regs, e);
+        else
+            set_regs_success(regs, 0, 0);
+        return true;
     case FFA_MSG_SEND_DIRECT_REQ_32:
 #ifdef CONFIG_ARM_64
     case FFA_MSG_SEND_DIRECT_REQ_64:
-- 
2.34.1

[XEN PATCH v7 15/20] xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h

[Jens Wiklander]

Moves the two helper functions regpair_to_uint64() and
uint64_to_regpair() from xen/arch/arm/tee/optee.c to the common arm
specific regs.h. This enables reuse of these functions in the FF-A
mediator in a subsequent patch.

Reviewed-by: Michal Orzel <michal.orzel@amd.com>
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/include/asm/regs.h | 12 ++++++++++++
 xen/arch/arm/tee/optee.c        | 11 -----------
 2 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/include/asm/regs.h b/xen/arch/arm/include/asm/regs.h
index 0693a681315f..aa39e83ee5f4 100644
--- a/xen/arch/arm/include/asm/regs.h
+++ b/xen/arch/arm/include/asm/regs.h
@@ -60,6 +60,18 @@ static inline bool guest_mode(const struct cpu_user_regs *r)
 register_t get_user_reg(struct cpu_user_regs *regs, int reg);
 void set_user_reg(struct cpu_user_regs *regs, int reg, register_t val);
 
+static inline uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
+{
+    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
+}
+
+static inline void uint64_to_regpair(register_t *reg0, register_t *reg1,
+                                     uint64_t val)
+{
+    *reg0 = val >> 32;
+    *reg1 = (uint32_t)val;
+}
+
 #endif
 
 #endif /* __ARM_REGS_H__ */
diff --git a/xen/arch/arm/tee/optee.c b/xen/arch/arm/tee/optee.c
index 9cb9f16d43cb..47027ecef47c 100644
--- a/xen/arch/arm/tee/optee.c
+++ b/xen/arch/arm/tee/optee.c
@@ -268,17 +268,6 @@ static int optee_domain_init(struct domain *d)
     return 0;
 }
 
-static uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
-{
-    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
-}
-
-static void uint64_to_regpair(register_t *reg0, register_t *reg1, uint64_t val)
-{
-    *reg0 = val >> 32;
-    *reg1 = (uint32_t)val;
-}
-
 static struct page_info *get_domain_ram_page(gfn_t gfn)
 {
     struct page_info *page;
-- 
2.34.1

[XEN PATCH v7 16/20] xen/arm: ffa: add defines for sharing memory

[Jens Wiklander]

Adds defines needed for sharing using the function FFA_MEM_SHARE and
friends.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 57 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 3571817c0bcd..bfd378f7fcd7 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -22,6 +22,14 @@
 #include <asm/tee/ffa.h>
 #include <asm/regs.h>
 
+/*
+ * References:
+ * FF-A-1.0-REL: FF-A specification version 1.0 available at
+ *               https://developer.arm.com/documentation/den0077/a
+ * FF-A-1.1-REL0: FF-A specification version 1.1 available at
+ *                https://developer.arm.com/documentation/den0077/e
+ */
+
 /* Error codes */
 #define FFA_RET_OK                      0
 #define FFA_RET_NOT_SUPPORTED           -1
@@ -66,6 +74,55 @@
  */
 #define FFA_PAGE_SIZE                   SZ_4K
 
+/*
+ * Limit for shared buffer size. Please note that this define limits
+ * number of pages. But user buffer can be not aligned to a page
+ * boundary. So it is possible that user would not be able to share
+ * exactly FFA_MAX_SHM_BUFFER_PG * FFA_PAGE_SIZE bytes.
+ *
+ * FF-A doesn't have any direct requirments on GlobalPlatform or vice
+ * versa, but an implementation can very well use FF-A in order to provide
+ * a GlobalPlatform interface on top.
+ *
+ * Global Platform specification for TEE requires that any TEE
+ * implementation should allow to share buffers with size of at least
+ * 512KB. Due to align issue mentioned above, we need to increase this
+ * value with one.
+ */
+#define FFA_MAX_SHM_PAGE_COUNT          (SZ_512K / FFA_PAGE_SIZE + 1)
+
+/*
+ * Limits the number of shared buffers that guest can have at once. This
+ * is to prevent case, when guests tricks XEN into exhausting its own
+ * memory by allocating many small buffers. This value has been chosen
+ * arbitrary.
+ */
+#define FFA_MAX_SHM_COUNT               32
+
+/* FF-A-1.1-REL0 section 10.9.2 Memory region handle, page 167 */
+#define FFA_HANDLE_HYP_FLAG             BIT(63, ULL)
+#define FFA_HANDLE_INVALID              0xffffffffffffffffULL
+
+/*
+ * The bits for FFA_NORMAL_MEM_REG_ATTR FFA_MEM_ACC_RW below are
+ * defined in FF-A-1.1-REL0 Table 10.18 at page 175.
+ */
+ /* Memory attributes: Normal memory, Write-Back cacheable, Inner shareable */
+#define FFA_NORMAL_MEM_REG_ATTR         0x2fU
+/* Memory access permissions: Read-write */
+#define FFA_MEM_ACC_RW                  0x2U
+
+/* FF-A-1.1-REL0 section 10.11.4 Flags usage, page 184-187 */
+/* Clear memory before mapping in receiver */
+#define FFA_MEMORY_REGION_FLAG_CLEAR            BIT(0, U)
+/* Relayer may time slice this operation */
+#define FFA_MEMORY_REGION_FLAG_TIME_SLICE       BIT(1, U)
+/* Clear memory after receiver relinquishes it */
+#define FFA_MEMORY_REGION_FLAG_CLEAR_RELINQUISH BIT(2, U)
+/* Share memory transaction */
+#define FFA_MEMORY_REGION_TRANSACTION_TYPE_SHARE (1U << 3)
+
+
 /* Framework direct request/response */
 #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
 #define FFA_MSG_TYPE_MASK               0xFFU;
-- 
2.34.1

[XEN PATCH v7 17/20] xen/arm: ffa: add ABI structs for sharing memory

[Jens Wiklander]

Adds the ABI structs used by function FFA_MEM_SHARE and friends for
sharing memory.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 74 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index bfd378f7fcd7..94c90b252454 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -197,6 +197,11 @@
 #define FFA_MSG_SEND                    0x8400006EU
 #define FFA_MSG_POLL                    0x8400006AU
 
+/*
+ * Structs below ending with _1_0 are defined in FF-A-1.0-REL and
+ * struct ending with _1_1 are defined in FF-A-1.1-REL0.
+ */
+
 /* Partition information descriptor */
 struct ffa_partition_info_1_0 {
     uint16_t id;
@@ -211,6 +216,75 @@ struct ffa_partition_info_1_1 {
     uint8_t uuid[16];
 };
 
+/* Constituent memory region descriptor */
+struct ffa_address_range {
+    uint64_t address;
+    uint32_t page_count;
+    uint32_t reserved;
+};
+
+/* Composite memory region descriptor */
+struct ffa_mem_region {
+    uint32_t total_page_count;
+    uint32_t address_range_count;
+    uint64_t reserved;
+    struct ffa_address_range address_range_array[];
+};
+
+/* Memory access permissions descriptor */
+struct ffa_mem_access_perm {
+    uint16_t endpoint_id;
+    uint8_t perm;
+    uint8_t flags;
+};
+
+/* Endpoint memory access descriptor */
+struct ffa_mem_access {
+    struct ffa_mem_access_perm access_perm;
+    uint32_t region_offs;
+    uint64_t reserved;
+};
+
+/* Lend, donate or share memory transaction descriptor */
+struct ffa_mem_transaction_1_0 {
+    uint16_t sender_id;
+    uint8_t mem_reg_attr;
+    uint8_t reserved0;
+    uint32_t flags;
+    uint64_t global_handle;
+    uint64_t tag;
+    uint32_t reserved1;
+    uint32_t mem_access_count;
+    struct ffa_mem_access mem_access_array[];
+};
+
+struct ffa_mem_transaction_1_1 {
+    uint16_t sender_id;
+    uint16_t mem_reg_attr;
+    uint32_t flags;
+    uint64_t global_handle;
+    uint64_t tag;
+    uint32_t mem_access_size;
+    uint32_t mem_access_count;
+    uint32_t mem_access_offs;
+    uint8_t reserved[12];
+};
+
+/* Endpoint RX/TX descriptor */
+struct ffa_endpoint_rxtx_descriptor_1_0 {
+    uint16_t sender_id;
+    uint16_t reserved;
+    uint32_t rx_range_count;
+    uint32_t tx_range_count;
+};
+
+struct ffa_endpoint_rxtx_descriptor_1_1 {
+    uint16_t sender_id;
+    uint16_t reserved;
+    uint32_t rx_region_offs;
+    uint32_t tx_region_offs;
+};
+
 struct ffa_ctx {
     void *rx;
     const void *tx;
-- 
2.34.1

[XEN PATCH v7 18/20] xen/arm: ffa: support sharing memory

[Jens Wiklander]

Adds support for a guest to share memory with an SP using FFA_MEM_SHARE
and FFA_MEM_RECLAIM. Only small memory regions can be shared using a
single call to FFA_MEM_SHARE are supported.

A memory region that doesn't need to be shared any longer can be
reclaimed with FFA_MEM_RECLAIM once the SP doesn't use it any longer.
This is checked by the SPMC and not in control of the mediator.

With this commit we have a FF-A version 1.1 [1] mediator able to
communicate with a Secure Partition in secure world using shared memory.
The secure world must use FF-A version 1.1, but the guest is free to use
version 1.0 or version 1.1.

Adds a check that the SP supports the needed FF-A features
FFA_MEM_SHARE_64 or FFA_MEM_SHARE_32.

[1] https://developer.arm.com/documentation/den0077/latest
Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 491 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 491 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 94c90b252454..cdc286caf62c 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -270,6 +270,38 @@ struct ffa_mem_transaction_1_1 {
     uint8_t reserved[12];
 };
 
+/* Calculate offset of struct ffa_mem_access from start of buffer */
+#define MEM_ACCESS_OFFSET(access_idx) \
+    ( sizeof(struct ffa_mem_transaction_1_1) + \
+      ( access_idx ) * sizeof(struct ffa_mem_access) )
+
+/* Calculate offset of struct ffa_mem_region from start of buffer */
+#define REGION_OFFSET(access_count, region_idx) \
+    ( MEM_ACCESS_OFFSET(access_count) + \
+      ( region_idx ) * sizeof(struct ffa_mem_region) )
+
+/* Calculate offset of struct ffa_address_range from start of buffer */
+#define ADDR_RANGE_OFFSET(access_count, region_count, range_idx) \
+    ( REGION_OFFSET(access_count, region_count) + \
+      ( range_idx ) * sizeof(struct ffa_address_range) )
+
+/*
+ * The parts needed from struct ffa_mem_transaction_1_0 or struct
+ * ffa_mem_transaction_1_1, used to provide an abstraction of difference in
+ * data structures between version 1.0 and 1.1. This is just an internal
+ * interface and can be changed without changing any ABI.
+ */
+struct ffa_mem_transaction_x {
+    uint16_t sender_id;
+    uint8_t mem_reg_attr;
+    uint8_t flags;
+    uint8_t mem_access_size;
+    uint8_t mem_access_count;
+    uint16_t mem_access_offs;
+    uint64_t global_handle;
+    uint64_t tag;
+};
+
 /* Endpoint RX/TX descriptor */
 struct ffa_endpoint_rxtx_descriptor_1_0 {
     uint16_t sender_id;
@@ -294,8 +326,20 @@ struct ffa_ctx {
     uint32_t guest_vers;
     bool tx_is_mine;
     bool interrupted;
+    struct list_head shm_list;
+    unsigned int shm_count;
     spinlock_t lock;
 };
+
+struct ffa_shm_mem {
+    struct list_head list;
+    uint16_t sender_id;
+    uint16_t ep_id;     /* endpoint, the one lending */
+    uint64_t handle;    /* FFA_HANDLE_INVALID if not set yet */
+    unsigned int page_count;
+    struct page_info *pages[];
+};
+
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
@@ -310,6 +354,8 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
  *
  * ffa_page_count is the number of pages used in each of these buffers.
  *
+ * The TX buffer is protected from concurrent usage with ffa_tx_buffer_lock.
+ *
  * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
  * Note that the SPMC is also tracking the ownership of our RX buffer so
  * for calls which uses our RX buffer to deliver a result we must call
@@ -319,6 +365,7 @@ static void *ffa_rx __read_mostly;
 static void *ffa_tx __read_mostly;
 static unsigned int ffa_page_count __read_mostly;
 static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
+static DEFINE_SPINLOCK(ffa_tx_buffer_lock);
 
 static bool ffa_get_version(uint32_t *vers)
 {
@@ -429,6 +476,42 @@ static int32_t ffa_rx_release(void)
     return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
 }
 
+static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
+                             register_t addr, uint32_t pg_count,
+                             uint64_t *handle)
+{
+    struct arm_smccc_1_2_regs arg = {
+        .a0 = FFA_MEM_SHARE_32,
+        .a1 = tot_len,
+        .a2 = frag_len,
+        .a3 = addr,
+        .a4 = pg_count,
+    };
+    struct arm_smccc_1_2_regs resp;
+
+    if ( IS_ENABLED(CONFIG_ARM_64) )
+        arg.a0 = FFA_MEM_SHARE_64;
+
+    arm_smccc_1_2_smc(&arg, &resp);
+
+    switch ( resp.a0 )
+    {
+    case FFA_ERROR:
+        if ( resp.a2 )
+            return resp.a2;
+        else
+            return FFA_RET_NOT_SUPPORTED;
+    case FFA_SUCCESS_32:
+        *handle = regpair_to_uint64(resp.a3, resp.a2);
+        return FFA_RET_OK;
+    case FFA_MEM_FRAG_RX:
+        *handle = regpair_to_uint64(resp.a2, resp.a1);
+        return resp.a3;
+    default:
+        return FFA_RET_NOT_SUPPORTED;
+    }
+}
+
 static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
                                       uint8_t msg)
 {
@@ -757,6 +840,404 @@ out:
              resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
 }
 
+/*
+ * Gets all page and assigns them to the supplied shared memory object. If
+ * this function fails then the caller is still expected to call
+ * put_shm_pages() as a cleanup.
+ */
+static int get_shm_pages(struct domain *d, struct ffa_shm_mem *shm,
+                         const struct ffa_address_range *range,
+                         uint32_t range_count, unsigned int start_page_idx,
+                         unsigned int *last_page_idx)
+{
+    unsigned int pg_idx = start_page_idx;
+    gfn_t gfn;
+    unsigned int n;
+    unsigned int m;
+    p2m_type_t t;
+    uint64_t addr;
+
+    for ( n = 0; n < range_count; n++ )
+    {
+        for ( m = 0; m < range[n].page_count; m++ )
+        {
+            if ( pg_idx >= shm->page_count )
+                return FFA_RET_INVALID_PARAMETERS;
+
+            addr = read_atomic(&range[n].address);
+            gfn = gaddr_to_gfn(addr + m * FFA_PAGE_SIZE);
+            shm->pages[pg_idx] = get_page_from_gfn(d, gfn_x(gfn), &t,
+						   P2M_ALLOC);
+            if ( !shm->pages[pg_idx] )
+                return FFA_RET_DENIED;
+            pg_idx++;
+            /* Only normal RAM for now */
+            if ( !p2m_is_ram(t) )
+                return FFA_RET_DENIED;
+        }
+    }
+
+    *last_page_idx = pg_idx;
+
+    return FFA_RET_OK;
+}
+
+static void put_shm_pages(struct ffa_shm_mem *shm)
+{
+    unsigned int n;
+
+    for ( n = 0; n < shm->page_count && shm->pages[n]; n++ )
+    {
+        put_page(shm->pages[n]);
+        shm->pages[n] = NULL;
+    }
+}
+
+static struct ffa_shm_mem *alloc_ffa_shm_mem(struct ffa_ctx *ctx,
+                                             unsigned int page_count)
+{
+    struct ffa_shm_mem *shm;
+
+    if ( page_count >= FFA_MAX_SHM_PAGE_COUNT ||
+         ctx->shm_count >= FFA_MAX_SHM_COUNT )
+        return NULL;
+
+    shm = xzalloc_flex_struct(struct ffa_shm_mem, pages, page_count);
+    if ( shm )
+    {
+        ctx->shm_count++;
+        shm->page_count = page_count;
+    }
+
+    return shm;
+}
+
+static void free_ffa_shm_mem(struct ffa_ctx *ctx, struct ffa_shm_mem *shm)
+{
+    if ( shm ) {
+        ASSERT(ctx->shm_count > 0);
+        ctx->shm_count--;
+        put_shm_pages(shm);
+        xfree(shm);
+    }
+}
+
+static void init_range(struct ffa_address_range *addr_range,
+                       paddr_t pa)
+{
+    memset(addr_range, 0, sizeof(*addr_range));
+    addr_range->address = pa;
+    addr_range->page_count = 1;
+}
+
+/*
+ * This function uses the ffa_tx buffer to transmit the memory transaction
+ * descriptor. The function depends ffa_tx_buffer_lock to be used to guard
+ * the buffer from concurrent use.
+ */
+static int share_shm(struct ffa_shm_mem *shm)
+{
+    const uint32_t max_frag_len = ffa_page_count * FFA_PAGE_SIZE;
+    struct ffa_mem_access *mem_access_array;
+    struct ffa_mem_transaction_1_1 *descr;
+    struct ffa_address_range *addr_range;
+    struct ffa_mem_region *region_descr;
+    const unsigned int region_count = 1;
+    void *buf = ffa_tx;
+    uint32_t frag_len;
+    uint32_t tot_len;
+    paddr_t last_pa;
+    unsigned int n;
+    paddr_t pa;
+
+    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
+    if ( !shm->page_count )
+    {
+        ASSERT_UNREACHABLE();
+        return FFA_RET_INVALID_PARAMETERS;
+    }
+
+    descr = buf;
+    memset(descr, 0, sizeof(*descr));
+    descr->sender_id = shm->sender_id;
+    descr->global_handle = shm->handle;
+    descr->mem_reg_attr = FFA_NORMAL_MEM_REG_ATTR;
+    descr->mem_access_count = 1;
+    descr->mem_access_size = sizeof(*mem_access_array);
+    descr->mem_access_offs = MEM_ACCESS_OFFSET(0);
+
+    mem_access_array = buf + descr->mem_access_offs;
+    memset(mem_access_array, 0, sizeof(*mem_access_array));
+    mem_access_array[0].access_perm.endpoint_id = shm->ep_id;
+    mem_access_array[0].access_perm.perm = FFA_MEM_ACC_RW;
+    mem_access_array[0].region_offs = REGION_OFFSET(descr->mem_access_count, 0);
+
+    region_descr = buf + mem_access_array[0].region_offs;
+    memset(region_descr, 0, sizeof(*region_descr));
+    region_descr->total_page_count = shm->page_count;
+
+    region_descr->address_range_count = 1;
+    last_pa = page_to_maddr(shm->pages[0]);
+    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
+    {
+        pa = page_to_maddr(shm->pages[n]);
+        if ( last_pa + FFA_PAGE_SIZE == pa )
+            continue;
+        region_descr->address_range_count++;
+    }
+
+    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
+                                region_descr->address_range_count);
+    if ( tot_len > max_frag_len )
+        return FFA_RET_NOT_SUPPORTED;
+
+    addr_range = region_descr->address_range_array;
+    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
+    last_pa = page_to_maddr(shm->pages[0]);
+    init_range(addr_range, last_pa);
+    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
+    {
+        pa = page_to_maddr(shm->pages[n]);
+        if ( last_pa + FFA_PAGE_SIZE == pa )
+        {
+            addr_range->page_count++;
+            continue;
+        }
+
+        frag_len += sizeof(*addr_range);
+        addr_range++;
+        init_range(addr_range, pa);
+    }
+
+    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
+}
+
+static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
+                                struct ffa_mem_transaction_x *trans)
+{
+    uint16_t mem_reg_attr;
+    uint32_t flags;
+    uint32_t count;
+    uint32_t offs;
+    uint32_t size;
+
+    if ( ffa_vers >= FFA_VERSION_1_1 )
+    {
+        const struct ffa_mem_transaction_1_1 *descr;
+
+        if ( blen < sizeof(*descr) )
+            return FFA_RET_INVALID_PARAMETERS;
+
+        descr = buf;
+        trans->sender_id = descr->sender_id;
+        mem_reg_attr = descr->mem_reg_attr;
+        flags = descr->flags;
+        trans->global_handle = descr->global_handle;
+        trans->tag = descr->tag;
+
+        count = descr->mem_access_count;
+        size = descr->mem_access_size;
+        offs = descr->mem_access_offs;
+    }
+    else
+    {
+        const struct ffa_mem_transaction_1_0 *descr;
+
+        if ( blen < sizeof(*descr) )
+            return FFA_RET_INVALID_PARAMETERS;
+
+        descr = buf;
+        trans->sender_id = descr->sender_id;
+        mem_reg_attr = descr->mem_reg_attr;
+        flags = descr->flags;
+        trans->global_handle = descr->global_handle;
+        trans->tag = descr->tag;
+
+        count = descr->mem_access_count;
+        size = sizeof(struct ffa_mem_access);
+        offs = offsetof(struct ffa_mem_transaction_1_0, mem_access_array);
+    }
+    /*
+     * Make sure that "descr" which is shared with the guest isn't accessed
+     * again after this point.
+     */
+    barrier();
+
+    /*
+     * We're doing a rough check to see that no information is lost when
+     * tranfering the values into a struct ffa_mem_transaction_x below. The
+     * fields in struct ffa_mem_transaction_x are wide enough to hold any
+     * valid value so being out of range means that something is wrong.
+     */
+    if ( mem_reg_attr > UINT8_MAX || flags > UINT8_MAX || size > UINT8_MAX ||
+        count > UINT8_MAX || offs > UINT16_MAX )
+        return FFA_RET_INVALID_PARAMETERS;
+
+    /* Check that the endpoint memory access descriptor array fits */
+    if ( size * count + offs > blen )
+        return FFA_RET_INVALID_PARAMETERS;
+
+    trans->mem_reg_attr = mem_reg_attr;
+    trans->flags = flags;
+    trans->mem_access_size = size;
+    trans->mem_access_count = count;
+    trans->mem_access_offs = offs;
+
+    return 0;
+}
+
+static void handle_mem_share(struct cpu_user_regs *regs)
+{
+    uint32_t tot_len = get_user_reg(regs, 1);
+    uint32_t frag_len = get_user_reg(regs, 2);
+    uint64_t addr = get_user_reg(regs, 3);
+    uint32_t page_count = get_user_reg(regs, 4);
+    const struct ffa_mem_region *region_descr;
+    const struct ffa_mem_access *mem_access;
+    struct ffa_mem_transaction_x trans;
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    struct ffa_shm_mem *shm = NULL;
+    unsigned int last_page_idx = 0;
+    register_t handle_hi = 0;
+    register_t handle_lo = 0;
+    int ret = FFA_RET_DENIED;
+    uint32_t range_count;
+    uint32_t region_offs;
+
+    /*
+     * We're only accepting memory transaction descriptors via the rx/tx
+     * buffer.
+     */
+    if ( addr )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_set_ret;
+    }
+
+    /* Check that fragment length doesn't exceed total length */
+    if ( frag_len > tot_len )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out_set_ret;
+    }
+
+    /* We currently only support a single fragment */
+    if ( frag_len != tot_len )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_set_ret;
+    }
+
+    spin_lock(&ctx->lock);
+
+    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
+        goto out_unlock;
+
+    if ( !ffa_page_count )
+    {
+        ret = FFA_RET_NO_MEMORY;
+        goto out_unlock;
+    }
+
+    ret = read_mem_transaction(ctx->guest_vers, ctx->tx, frag_len, &trans);
+    if ( ret )
+        goto out_unlock;
+
+    if ( trans.mem_reg_attr != FFA_NORMAL_MEM_REG_ATTR )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_unlock;
+    }
+
+    /* Only supports sharing it with one SP for now */
+    if ( trans.mem_access_count != 1 )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_unlock;
+    }
+
+    if ( trans.sender_id != get_vm_id(d) )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out_unlock;
+    }
+
+    /* Check that it fits in the supplied data */
+    if ( trans.mem_access_offs + trans.mem_access_size > frag_len )
+        goto out_unlock;
+
+    mem_access = ctx->tx + trans.mem_access_offs;
+    if ( read_atomic(&mem_access->access_perm.perm) != FFA_MEM_ACC_RW )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_unlock;
+    }
+
+    region_offs = read_atomic(&mem_access->region_offs);
+    if ( sizeof(*region_descr) + region_offs > frag_len )
+    {
+        ret = FFA_RET_NOT_SUPPORTED;
+        goto out_unlock;
+    }
+
+    region_descr = ctx->tx + region_offs;
+    range_count = read_atomic(&region_descr->address_range_count);
+    page_count = read_atomic(&region_descr->total_page_count);
+
+    shm = alloc_ffa_shm_mem(ctx, page_count);
+    if ( !shm )
+    {
+        ret = FFA_RET_NO_MEMORY;
+        goto out_unlock;
+    }
+    shm->sender_id = trans.sender_id;
+    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
+
+    /*
+     * Check that the Composite memory region descriptor fits.
+     */
+    if ( sizeof(*region_descr) + region_offs +
+         range_count * sizeof(struct ffa_address_range) > frag_len )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out;
+    }
+
+    ret = get_shm_pages(d, shm, region_descr->address_range_array, range_count,
+                        0, &last_page_idx);
+    if ( ret )
+        goto out;
+    if ( last_page_idx != shm->page_count )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out;
+    }
+
+    /* Note that share_shm() uses our tx buffer */
+    spin_lock(&ffa_tx_buffer_lock);
+    ret = share_shm(shm);
+    spin_unlock(&ffa_tx_buffer_lock);
+    if ( ret )
+        goto out;
+
+    list_add_tail(&shm->list, &ctx->shm_list);
+
+    uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
+
+out:
+    if ( ret )
+        free_ffa_shm_mem(ctx, shm);
+out_unlock:
+    spin_unlock(&ctx->lock);
+
+out_set_ret:
+    if ( ret == 0)
+            set_regs_success(regs, handle_lo, handle_hi);
+    else
+            set_regs_error(regs, ret);
+}
+
 static bool ffa_handle_call(struct cpu_user_regs *regs)
 {
     uint32_t fid = get_user_reg(regs, 0);
@@ -818,6 +1299,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
 #endif
         handle_msg_send_direct_req(regs, fid);
         return true;
+    case FFA_MEM_SHARE_32:
+#ifdef CONFIG_ARM_64
+    case FFA_MEM_SHARE_64:
+#endif
+        handle_mem_share(regs);
+        return true;
 
     default:
         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
@@ -857,6 +1344,8 @@ static int ffa_domain_init(struct domain *d)
         }
     }
 
+    INIT_LIST_HEAD(&ctx->shm_list);
+
     d->arch.tee = ctx;
 
     return 0;
@@ -1012,11 +1501,13 @@ static bool ffa_probe(void)
          !check_mandatory_feature(FFA_RX_RELEASE) ||
 #ifdef CONFIG_ARM_64
          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
+         !check_mandatory_feature(FFA_MEM_SHARE_64) ||
 #endif
 #ifdef CONFIG_ARM_32
          !check_mandatory_feature(FFA_RXTX_MAP_32) ||
 #endif
          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
+         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
         return false;
 
-- 
2.34.1

[XEN PATCH v7 19/20] xen/arm: ffa: add support to reclaim shared memory

[Jens Wiklander]

Adds support to reclaim memory previously shared with FFA_MEM_SHARE.

Adds a check that the SP supports the needed FF-A feature
FFA_MEM_RECLAIM.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 53 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index cdc286caf62c..3557edc455d0 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -512,6 +512,12 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
     }
 }
 
+static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
+                               uint32_t flags)
+{
+    return ffa_simple_call(FFA_MEM_RECLAIM, handle_lo, handle_hi, flags, 0);
+}
+
 static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
                                       uint8_t msg)
 {
@@ -1238,6 +1244,43 @@ out_set_ret:
             set_regs_error(regs, ret);
 }
 
+static int handle_mem_reclaim(uint64_t handle, uint32_t flags)
+{
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    struct ffa_shm_mem *shm;
+    register_t handle_hi;
+    register_t handle_lo;
+    int ret;
+
+    spin_lock(&ctx->lock);
+    list_for_each_entry(shm, &ctx->shm_list, list)
+    {
+        if ( shm->handle == handle )
+            goto found_it;
+    }
+    shm = NULL;
+    ret = FFA_RET_INVALID_PARAMETERS;
+    goto out;
+found_it:
+
+    uint64_to_regpair(&handle_hi, &handle_lo, handle);
+    ret = ffa_mem_reclaim(handle_lo, handle_hi, flags);
+    if ( ret )
+    {
+        shm = NULL;
+        goto out;
+    }
+
+    list_del(&shm->list);
+
+out:
+    free_ffa_shm_mem(ctx, shm);
+    spin_unlock(&ctx->lock);
+
+    return ret;
+}
+
 static bool ffa_handle_call(struct cpu_user_regs *regs)
 {
     uint32_t fid = get_user_reg(regs, 0);
@@ -1305,6 +1348,15 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
 #endif
         handle_mem_share(regs);
         return true;
+    case FFA_MEM_RECLAIM:
+        e = handle_mem_reclaim(regpair_to_uint64(get_user_reg(regs, 2),
+                                                 get_user_reg(regs, 1)),
+                               get_user_reg(regs, 3));
+        if ( e )
+            set_regs_error(regs, e);
+        else
+            set_regs_success(regs, 0, 0);
+        return true;
 
     default:
         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
@@ -1508,6 +1560,7 @@ static bool ffa_probe(void)
 #endif
          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
          !check_mandatory_feature(FFA_MEM_SHARE_32) ||
+         !check_mandatory_feature(FFA_MEM_RECLAIM) ||
          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
         return false;
 
-- 
2.34.1

[XEN PATCH v7 20/20] xen/arm: ffa: support sharing large memory ranges

[Jens Wiklander]

Adds support for sharing large memory ranges transmitted in fragments
using FFA_MEM_FRAG_TX.

The implementation is the bare minimum to be able to communicate with
OP-TEE running as an SPMC at S-EL1.

Adds a check that the SP supports the needed FF-A feature
FFA_MEM_FRAG_TX.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
---
 xen/arch/arm/tee/ffa.c | 254 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 240 insertions(+), 14 deletions(-)

diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
index 3557edc455d0..72c0249d8cad 100644
--- a/xen/arch/arm/tee/ffa.c
+++ b/xen/arch/arm/tee/ffa.c
@@ -326,6 +326,7 @@ struct ffa_ctx {
     uint32_t guest_vers;
     bool tx_is_mine;
     bool interrupted;
+    struct list_head frag_list;
     struct list_head shm_list;
     unsigned int shm_count;
     spinlock_t lock;
@@ -340,6 +341,18 @@ struct ffa_shm_mem {
     struct page_info *pages[];
 };
 
+struct mem_frag_state {
+    struct list_head list;
+    struct ffa_shm_mem *shm;
+    uint32_t range_count;
+    unsigned int current_page_idx;
+    unsigned int frag_offset;
+    unsigned int range_offset;
+    const uint8_t *buf;
+    unsigned int buf_size;
+    struct ffa_address_range range;
+};
+
 /* Negotiated FF-A version to use with the SPMC */
 static uint32_t ffa_version __ro_after_init;
 
@@ -512,6 +525,36 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
     }
 }
 
+static int32_t ffa_mem_frag_tx(uint64_t handle, uint32_t frag_len,
+                               uint16_t sender_id)
+{
+    struct arm_smccc_1_2_regs arg = {
+        .a0 = FFA_MEM_FRAG_TX,
+        .a1 = handle & UINT32_MAX,
+        .a2 = handle >> 32,
+        .a3 = frag_len,
+        .a4 = (uint32_t)sender_id << 16,
+    };
+    struct arm_smccc_1_2_regs resp;
+
+    arm_smccc_1_2_smc(&arg, &resp);
+
+    switch ( resp.a0 )
+    {
+    case FFA_ERROR:
+        if ( resp.a2 )
+            return resp.a2;
+        else
+            return FFA_RET_NOT_SUPPORTED;
+    case FFA_SUCCESS_32:
+        return FFA_RET_OK;
+    case FFA_MEM_FRAG_RX:
+        return resp.a3;
+    default:
+            return FFA_RET_NOT_SUPPORTED;
+    }
+}
+
 static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
                                uint32_t flags)
 {
@@ -586,6 +629,14 @@ static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
     set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
 }
 
+static void set_regs_frag_rx(struct cpu_user_regs *regs, uint32_t handle_lo,
+                             uint32_t handle_hi, uint32_t frag_offset,
+                             uint16_t sender_id)
+{
+    set_regs(regs, FFA_MEM_FRAG_RX, handle_lo, handle_hi, frag_offset,
+             (uint32_t)sender_id << 16, 0, 0, 0);
+}
+
 static void handle_version(struct cpu_user_regs *regs)
 {
     struct domain *d = current->domain;
@@ -955,6 +1006,8 @@ static int share_shm(struct ffa_shm_mem *shm)
     paddr_t last_pa;
     unsigned int n;
     paddr_t pa;
+    bool first;
+    int ret;
 
     ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
     if ( !shm->page_count )
@@ -994,13 +1047,23 @@ static int share_shm(struct ffa_shm_mem *shm)
 
     tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
                                 region_descr->address_range_count);
-    if ( tot_len > max_frag_len )
-        return FFA_RET_NOT_SUPPORTED;
 
+    /*
+     * Sharing memory with secure world may have to be done with multiple
+     * calls depending on how many address ranges will be needed. If we're
+     * sharing physically contiguous memory we will only need one range but
+     * we will also need to deal with the worst case where all physical
+     * pages are non-contiguous. For the first batch of address ranges we
+     * call ffa_mem_share() and for all that follows ffa_mem_frag_tx().
+     *
+     * We use frag_len to keep track of how far into the transmit buffer we
+     * have gone.
+     */
     addr_range = region_descr->address_range_array;
     frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
     last_pa = page_to_maddr(shm->pages[0]);
     init_range(addr_range, last_pa);
+    first = true;
     for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
     {
         pa = page_to_maddr(shm->pages[n]);
@@ -1010,12 +1073,34 @@ static int share_shm(struct ffa_shm_mem *shm)
             continue;
         }
 
-        frag_len += sizeof(*addr_range);
-        addr_range++;
+        if ( frag_len == max_frag_len )
+        {
+            if ( first )
+            {
+                ret = ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
+                first = false;
+            }
+            else
+            {
+                ret = ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
+            }
+            if ( ret <= 0 )
+                return ret;
+            frag_len = sizeof(*addr_range);
+            addr_range = buf;
+        }
+        else
+        {
+            frag_len += sizeof(*addr_range);
+            addr_range++;
+        }
         init_range(addr_range, pa);
     }
 
-    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
+    if ( first )
+        return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
+    else
+        return ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
 }
 
 static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
@@ -1092,8 +1177,53 @@ static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
     return 0;
 }
 
+static int add_mem_share_frag(struct mem_frag_state *s, unsigned int offs,
+                              unsigned int frag_len)
+{
+    struct domain *d = current->domain;
+    unsigned int o = offs;
+    unsigned int l;
+    int ret;
+
+    if ( frag_len < o )
+        return FFA_RET_INVALID_PARAMETERS;
+
+    /* Fill up the first struct ffa_address_range */
+    l = min_t(unsigned int, frag_len - o, sizeof(s->range) - s->range_offset);
+    memcpy((uint8_t *)&s->range + s->range_offset, s->buf + o, l);
+    s->range_offset += l;
+    o += l;
+    if ( s->range_offset != sizeof(s->range) )
+        goto out;
+    s->range_offset = 0;
+
+    while ( true )
+    {
+        ret = get_shm_pages(d, s->shm, &s->range, 1, s->current_page_idx,
+                            &s->current_page_idx);
+        if ( ret )
+            return ret;
+        if ( s->range_count == 1 )
+            return 0;
+        s->range_count--;
+        if ( frag_len - o < sizeof(s->range) )
+            break;
+        memcpy(&s->range, s->buf + o, sizeof(s->range));
+        o += sizeof(s->range);
+    }
+
+    /* Collect any remaining bytes for the next struct ffa_address_range */
+    s->range_offset = frag_len - o;
+    memcpy(&s->range, s->buf + o, frag_len - o);
+out:
+    s->frag_offset += frag_len;
+
+    return s->frag_offset;
+}
+
 static void handle_mem_share(struct cpu_user_regs *regs)
 {
+    static uint64_t next_handle = FFA_HANDLE_HYP_FLAG;
     uint32_t tot_len = get_user_reg(regs, 1);
     uint32_t frag_len = get_user_reg(regs, 2);
     uint64_t addr = get_user_reg(regs, 3);
@@ -1128,13 +1258,6 @@ static void handle_mem_share(struct cpu_user_regs *regs)
         goto out_set_ret;
     }
 
-    /* We currently only support a single fragment */
-    if ( frag_len != tot_len )
-    {
-        ret = FFA_RET_NOT_SUPPORTED;
-        goto out_set_ret;
-    }
-
     spin_lock(&ctx->lock);
 
     if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
@@ -1195,11 +1318,41 @@ static void handle_mem_share(struct cpu_user_regs *regs)
     if ( !shm )
     {
         ret = FFA_RET_NO_MEMORY;
-        goto out_unlock;
+        goto out;
     }
     shm->sender_id = trans.sender_id;
     shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
 
+    if ( frag_len != tot_len )
+    {
+        struct mem_frag_state *s = xzalloc(struct mem_frag_state);
+
+        if ( !s )
+        {
+            ret = FFA_RET_NO_MEMORY;
+            goto out;
+        }
+        s->shm = shm;
+        s->range_count = range_count;
+        s->buf = ctx->tx;
+        s->buf_size = ffa_page_count * FFA_PAGE_SIZE;
+        ret = add_mem_share_frag(s, sizeof(*region_descr)  + region_offs,
+                                 frag_len);
+        if ( ret <= 0 )
+        {
+            xfree(s);
+            if ( ret < 0 )
+                goto out;
+        }
+        else
+        {
+            shm->handle = next_handle++;
+            uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
+            list_add_tail(&s->list, &ctx->frag_list);
+        }
+        goto out_unlock;
+    }
+
     /*
      * Check that the Composite memory region descriptor fits.
      */
@@ -1238,7 +1391,75 @@ out_unlock:
     spin_unlock(&ctx->lock);
 
 out_set_ret:
-    if ( ret == 0)
+    if ( ret > 0 )
+            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, trans.sender_id);
+    else if ( ret == 0)
+            set_regs_success(regs, handle_lo, handle_hi);
+    else
+            set_regs_error(regs, ret);
+}
+
+static struct mem_frag_state *find_frag_state(struct ffa_ctx *ctx,
+                                              uint64_t handle)
+{
+    struct mem_frag_state *s;
+
+    list_for_each_entry(s, &ctx->frag_list, list)
+        if ( s->shm->handle == handle )
+            return s;
+
+    return NULL;
+}
+
+static void handle_mem_frag_tx(struct cpu_user_regs *regs)
+{
+    struct domain *d = current->domain;
+    struct ffa_ctx *ctx = d->arch.tee;
+    uint32_t frag_len = get_user_reg(regs, 3);
+    uint32_t handle_lo = get_user_reg(regs, 1);
+    uint32_t handle_hi = get_user_reg(regs, 2);
+    uint64_t handle = regpair_to_uint64(handle_hi, handle_lo);
+    struct mem_frag_state *s;
+    uint16_t sender_id = 0;
+    int ret;
+
+    spin_lock(&ctx->lock);
+    s = find_frag_state(ctx, handle);
+    if ( !s )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out;
+    }
+    sender_id = s->shm->sender_id;
+
+    if ( frag_len > s->buf_size )
+    {
+        ret = FFA_RET_INVALID_PARAMETERS;
+        goto out;
+    }
+
+    ret = add_mem_share_frag(s, 0, frag_len);
+    if ( ret == 0 )
+    {
+        /* Note that share_shm() uses our tx buffer */
+        spin_lock(&ffa_tx_buffer_lock);
+        ret = share_shm(s->shm);
+        spin_unlock(&ffa_tx_buffer_lock);
+        if ( ret == 0 )
+            list_add_tail(&s->shm->list, &ctx->shm_list);
+        else
+            free_ffa_shm_mem(ctx, s->shm);
+    }
+    else if ( ret < 0 )
+        free_ffa_shm_mem(ctx, s->shm);
+    list_del(&s->list);
+    xfree(s);
+out:
+    spin_unlock(&ctx->lock);
+
+    if ( ret > 0 )
+            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, sender_id);
+    else if ( ret == 0)
             set_regs_success(regs, handle_lo, handle_hi);
     else
             set_regs_error(regs, ret);
@@ -1357,6 +1578,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
         else
             set_regs_success(regs, 0, 0);
         return true;
+    case FFA_MEM_FRAG_TX:
+        handle_mem_frag_tx(regs);
+        return true;
 
     default:
         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
@@ -1396,6 +1620,7 @@ static int ffa_domain_init(struct domain *d)
         }
     }
 
+    INIT_LIST_HEAD(&ctx->frag_list);
     INIT_LIST_HEAD(&ctx->shm_list);
 
     d->arch.tee = ctx;
@@ -1560,6 +1785,7 @@ static bool ffa_probe(void)
 #endif
          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
          !check_mandatory_feature(FFA_MEM_SHARE_32) ||
+         !check_mandatory_feature(FFA_MEM_FRAG_TX) ||
          !check_mandatory_feature(FFA_MEM_RECLAIM) ||
          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
         return false;
-- 
2.34.1

Re: [XEN PATCH v7 01/20] xen/arm: smccc: add support for SMCCCv1.2 extended input/output registers

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:32, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> SMCCC v1.2 [1] AArch64 allows x0-x17 to be used as both parameter
> registers and result registers for the SMC and HVC instructions.
> 
> Arm Firmware Framework for Armv8-A specification makes use of x0-x7 as
> parameter and result registers.
> 
> Let us add new interface to support this extended set of input/output
> registers.
> 
> This is based on 3fdc0cb59d97 ("arm64: smccc: Add support for SMCCCv1.2
> extended input/output registers") by Sudeep Holla from the Linux kernel
> 
> The SMCCC version reported to the VM is bumped to 1.2 in order to support
> handling FF-A messages.
> 
> [1] https://developer.arm.com/documentation/den0028/c/?lang=en
> 
> Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> xen/arch/arm/arm64/asm-offsets.c |  9 +++++++
> xen/arch/arm/arm64/smc.S         | 42 ++++++++++++++++++++++++++++++++
> xen/arch/arm/include/asm/smccc.h | 40 ++++++++++++++++++++++++++++++
> xen/arch/arm/vsmc.c              |  2 +-
> 4 files changed, 92 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/arm64/asm-offsets.c b/xen/arch/arm/arm64/asm-offsets.c
> index 7226cd9b2eb0..7adb67a1b81a 100644
> --- a/xen/arch/arm/arm64/asm-offsets.c
> +++ b/xen/arch/arm/arm64/asm-offsets.c
> @@ -57,6 +57,15 @@ void __dummy__(void)
>    BLANK();
>    OFFSET(SMCCC_RES_a0, struct arm_smccc_res, a0);
>    OFFSET(SMCCC_RES_a2, struct arm_smccc_res, a2);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X0_OFFS, struct arm_smccc_1_2_regs, a0);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X2_OFFS, struct arm_smccc_1_2_regs, a2);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X4_OFFS, struct arm_smccc_1_2_regs, a4);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X6_OFFS, struct arm_smccc_1_2_regs, a6);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X8_OFFS, struct arm_smccc_1_2_regs, a8);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X10_OFFS, struct arm_smccc_1_2_regs, a10);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X12_OFFS, struct arm_smccc_1_2_regs, a12);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X14_OFFS, struct arm_smccc_1_2_regs, a14);
> +   OFFSET(ARM_SMCCC_1_2_REGS_X16_OFFS, struct arm_smccc_1_2_regs, a16);
> }
> 
> /*
> diff --git a/xen/arch/arm/arm64/smc.S b/xen/arch/arm/arm64/smc.S
> index 91bae62dd4d2..fc6b676e2ee3 100644
> --- a/xen/arch/arm/arm64/smc.S
> +++ b/xen/arch/arm/arm64/smc.S
> @@ -27,3 +27,45 @@ ENTRY(__arm_smccc_1_0_smc)
>         stp     x2, x3, [x4, #SMCCC_RES_a2]
> 1:
>         ret
> +
> +/*
> + * void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
> + *                        struct arm_smccc_1_2_regs *res)
> + */
> +ENTRY(arm_smccc_1_2_smc)
> +    /* Save `res` and free a GPR that won't be clobbered by SMC call */
> +    stp     x1, x19, [sp, #-16]!
> +
> +    /* Ensure `args` won't be clobbered while loading regs in next step */
> +    mov x19, x0
> +
> +    /* Load the registers x0 - x17 from the struct arm_smccc_1_2_regs */
> +    ldp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
> +    ldp x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
> +    ldp x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
> +    ldp x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
> +    ldp x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
> +    ldp x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
> +    ldp x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
> +    ldp x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
> +    ldp x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
> +
> +    smc #0
> +
> +    /* Load the `res` from the stack */
> +    ldr x19, [sp]
> +
> +    /* Store the registers x0 - x17 into the result structure */
> +    stp x0, x1, [x19, #ARM_SMCCC_1_2_REGS_X0_OFFS]
> +    stp x2, x3, [x19, #ARM_SMCCC_1_2_REGS_X2_OFFS]
> +    stp x4, x5, [x19, #ARM_SMCCC_1_2_REGS_X4_OFFS]
> +    stp x6, x7, [x19, #ARM_SMCCC_1_2_REGS_X6_OFFS]
> +    stp x8, x9, [x19, #ARM_SMCCC_1_2_REGS_X8_OFFS]
> +    stp x10, x11, [x19, #ARM_SMCCC_1_2_REGS_X10_OFFS]
> +    stp x12, x13, [x19, #ARM_SMCCC_1_2_REGS_X12_OFFS]
> +    stp x14, x15, [x19, #ARM_SMCCC_1_2_REGS_X14_OFFS]
> +    stp x16, x17, [x19, #ARM_SMCCC_1_2_REGS_X16_OFFS]
> +
> +    /* Restore original x19 */
> +    ldp     xzr, x19, [sp], #16
> +    ret
> diff --git a/xen/arch/arm/include/asm/smccc.h b/xen/arch/arm/include/asm/smccc.h
> index b3dbeecc90ad..1adcd37443c7 100644
> --- a/xen/arch/arm/include/asm/smccc.h
> +++ b/xen/arch/arm/include/asm/smccc.h
> @@ -33,6 +33,7 @@
> 
> #define ARM_SMCCC_VERSION_1_0   SMCCC_VERSION(1, 0)
> #define ARM_SMCCC_VERSION_1_1   SMCCC_VERSION(1, 1)
> +#define ARM_SMCCC_VERSION_1_2   SMCCC_VERSION(1, 2)
> 
> /*
>  * This file provides common defines for ARM SMC Calling Convention as
> @@ -265,6 +266,45 @@ void __arm_smccc_1_0_smc(register_t a0, register_t a1, register_t a2,
>         else                                                    \
>             arm_smccc_1_0_smc(__VA_ARGS__);                     \
>     } while ( 0 )
> +
> +/*
> + * struct arm_smccc_1_2_regs - Arguments for or Results from SMC call
> + * @a0-a17 argument values from registers 0 to 17
> + */
> +struct arm_smccc_1_2_regs {
> +    unsigned long a0;
> +    unsigned long a1;
> +    unsigned long a2;
> +    unsigned long a3;
> +    unsigned long a4;
> +    unsigned long a5;
> +    unsigned long a6;
> +    unsigned long a7;
> +    unsigned long a8;
> +    unsigned long a9;
> +    unsigned long a10;
> +    unsigned long a11;
> +    unsigned long a12;
> +    unsigned long a13;
> +    unsigned long a14;
> +    unsigned long a15;
> +    unsigned long a16;
> +    unsigned long a17;
> +};
> +
> +/*
> + * arm_smccc_1_2_smc() - make SMC calls
> + * @args: arguments passed via struct arm_smccc_1_2_regs
> + * @res: result values via struct arm_smccc_1_2_regs
> + *
> + * This function is used to make SMC calls following SMC Calling Convention
> + * v1.2 or above. The content of the supplied param are copied from the
> + * structure to registers prior to the SMC instruction. The return values
> + * are updated with the content from registers on return from the SMC
> + * instruction.
> + */
> +void arm_smccc_1_2_smc(const struct arm_smccc_1_2_regs *args,
> +                       struct arm_smccc_1_2_regs *res);
> #endif /* CONFIG_ARM_64 */
> 
> #endif /* __ASSEMBLY__ */
> diff --git a/xen/arch/arm/vsmc.c b/xen/arch/arm/vsmc.c
> index 7335276f3fa1..cd68fa80e98a 100644
> --- a/xen/arch/arm/vsmc.c
> +++ b/xen/arch/arm/vsmc.c
> @@ -85,7 +85,7 @@ static bool handle_arch(struct cpu_user_regs *regs)
>     switch ( fid )
>     {
>     case ARM_SMCCC_VERSION_FID:
> -        set_user_reg(regs, 0, ARM_SMCCC_VERSION_1_1);
> +        set_user_reg(regs, 0, ARM_SMCCC_VERSION_1_2);
>         return true;
> 
>     case ARM_SMCCC_ARCH_FEATURES_FID:
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 01/20] xen/arm: smccc: add support for SMCCCv1.2 extended input/output registers

[Julien Grall]

Hi Jens,

On 22/02/2023 15:32, Jens Wiklander wrote:
> SMCCC v1.2 [1] AArch64 allows x0-x17 to be used as both parameter
> registers and result registers for the SMC and HVC instructions.
> 
> Arm Firmware Framework for Armv8-A specification makes use of x0-x7 as
> parameter and result registers.
> 
> Let us add new interface to support this extended set of input/output
> registers.
> 
> This is based on 3fdc0cb59d97 ("arm64: smccc: Add support for SMCCCv1.2
> extended input/output registers") by Sudeep Holla from the Linux kernel
> 
> The SMCCC version reported to the VM is bumped to 1.2 in order to support
> handling FF-A messages.
> 
> [1] https://developer.arm.com/documentation/den0028/c/?lang=en
> 
> Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

One remark here. The tags are usually added chronologically. So your 
signed-off-by should be first.

I am not planning to commit this patch until one of the patch using the 
call is ready. If this happen in this version, then I am happy to fix it 
on commit. Otherwise, please do it in the next version:

Acked-by: Julien Grall <jgrall@amazon.com>

Cheers,

-- 
Julien Grall

Re: [XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Julien Grall]

Hi,

I have only skimmed through the patch so far and I have one question below.

On 22/02/2023 15:32, Jens Wiklander wrote:
> Adds a FF-A version 1.1 [1] mediator to communicate with a Secure
> Partition in secure world.
> 
> This commit brings in only the parts needed to negotiate FF-A version
> number with guest and SPMC.
> 
> [1] https://developer.arm.com/documentation/den0077/e
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
>   xen/arch/arm/include/asm/domain.h  |   2 +-
>   xen/arch/arm/include/asm/psci.h    |   4 +
>   xen/arch/arm/include/asm/tee/ffa.h |  35 +++++
>   xen/arch/arm/tee/Kconfig           |  11 ++
>   xen/arch/arm/tee/Makefile          |   1 +
>   xen/arch/arm/tee/ffa.c             | 217 +++++++++++++++++++++++++++++
>   xen/arch/arm/vsmc.c                |  17 ++-
>   xen/include/public/arch-arm.h      |   1 +
>   8 files changed, 284 insertions(+), 4 deletions(-)
>   create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
>   create mode 100644 xen/arch/arm/tee/ffa.c
> 
> diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
> index 0e310601e846..754daa8efa04 100644
> --- a/xen/arch/arm/include/asm/domain.h
> +++ b/xen/arch/arm/include/asm/domain.h
> @@ -110,7 +110,7 @@ struct arch_domain
>       struct vpl011 vpl011;
>   #endif
>   
> -#ifdef CONFIG_TEE
> +#if defined(CONFIG_TEE) || defined(CONFIG_FFA)
I am confused, AFAICT, you are implementing FFA using TEE, so doesn't 
this mean that CONFIG_FFA depends on CONFIG_TEE?

If so, then you should not need the extra defined here.

>       void *tee;
>   #endif

-- 
Julien Grall

Re: [XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:32, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds a FF-A version 1.1 [1] mediator to communicate with a Secure
> Partition in secure world.
> 
> This commit brings in only the parts needed to negotiate FF-A version
> number with guest and SPMC.
> 
> [1] https://developer.arm.com/documentation/den0077/e
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

Just some minor comments from my side after.

> ---
> xen/arch/arm/include/asm/domain.h  |   2 +-
> xen/arch/arm/include/asm/psci.h    |   4 +
> xen/arch/arm/include/asm/tee/ffa.h |  35 +++++
> xen/arch/arm/tee/Kconfig           |  11 ++
> xen/arch/arm/tee/Makefile          |   1 +
> xen/arch/arm/tee/ffa.c             | 217 +++++++++++++++++++++++++++++
> xen/arch/arm/vsmc.c                |  17 ++-
> xen/include/public/arch-arm.h      |   1 +
> 8 files changed, 284 insertions(+), 4 deletions(-)
> create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
> create mode 100644 xen/arch/arm/tee/ffa.c
> 
> diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
> index 0e310601e846..754daa8efa04 100644
> --- a/xen/arch/arm/include/asm/domain.h
> +++ b/xen/arch/arm/include/asm/domain.h
> @@ -110,7 +110,7 @@ struct arch_domain
>     struct vpl011 vpl011;
> #endif
> 
> -#ifdef CONFIG_TEE
> +#if defined(CONFIG_TEE) || defined(CONFIG_FFA)
>     void *tee;
> #endif
> 
> diff --git a/xen/arch/arm/include/asm/psci.h b/xen/arch/arm/include/asm/psci.h
> index 832f77afff3a..4780972621bb 100644
> --- a/xen/arch/arm/include/asm/psci.h
> +++ b/xen/arch/arm/include/asm/psci.h
> @@ -24,6 +24,10 @@ void call_psci_cpu_off(void);
> void call_psci_system_off(void);
> void call_psci_system_reset(void);
> 
> +/* Range of allocated PSCI function numbers */
> +#define PSCI_FNUM_MIN_VALUE                 _AC(0,U)
> +#define PSCI_FNUM_MAX_VALUE                 _AC(0x1f,U)
> +
> /* PSCI v0.2 interface */
> #define PSCI_0_2_FN32(nr) ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,             \
>                                              ARM_SMCCC_CONV_32,               \
> diff --git a/xen/arch/arm/include/asm/tee/ffa.h b/xen/arch/arm/include/asm/tee/ffa.h
> new file mode 100644
> index 000000000000..94960100718e
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/tee/ffa.h
> @@ -0,0 +1,35 @@
> +/* SPDX-License-Identifier: MIT */

Why is this file in MIT license ?

Any chance to set it as BSD-2 or maybe no license at all (most private headers do not have one) ?

> +/*
> + * xen/arch/arm/include/asm/tee/ffa.h
> + *
> + * Arm Firmware Framework for ARMv8-A(FFA) mediator
> + *
> + * Copyright (C) 2023  Linaro Limited
> + */
> +
> +#ifndef __ASM_ARM_TEE_FFA_H__
> +#define __ASM_ARM_TEE_FFA_H__
> +
> +#include <xen/const.h>
> +#include <xen/kconfig.h>
> +
> +#include <asm/smccc.h>
> +#include <asm/types.h>
> +
> +#define FFA_FNUM_MIN_VALUE              _AC(0x60,U)
> +#define FFA_FNUM_MAX_VALUE              _AC(0x86,U)
> +
> +static inline bool is_ffa_fid(uint32_t fid)
> +{
> +    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
> +
> +    return fn >= FFA_FNUM_MIN_VALUE && fn <= FFA_FNUM_MAX_VALUE;
> +}
> +
> +#ifdef CONFIG_FFA
> +#define FFA_NR_FUNCS    (9 + (IS_ENABLED(CONFIG_ARM_64) ? 3 : 0) )

I am not quite sure of the usage of IS_ENABLED for a constant

Maybe just doing:
#ifdef CONFIG_FFA
#ifdef CONFIG_ARM_64
#define FFA_NR_FUNCS   12
#else
#define FFA_NR_FUNCS    9
#endif
#else
#define FFA_NR_FUNCS    0
#endif

> +#else
> +#define FFA_NR_FUNCS    0
> +#endif
> +
> +#endif /*__ASM_ARM_TEE_FFA_H__*/
> diff --git a/xen/arch/arm/tee/Kconfig b/xen/arch/arm/tee/Kconfig
> index 392169b2559d..923f08ba8cb7 100644
> --- a/xen/arch/arm/tee/Kconfig
> +++ b/xen/arch/arm/tee/Kconfig
> @@ -8,3 +8,14 @@ config OPTEE
>  virtualization-enabled OP-TEE present. You can learn more
>  about virtualization for OP-TEE at
>  https://optee.readthedocs.io/architecture/virtualization.html
> +
> +config FFA
> + bool "Enable FF-A mediator support (UNSUPPORTED)" if UNSUPPORTED
> + default n
> + depends on ARM_64
> + help
> +  This option enables a minimal FF-A mediator. The mediator is
> +  generic as it follows the FF-A specification [1], but it only
> +  implements a small subset of the specification.
> +
> +  [1] https://developer.arm.com/documentation/den0077/latest
> diff --git a/xen/arch/arm/tee/Makefile b/xen/arch/arm/tee/Makefile
> index 982c87968447..58a1015e40e0 100644
> --- a/xen/arch/arm/tee/Makefile
> +++ b/xen/arch/arm/tee/Makefile
> @@ -1,2 +1,3 @@
> +obj-$(CONFIG_FFA) += ffa.o
> obj-y += tee.o
> obj-$(CONFIG_OPTEE) += optee.o
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> new file mode 100644
> index 000000000000..824153c9303a
> --- /dev/null
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -0,0 +1,217 @@
> +/* SPDX-License-Identifier: GPL-2.0 */

I think now you have to say either GPL-2.0-only or GPL-2.0-or-later

> +/*
> + * xen/arch/arm/tee/ffa.c
> + *
> + * Arm Firmware Framework for ARMv8-A (FF-A) mediator
> + *
> + * Copyright (C) 2023  Linaro Limited
> + */
> +
> +#include <xen/domain_page.h>
> +#include <xen/errno.h>
> +#include <xen/init.h>
> +#include <xen/lib.h>
> +#include <xen/sched.h>
> +#include <xen/types.h>
> +#include <xen/sizes.h>
> +#include <xen/bitops.h>

Please try to put the headers in alphabetical order.

> +
> +#include <asm/smccc.h>
> +#include <asm/event.h>
> +#include <asm/tee/tee.h>
> +#include <asm/tee/ffa.h>
> +#include <asm/regs.h>

Same here

The rest looks ok to me.

Cheers
Bertrand

> +
> +/* Error codes */
> +#define FFA_RET_OK                      0
> +#define FFA_RET_NOT_SUPPORTED           -1
> +#define FFA_RET_INVALID_PARAMETERS      -2
> +#define FFA_RET_NO_MEMORY               -3
> +#define FFA_RET_BUSY                    -4
> +#define FFA_RET_INTERRUPTED             -5
> +#define FFA_RET_DENIED                  -6
> +#define FFA_RET_RETRY                   -7
> +#define FFA_RET_ABORTED                 -8
> +
> +/* FFA_VERSION helpers */
> +#define FFA_VERSION_MAJOR_SHIFT         16U
> +#define FFA_VERSION_MAJOR_MASK          0x7FFFU
> +#define FFA_VERSION_MINOR_SHIFT         0U
> +#define FFA_VERSION_MINOR_MASK          0xFFFFU
> +#define MAKE_FFA_VERSION(major, minor)  \
> +        ((((major) & FFA_VERSION_MAJOR_MASK) << FFA_VERSION_MAJOR_SHIFT) | \
> +         ((minor) & FFA_VERSION_MINOR_MASK))
> +
> +#define FFA_MIN_VERSION         MAKE_FFA_VERSION(1, 0)
> +#define FFA_VERSION_1_0         MAKE_FFA_VERSION(1, 0)
> +#define FFA_VERSION_1_1         MAKE_FFA_VERSION(1, 1)
> +
> +/*
> + * This is the version we want to use in communication with guests and SPs.
> + * During negotiation with a guest or a SP we may need to lower it for
> + * that particular guest or SP.
> + */
> +#define FFA_MY_VERSION_MAJOR    1U
> +#define FFA_MY_VERSION_MINOR    1U
> +#define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
> +                                                 FFA_MY_VERSION_MINOR)
> +
> +/* Function IDs */
> +#define FFA_ERROR                       0x84000060U
> +#define FFA_SUCCESS_32                  0x84000061U
> +#define FFA_VERSION                     0x84000063U
> +
> +struct ffa_ctx {
> +    uint32_t guest_vers;
> +};
> +
> +/* Negotiated FF-A version to use with the SPMC */
> +static uint32_t ffa_version __ro_after_init;
> +
> +static bool ffa_get_version(uint32_t *vers)
> +{
> +    const struct arm_smccc_1_2_regs arg = {
> +        .a0 = FFA_VERSION,
> +        .a1 = FFA_MY_VERSION,
> +    };
> +    struct arm_smccc_1_2_regs resp;
> +
> +    arm_smccc_1_2_smc(&arg, &resp);
> +    if ( resp.a0 == FFA_RET_NOT_SUPPORTED )
> +    {
> +        gprintk(XENLOG_ERR, "ffa: FFA_VERSION returned not supported\n");
> +        return false;
> +    }
> +
> +    *vers = resp.a0;
> +
> +    return true;
> +}
> +
> +static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> +                     register_t v2, register_t v3, register_t v4, register_t v5,
> +                     register_t v6, register_t v7)
> +{
> +        set_user_reg(regs, 0, v0);
> +        set_user_reg(regs, 1, v1);
> +        set_user_reg(regs, 2, v2);
> +        set_user_reg(regs, 3, v3);
> +        set_user_reg(regs, 4, v4);
> +        set_user_reg(regs, 5, v5);
> +        set_user_reg(regs, 6, v6);
> +        set_user_reg(regs, 7, v7);
> +}
> +
> +static void handle_version(struct cpu_user_regs *regs)
> +{
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    uint32_t vers = get_user_reg(regs, 1);
> +
> +    if ( vers < FFA_VERSION_1_1 )
> +        vers = FFA_VERSION_1_0;
> +    else
> +        vers = FFA_VERSION_1_1;
> +
> +    ctx->guest_vers = vers;
> +    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> +}
> +
> +static bool ffa_handle_call(struct cpu_user_regs *regs)
> +{
> +    uint32_t fid = get_user_reg(regs, 0);
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +
> +    if ( !ctx )
> +        return false;
> +
> +    switch ( fid )
> +    {
> +    case FFA_VERSION:
> +        handle_version(regs);
> +        return true;
> +
> +    default:
> +        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> +        return false;
> +    }
> +}
> +
> +static int ffa_domain_init(struct domain *d)
> +{
> +    struct ffa_ctx *ctx;
> +
> +    if ( !ffa_version )
> +        return -ENODEV;
> +
> +    ctx = xzalloc(struct ffa_ctx);
> +    if ( !ctx )
> +        return -ENOMEM;
> +
> +    d->arch.tee = ctx;
> +
> +    return 0;
> +}
> +
> +/* This function is supposed to undo what ffa_domain_init() has done */
> +static int ffa_relinquish_resources(struct domain *d)
> +{
> +    struct ffa_ctx *ctx = d->arch.tee;
> +
> +    if ( !ctx )
> +        return 0;
> +
> +    XFREE(d->arch.tee);
> +
> +    return 0;
> +}
> +
> +static bool ffa_probe(void)
> +{
> +    uint32_t vers;
> +    unsigned int major_vers;
> +    unsigned int minor_vers;
> +
> +    /*
> +     * psci_init_smccc() updates this value with what's reported by EL-3
> +     * or secure world.
> +     */
> +    if ( smccc_ver < ARM_SMCCC_VERSION_1_2 )
> +    {
> +        printk(XENLOG_ERR
> +               "ffa: unsupported SMCCC version %#x (need at least %#x)\n",
> +               smccc_ver, ARM_SMCCC_VERSION_1_2);
> +        return false;
> +    }
> +
> +    if ( !ffa_get_version(&vers) )
> +        return false;
> +
> +    if ( vers < FFA_MIN_VERSION || vers > FFA_MY_VERSION )
> +    {
> +        printk(XENLOG_ERR "ffa: Incompatible version %#x found\n", vers);
> +        return false;
> +    }
> +
> +    major_vers = (vers >> FFA_VERSION_MAJOR_SHIFT) & FFA_VERSION_MAJOR_MASK;
> +    minor_vers = vers & FFA_VERSION_MINOR_MASK;
> +    printk(XENLOG_INFO "ARM FF-A Mediator version %u.%u\n",
> +           FFA_MY_VERSION_MAJOR, FFA_MY_VERSION_MINOR);
> +    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> +           major_vers, minor_vers);
> +
> +    ffa_version = vers;
> +
> +    return true;
> +}
> +
> +static const struct tee_mediator_ops ffa_ops =
> +{
> +    .probe = ffa_probe,
> +    .domain_init = ffa_domain_init,
> +    .relinquish_resources = ffa_relinquish_resources,
> +    .handle_call = ffa_handle_call,
> +};
> +
> +REGISTER_TEE_MEDIATOR(ffa, "FF-A", XEN_DOMCTL_CONFIG_TEE_FFA, &ffa_ops);
> diff --git a/xen/arch/arm/vsmc.c b/xen/arch/arm/vsmc.c
> index cd68fa80e98a..7f2f5eb9ce3d 100644
> --- a/xen/arch/arm/vsmc.c
> +++ b/xen/arch/arm/vsmc.c
> @@ -15,6 +15,7 @@
> #include <asm/monitor.h>
> #include <asm/regs.h>
> #include <asm/smccc.h>
> +#include <asm/tee/ffa.h>
> #include <asm/tee/tee.h>
> #include <asm/traps.h>
> #include <asm/vpsci.h>
> @@ -24,7 +25,7 @@
> #define XEN_SMCCC_FUNCTION_COUNT 3
> 
> /* Number of functions currently supported by Standard Service Service Calls. */
> -#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS)
> +#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS + FFA_NR_FUNCS)
> 
> static bool fill_uid(struct cpu_user_regs *regs, xen_uuid_t uuid)
> {
> @@ -188,13 +189,23 @@ static bool handle_existing_apis(struct cpu_user_regs *regs)
>     return do_vpsci_0_1_call(regs, fid);
> }
> 
> +static bool is_psci_fid(uint32_t fid)
> +{
> +    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
> +
> +    return fn >= PSCI_FNUM_MIN_VALUE && fn <= PSCI_FNUM_MAX_VALUE;
> +}
> +
> /* PSCI 0.2 interface and other Standard Secure Calls */
> static bool handle_sssc(struct cpu_user_regs *regs)
> {
>     uint32_t fid = (uint32_t)get_user_reg(regs, 0);
> 
> -    if ( do_vpsci_0_2_call(regs, fid) )
> -        return true;
> +    if ( is_psci_fid(fid) )
> +        return do_vpsci_0_2_call(regs, fid);
> +
> +    if ( is_ffa_fid(fid) )
> +        return tee_handle_call(regs);
> 
>     switch ( fid )
>     {
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 1528ced5097a..92aff923056a 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -296,6 +296,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> 
> #define XEN_DOMCTL_CONFIG_TEE_NONE      0
> #define XEN_DOMCTL_CONFIG_TEE_OPTEE     1
> +#define XEN_DOMCTL_CONFIG_TEE_FFA       2
> 
> struct xen_arch_domainconfig {
>     /* IN/OUT */
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 03/20] tools: add Arm FF-A mediator

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds a new "ffa" value to the Enumeration "tee_type" to indicate if a
> guest is trusted to use FF-A.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> tools/libs/light/libxl_arm.c     | 3 +++
> tools/libs/light/libxl_types.idl | 3 ++-
> 2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index ddc7b2a15975..601890dda1ce 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -205,6 +205,9 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
>     case LIBXL_TEE_TYPE_OPTEE:
>         config->arch.tee_type = XEN_DOMCTL_CONFIG_TEE_OPTEE;
>         break;
> +    case LIBXL_TEE_TYPE_FFA:
> +        config->arch.tee_type = XEN_DOMCTL_CONFIG_TEE_FFA;
> +        break;
>     default:
>         LOG(ERROR, "Unknown TEE type %d",
>             d_config->b_info.tee);
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 0cfad8508dbd..64fb570bc19a 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
> @@ -494,7 +494,8 @@ libxl_gic_version = Enumeration("gic_version", [
> 
> libxl_tee_type = Enumeration("tee_type", [
>     (0, "none"),
> -    (1, "optee")
> +    (1, "optee"),
> +    (2, "ffa")
>     ], init_val = "LIBXL_TEE_TYPE_NONE")
> 
> libxl_rdm_reserve = Struct("rdm_reserve", [
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Julien Grall]

On 23/02/2023 14:46, Bertrand Marquis wrote:
>> diff --git a/xen/arch/arm/include/asm/tee/ffa.h b/xen/arch/arm/include/asm/tee/ffa.h
>> new file mode 100644
>> index 000000000000..94960100718e
>> --- /dev/null
>> +++ b/xen/arch/arm/include/asm/tee/ffa.h
>> @@ -0,0 +1,35 @@
>> +/* SPDX-License-Identifier: MIT */
> 
> Why is this file in MIT license ?
> 
> Any chance to set it as BSD-2 or maybe no license at all (most private headers do not have one) ?
When the header has no license boilerplate, then it will be using the 
default Xen license (i.e. GPLv2).

Now that we are starting to use SPDX, I think it would be better if we 
add it in every file (even if it is GPLv2.0).
Cheers,

-- 
Julien Grall

Re: [XEN PATCH v7 04/20] docs: add Arm FF-A mediator

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Describes a FF-A version 1.1 [1] mediator to communicate with a Secure
> Partition in secure world.
> 
> [1] https://developer.arm.com/documentation/den0077/latest
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

It is a little bit hard to check what is said here as the limitations
are probably due to some things done after this patch.

I would suggest to move this patch later in the serie.

Cheers
Bertrand

> ---
> SUPPORT.md               |  7 +++++++
> docs/man/xl.cfg.5.pod.in | 15 +++++++++++++++
> 2 files changed, 22 insertions(+)
> 
> diff --git a/SUPPORT.md b/SUPPORT.md
> index aa1940e55f09..5e0595419684 100644
> --- a/SUPPORT.md
> +++ b/SUPPORT.md
> @@ -818,6 +818,13 @@ that covers the DMA of the device to be passed through.
> 
> No support for QEMU backends in a 16K or 64K domain.
> 
> +### ARM: Firmware Framework for Arm A-profile (FF-A) Mediator
> +
> +    Status, Arm64: Tech Preview
> +
> +There are still some code paths where a vCPU may hog a pCPU longer than
> +necessary. The FF-A mediator is not yet implemented for Arm32.
> +
> ### ARM: Guest Device Tree support
> 
>     Status: Supported
> diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> index 024bceeb61b2..ca4fc3e67b0c 100644
> --- a/docs/man/xl.cfg.5.pod.in
> +++ b/docs/man/xl.cfg.5.pod.in
> @@ -1645,6 +1645,21 @@ in OP-TEE.
> 
> This feature is a B<technology preview>.
> 
> +=item B<ffa>
> +
> +B<Arm only.> Allow a guest to communicate via FF-A with Secure Partitions
> +(SP), default false.
> +
> +Currently is only a small subset of the FF-A specification supported. Just
> +enough to communicate with OP-TEE. In general only direct messaging and
> +sharing memory with one SP. More advanced use cases where memory might be
> +shared or donated to multple SPs is not supported.
> +
> +See L<https://developer.arm.com/documentation/den0077/latest> for more
> +informantion about FF-A.
> +
> +This feature is a B<technology preview>.
> +
> =back
> 
> =back
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 05/20] xen/arm: ffa: add remaining SMC function IDs

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds the remaining SMC function IDs from FF-A 1.1 specification.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

All number are coherent with the spec.

I guess you did not include the notification ones because you do not plan to support them for now ?

Anyway:
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> xen/arch/arm/tee/ffa.c | 34 ++++++++++++++++++++++++++++++++++
> 1 file changed, 34 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 824153c9303a..aa6cdbe0a4f9 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -59,7 +59,41 @@
> /* Function IDs */
> #define FFA_ERROR                       0x84000060U
> #define FFA_SUCCESS_32                  0x84000061U
> +#define FFA_SUCCESS_64                  0xC4000061U
> +#define FFA_INTERRUPT                   0x84000062U
> #define FFA_VERSION                     0x84000063U
> +#define FFA_FEATURES                    0x84000064U
> +#define FFA_RX_ACQUIRE                  0x84000084U
> +#define FFA_RX_RELEASE                  0x84000065U
> +#define FFA_RXTX_MAP_32                 0x84000066U
> +#define FFA_RXTX_MAP_64                 0xC4000066U
> +#define FFA_RXTX_UNMAP                  0x84000067U
> +#define FFA_PARTITION_INFO_GET          0x84000068U
> +#define FFA_ID_GET                      0x84000069U
> +#define FFA_SPM_ID_GET                  0x84000085U
> +#define FFA_MSG_WAIT                    0x8400006BU
> +#define FFA_MSG_YIELD                   0x8400006CU
> +#define FFA_MSG_RUN                     0x8400006DU
> +#define FFA_MSG_SEND2                   0x84000086U
> +#define FFA_MSG_SEND_DIRECT_REQ_32      0x8400006FU
> +#define FFA_MSG_SEND_DIRECT_REQ_64      0xC400006FU
> +#define FFA_MSG_SEND_DIRECT_RESP_32     0x84000070U
> +#define FFA_MSG_SEND_DIRECT_RESP_64     0xC4000070U
> +#define FFA_MEM_DONATE_32               0x84000071U
> +#define FFA_MEM_DONATE_64               0xC4000071U
> +#define FFA_MEM_LEND_32                 0x84000072U
> +#define FFA_MEM_LEND_64                 0xC4000072U
> +#define FFA_MEM_SHARE_32                0x84000073U
> +#define FFA_MEM_SHARE_64                0xC4000073U
> +#define FFA_MEM_RETRIEVE_REQ_32         0x84000074U
> +#define FFA_MEM_RETRIEVE_REQ_64         0xC4000074U
> +#define FFA_MEM_RETRIEVE_RESP           0x84000075U
> +#define FFA_MEM_RELINQUISH              0x84000076U
> +#define FFA_MEM_RECLAIM                 0x84000077U
> +#define FFA_MEM_FRAG_RX                 0x8400007AU
> +#define FFA_MEM_FRAG_TX                 0x8400007BU
> +#define FFA_MSG_SEND                    0x8400006EU
> +#define FFA_MSG_POLL                    0x8400006AU
> 
> struct ffa_ctx {
>     uint32_t guest_vers;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 03/20] tools: add Arm FF-A mediator

[Anthony PERARD]

On Wed, Feb 22, 2023 at 04:33:00PM +0100, Jens Wiklander wrote:
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 0cfad8508dbd..64fb570bc19a 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
> @@ -494,7 +494,8 @@ libxl_gic_version = Enumeration("gic_version", [
>  
>  libxl_tee_type = Enumeration("tee_type", [
>      (0, "none"),
> -    (1, "optee")
> +    (1, "optee"),
> +    (2, "ffa")

Could you add a comma at the end of this line? This will avoid the need
to change two lines the next time we are adding a tee_type (like you
have to do now).

Also, as you are changing libxl's API, could you add a LIBXL_HAVE_*
macro in libxl.h? Something like:
    /*
     * arch_arm.tee field in libxl_domain_build_info has ffa value.
     */
    #define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE_FFA 1

Thanks,

-- 
Anthony PERARD

Re: [XEN PATCH v7 01/20] xen/arm: smccc: add support for SMCCCv1.2 extended input/output registers

[Jens Wiklander]

On Thu, Feb 23, 2023 at 3:35 PM Julien Grall <julien@xen.org> wrote:
>
> Hi Jens,
>
> On 22/02/2023 15:32, Jens Wiklander wrote:
> > SMCCC v1.2 [1] AArch64 allows x0-x17 to be used as both parameter
> > registers and result registers for the SMC and HVC instructions.
> >
> > Arm Firmware Framework for Armv8-A specification makes use of x0-x7 as
> > parameter and result registers.
> >
> > Let us add new interface to support this extended set of input/output
> > registers.
> >
> > This is based on 3fdc0cb59d97 ("arm64: smccc: Add support for SMCCCv1.2
> > extended input/output registers") by Sudeep Holla from the Linux kernel
> >
> > The SMCCC version reported to the VM is bumped to 1.2 in order to support
> > handling FF-A messages.
> >
> > [1] https://developer.arm.com/documentation/den0028/c/?lang=en
> >
> > Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> One remark here. The tags are usually added chronologically. So your
> signed-off-by should be first.
>
> I am not planning to commit this patch until one of the patch using the
> call is ready. If this happen in this version, then I am happy to fix it
> on commit. Otherwise, please do it in the next version:
>
> Acked-by: Julien Grall <jgrall@amazon.com>

Thanks, I'll keep that in mind.

Cheers,
Jens

>
> Cheers,
>
> --
> Julien Grall

Re: [XEN PATCH v7 04/20] docs: add Arm FF-A mediator

[Jens Wiklander]

Hi Bertrand,

On Thu, Feb 23, 2023 at 4:10 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Describes a FF-A version 1.1 [1] mediator to communicate with a Secure
> > Partition in secure world.
> >
> > [1] https://developer.arm.com/documentation/den0077/latest
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> It is a little bit hard to check what is said here as the limitations
> are probably due to some things done after this patch.
>
> I would suggest to move this patch later in the serie.

OK, I'll put it last then.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > ---
> > SUPPORT.md               |  7 +++++++
> > docs/man/xl.cfg.5.pod.in | 15 +++++++++++++++
> > 2 files changed, 22 insertions(+)
> >
> > diff --git a/SUPPORT.md b/SUPPORT.md
> > index aa1940e55f09..5e0595419684 100644
> > --- a/SUPPORT.md
> > +++ b/SUPPORT.md
> > @@ -818,6 +818,13 @@ that covers the DMA of the device to be passed through.
> >
> > No support for QEMU backends in a 16K or 64K domain.
> >
> > +### ARM: Firmware Framework for Arm A-profile (FF-A) Mediator
> > +
> > +    Status, Arm64: Tech Preview
> > +
> > +There are still some code paths where a vCPU may hog a pCPU longer than
> > +necessary. The FF-A mediator is not yet implemented for Arm32.
> > +
> > ### ARM: Guest Device Tree support
> >
> >     Status: Supported
> > diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
> > index 024bceeb61b2..ca4fc3e67b0c 100644
> > --- a/docs/man/xl.cfg.5.pod.in
> > +++ b/docs/man/xl.cfg.5.pod.in
> > @@ -1645,6 +1645,21 @@ in OP-TEE.
> >
> > This feature is a B<technology preview>.
> >
> > +=item B<ffa>
> > +
> > +B<Arm only.> Allow a guest to communicate via FF-A with Secure Partitions
> > +(SP), default false.
> > +
> > +Currently is only a small subset of the FF-A specification supported. Just
> > +enough to communicate with OP-TEE. In general only direct messaging and
> > +sharing memory with one SP. More advanced use cases where memory might be
> > +shared or donated to multple SPs is not supported.
> > +
> > +See L<https://developer.arm.com/documentation/den0077/latest> for more
> > +informantion about FF-A.
> > +
> > +This feature is a B<technology preview>.
> > +
> > =back
> >
> > =back
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 03/20] tools: add Arm FF-A mediator

[Jens Wiklander]

On Thu, Feb 23, 2023 at 5:50 PM Anthony PERARD
<anthony.perard@citrix.com> wrote:
>
> On Wed, Feb 22, 2023 at 04:33:00PM +0100, Jens Wiklander wrote:
> > diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> > index 0cfad8508dbd..64fb570bc19a 100644
> > --- a/tools/libs/light/libxl_types.idl
> > +++ b/tools/libs/light/libxl_types.idl
> > @@ -494,7 +494,8 @@ libxl_gic_version = Enumeration("gic_version", [
> >
> >  libxl_tee_type = Enumeration("tee_type", [
> >      (0, "none"),
> > -    (1, "optee")
> > +    (1, "optee"),
> > +    (2, "ffa")
>
> Could you add a comma at the end of this line? This will avoid the need
> to change two lines the next time we are adding a tee_type (like you
> have to do now).
>
> Also, as you are changing libxl's API, could you add a LIBXL_HAVE_*
> macro in libxl.h? Something like:
>     /*
>      * arch_arm.tee field in libxl_domain_build_info has ffa value.
>      */
>     #define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE_FFA 1

OK, I'll fix that.

Thanks,
Jens

>
> Thanks,
>
> --
> Anthony PERARD

Re: [XEN PATCH v7 05/20] xen/arm: ffa: add remaining SMC function IDs

[Jens Wiklander]

Hi Bertrand,

On Thu, Feb 23, 2023 at 4:28 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds the remaining SMC function IDs from FF-A 1.1 specification.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> All number are coherent with the spec.

Thanks for verifying.

>
> I guess you did not include the notification ones because you do not plan to support them for now ?

That's correct.

>
> Anyway:
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > ---
> > xen/arch/arm/tee/ffa.c | 34 ++++++++++++++++++++++++++++++++++
> > 1 file changed, 34 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 824153c9303a..aa6cdbe0a4f9 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -59,7 +59,41 @@
> > /* Function IDs */
> > #define FFA_ERROR                       0x84000060U
> > #define FFA_SUCCESS_32                  0x84000061U
> > +#define FFA_SUCCESS_64                  0xC4000061U
> > +#define FFA_INTERRUPT                   0x84000062U
> > #define FFA_VERSION                     0x84000063U
> > +#define FFA_FEATURES                    0x84000064U
> > +#define FFA_RX_ACQUIRE                  0x84000084U
> > +#define FFA_RX_RELEASE                  0x84000065U
> > +#define FFA_RXTX_MAP_32                 0x84000066U
> > +#define FFA_RXTX_MAP_64                 0xC4000066U
> > +#define FFA_RXTX_UNMAP                  0x84000067U
> > +#define FFA_PARTITION_INFO_GET          0x84000068U
> > +#define FFA_ID_GET                      0x84000069U
> > +#define FFA_SPM_ID_GET                  0x84000085U
> > +#define FFA_MSG_WAIT                    0x8400006BU
> > +#define FFA_MSG_YIELD                   0x8400006CU
> > +#define FFA_MSG_RUN                     0x8400006DU
> > +#define FFA_MSG_SEND2                   0x84000086U
> > +#define FFA_MSG_SEND_DIRECT_REQ_32      0x8400006FU
> > +#define FFA_MSG_SEND_DIRECT_REQ_64      0xC400006FU
> > +#define FFA_MSG_SEND_DIRECT_RESP_32     0x84000070U
> > +#define FFA_MSG_SEND_DIRECT_RESP_64     0xC4000070U
> > +#define FFA_MEM_DONATE_32               0x84000071U
> > +#define FFA_MEM_DONATE_64               0xC4000071U
> > +#define FFA_MEM_LEND_32                 0x84000072U
> > +#define FFA_MEM_LEND_64                 0xC4000072U
> > +#define FFA_MEM_SHARE_32                0x84000073U
> > +#define FFA_MEM_SHARE_64                0xC4000073U
> > +#define FFA_MEM_RETRIEVE_REQ_32         0x84000074U
> > +#define FFA_MEM_RETRIEVE_REQ_64         0xC4000074U
> > +#define FFA_MEM_RETRIEVE_RESP           0x84000075U
> > +#define FFA_MEM_RELINQUISH              0x84000076U
> > +#define FFA_MEM_RECLAIM                 0x84000077U
> > +#define FFA_MEM_FRAG_RX                 0x8400007AU
> > +#define FFA_MEM_FRAG_TX                 0x8400007BU
> > +#define FFA_MSG_SEND                    0x8400006EU
> > +#define FFA_MSG_POLL                    0x8400006AU
> >
> > struct ffa_ctx {
> >     uint32_t guest_vers;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Jens Wiklander]

Hi,

On Thu, Feb 23, 2023 at 3:43 PM Julien Grall <julien@xen.org> wrote:
>
> Hi,
>
> I have only skimmed through the patch so far and I have one question below.
>
> On 22/02/2023 15:32, Jens Wiklander wrote:
> > Adds a FF-A version 1.1 [1] mediator to communicate with a Secure
> > Partition in secure world.
> >
> > This commit brings in only the parts needed to negotiate FF-A version
> > number with guest and SPMC.
> >
> > [1] https://developer.arm.com/documentation/den0077/e
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> >   xen/arch/arm/include/asm/domain.h  |   2 +-
> >   xen/arch/arm/include/asm/psci.h    |   4 +
> >   xen/arch/arm/include/asm/tee/ffa.h |  35 +++++
> >   xen/arch/arm/tee/Kconfig           |  11 ++
> >   xen/arch/arm/tee/Makefile          |   1 +
> >   xen/arch/arm/tee/ffa.c             | 217 +++++++++++++++++++++++++++++
> >   xen/arch/arm/vsmc.c                |  17 ++-
> >   xen/include/public/arch-arm.h      |   1 +
> >   8 files changed, 284 insertions(+), 4 deletions(-)
> >   create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
> >   create mode 100644 xen/arch/arm/tee/ffa.c
> >
> > diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
> > index 0e310601e846..754daa8efa04 100644
> > --- a/xen/arch/arm/include/asm/domain.h
> > +++ b/xen/arch/arm/include/asm/domain.h
> > @@ -110,7 +110,7 @@ struct arch_domain
> >       struct vpl011 vpl011;
> >   #endif
> >
> > -#ifdef CONFIG_TEE
> > +#if defined(CONFIG_TEE) || defined(CONFIG_FFA)
> I am confused, AFAICT, you are implementing FFA using TEE, so doesn't
> this mean that CONFIG_FFA depends on CONFIG_TEE?
>
> If so, then you should not need the extra defined here.

You're right, I'll fix it.

Thanks,
Jens

>
> >       void *tee;
> >   #endif
>
> --
> Julien Grall

Re: [XEN PATCH v7 02/20] xen/arm: tee: add a primitive FF-A mediator

[Jens Wiklander]

Hi,

On Thu, Feb 23, 2023 at 3:47 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:32, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds a FF-A version 1.1 [1] mediator to communicate with a Secure
> > Partition in secure world.
> >
> > This commit brings in only the parts needed to negotiate FF-A version
> > number with guest and SPMC.
> >
> > [1] https://developer.arm.com/documentation/den0077/e
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> Just some minor comments from my side after.
>
> > ---
> > xen/arch/arm/include/asm/domain.h  |   2 +-
> > xen/arch/arm/include/asm/psci.h    |   4 +
> > xen/arch/arm/include/asm/tee/ffa.h |  35 +++++
> > xen/arch/arm/tee/Kconfig           |  11 ++
> > xen/arch/arm/tee/Makefile          |   1 +
> > xen/arch/arm/tee/ffa.c             | 217 +++++++++++++++++++++++++++++
> > xen/arch/arm/vsmc.c                |  17 ++-
> > xen/include/public/arch-arm.h      |   1 +
> > 8 files changed, 284 insertions(+), 4 deletions(-)
> > create mode 100644 xen/arch/arm/include/asm/tee/ffa.h
> > create mode 100644 xen/arch/arm/tee/ffa.c
> >
> > diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
> > index 0e310601e846..754daa8efa04 100644
> > --- a/xen/arch/arm/include/asm/domain.h
> > +++ b/xen/arch/arm/include/asm/domain.h
> > @@ -110,7 +110,7 @@ struct arch_domain
> >     struct vpl011 vpl011;
> > #endif
> >
> > -#ifdef CONFIG_TEE
> > +#if defined(CONFIG_TEE) || defined(CONFIG_FFA)
> >     void *tee;
> > #endif
> >
> > diff --git a/xen/arch/arm/include/asm/psci.h b/xen/arch/arm/include/asm/psci.h
> > index 832f77afff3a..4780972621bb 100644
> > --- a/xen/arch/arm/include/asm/psci.h
> > +++ b/xen/arch/arm/include/asm/psci.h
> > @@ -24,6 +24,10 @@ void call_psci_cpu_off(void);
> > void call_psci_system_off(void);
> > void call_psci_system_reset(void);
> >
> > +/* Range of allocated PSCI function numbers */
> > +#define PSCI_FNUM_MIN_VALUE                 _AC(0,U)
> > +#define PSCI_FNUM_MAX_VALUE                 _AC(0x1f,U)
> > +
> > /* PSCI v0.2 interface */
> > #define PSCI_0_2_FN32(nr) ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL,             \
> >                                              ARM_SMCCC_CONV_32,               \
> > diff --git a/xen/arch/arm/include/asm/tee/ffa.h b/xen/arch/arm/include/asm/tee/ffa.h
> > new file mode 100644
> > index 000000000000..94960100718e
> > --- /dev/null
> > +++ b/xen/arch/arm/include/asm/tee/ffa.h
> > @@ -0,0 +1,35 @@
> > +/* SPDX-License-Identifier: MIT */
>
> Why is this file in MIT license ?
>
> Any chance to set it as BSD-2 or maybe no license at all (most private headers do not have one) ?

I'll change it to GPL-2.0-only.

>
> > +/*
> > + * xen/arch/arm/include/asm/tee/ffa.h
> > + *
> > + * Arm Firmware Framework for ARMv8-A(FFA) mediator
> > + *
> > + * Copyright (C) 2023  Linaro Limited
> > + */
> > +
> > +#ifndef __ASM_ARM_TEE_FFA_H__
> > +#define __ASM_ARM_TEE_FFA_H__
> > +
> > +#include <xen/const.h>
> > +#include <xen/kconfig.h>
> > +
> > +#include <asm/smccc.h>
> > +#include <asm/types.h>
> > +
> > +#define FFA_FNUM_MIN_VALUE              _AC(0x60,U)
> > +#define FFA_FNUM_MAX_VALUE              _AC(0x86,U)
> > +
> > +static inline bool is_ffa_fid(uint32_t fid)
> > +{
> > +    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
> > +
> > +    return fn >= FFA_FNUM_MIN_VALUE && fn <= FFA_FNUM_MAX_VALUE;
> > +}
> > +
> > +#ifdef CONFIG_FFA
> > +#define FFA_NR_FUNCS    (9 + (IS_ENABLED(CONFIG_ARM_64) ? 3 : 0) )
>
> I am not quite sure of the usage of IS_ENABLED for a constant
>
> Maybe just doing:
> #ifdef CONFIG_FFA
> #ifdef CONFIG_ARM_64
> #define FFA_NR_FUNCS   12
> #else
> #define FFA_NR_FUNCS    9
> #endif
> #else
> #define FFA_NR_FUNCS    0
> #endif

OK, I'll change it.

>
> > +#else
> > +#define FFA_NR_FUNCS    0
> > +#endif
> > +
> > +#endif /*__ASM_ARM_TEE_FFA_H__*/
> > diff --git a/xen/arch/arm/tee/Kconfig b/xen/arch/arm/tee/Kconfig
> > index 392169b2559d..923f08ba8cb7 100644
> > --- a/xen/arch/arm/tee/Kconfig
> > +++ b/xen/arch/arm/tee/Kconfig
> > @@ -8,3 +8,14 @@ config OPTEE
> >  virtualization-enabled OP-TEE present. You can learn more
> >  about virtualization for OP-TEE at
> >  https://optee.readthedocs.io/architecture/virtualization.html
> > +
> > +config FFA
> > + bool "Enable FF-A mediator support (UNSUPPORTED)" if UNSUPPORTED
> > + default n
> > + depends on ARM_64
> > + help
> > +  This option enables a minimal FF-A mediator. The mediator is
> > +  generic as it follows the FF-A specification [1], but it only
> > +  implements a small subset of the specification.
> > +
> > +  [1] https://developer.arm.com/documentation/den0077/latest
> > diff --git a/xen/arch/arm/tee/Makefile b/xen/arch/arm/tee/Makefile
> > index 982c87968447..58a1015e40e0 100644
> > --- a/xen/arch/arm/tee/Makefile
> > +++ b/xen/arch/arm/tee/Makefile
> > @@ -1,2 +1,3 @@
> > +obj-$(CONFIG_FFA) += ffa.o
> > obj-y += tee.o
> > obj-$(CONFIG_OPTEE) += optee.o
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > new file mode 100644
> > index 000000000000..824153c9303a
> > --- /dev/null
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -0,0 +1,217 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
>
> I think now you have to say either GPL-2.0-only or GPL-2.0-or-later

OK, I'll change it to GPL-2.0-only

>
> > +/*
> > + * xen/arch/arm/tee/ffa.c
> > + *
> > + * Arm Firmware Framework for ARMv8-A (FF-A) mediator
> > + *
> > + * Copyright (C) 2023  Linaro Limited
> > + */
> > +
> > +#include <xen/domain_page.h>
> > +#include <xen/errno.h>
> > +#include <xen/init.h>
> > +#include <xen/lib.h>
> > +#include <xen/sched.h>
> > +#include <xen/types.h>
> > +#include <xen/sizes.h>
> > +#include <xen/bitops.h>
>
> Please try to put the headers in alphabetical order.
>
> > +
> > +#include <asm/smccc.h>
> > +#include <asm/event.h>
> > +#include <asm/tee/tee.h>
> > +#include <asm/tee/ffa.h>
> > +#include <asm/regs.h>
>
> Same here

I'll fix it.

>
> The rest looks ok to me.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > +
> > +/* Error codes */
> > +#define FFA_RET_OK                      0
> > +#define FFA_RET_NOT_SUPPORTED           -1
> > +#define FFA_RET_INVALID_PARAMETERS      -2
> > +#define FFA_RET_NO_MEMORY               -3
> > +#define FFA_RET_BUSY                    -4
> > +#define FFA_RET_INTERRUPTED             -5
> > +#define FFA_RET_DENIED                  -6
> > +#define FFA_RET_RETRY                   -7
> > +#define FFA_RET_ABORTED                 -8
> > +
> > +/* FFA_VERSION helpers */
> > +#define FFA_VERSION_MAJOR_SHIFT         16U
> > +#define FFA_VERSION_MAJOR_MASK          0x7FFFU
> > +#define FFA_VERSION_MINOR_SHIFT         0U
> > +#define FFA_VERSION_MINOR_MASK          0xFFFFU
> > +#define MAKE_FFA_VERSION(major, minor)  \
> > +        ((((major) & FFA_VERSION_MAJOR_MASK) << FFA_VERSION_MAJOR_SHIFT) | \
> > +         ((minor) & FFA_VERSION_MINOR_MASK))
> > +
> > +#define FFA_MIN_VERSION         MAKE_FFA_VERSION(1, 0)
> > +#define FFA_VERSION_1_0         MAKE_FFA_VERSION(1, 0)
> > +#define FFA_VERSION_1_1         MAKE_FFA_VERSION(1, 1)
> > +
> > +/*
> > + * This is the version we want to use in communication with guests and SPs.
> > + * During negotiation with a guest or a SP we may need to lower it for
> > + * that particular guest or SP.
> > + */
> > +#define FFA_MY_VERSION_MAJOR    1U
> > +#define FFA_MY_VERSION_MINOR    1U
> > +#define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
> > +                                                 FFA_MY_VERSION_MINOR)
> > +
> > +/* Function IDs */
> > +#define FFA_ERROR                       0x84000060U
> > +#define FFA_SUCCESS_32                  0x84000061U
> > +#define FFA_VERSION                     0x84000063U
> > +
> > +struct ffa_ctx {
> > +    uint32_t guest_vers;
> > +};
> > +
> > +/* Negotiated FF-A version to use with the SPMC */
> > +static uint32_t ffa_version __ro_after_init;
> > +
> > +static bool ffa_get_version(uint32_t *vers)
> > +{
> > +    const struct arm_smccc_1_2_regs arg = {
> > +        .a0 = FFA_VERSION,
> > +        .a1 = FFA_MY_VERSION,
> > +    };
> > +    struct arm_smccc_1_2_regs resp;
> > +
> > +    arm_smccc_1_2_smc(&arg, &resp);
> > +    if ( resp.a0 == FFA_RET_NOT_SUPPORTED )
> > +    {
> > +        gprintk(XENLOG_ERR, "ffa: FFA_VERSION returned not supported\n");
> > +        return false;
> > +    }
> > +
> > +    *vers = resp.a0;
> > +
> > +    return true;
> > +}
> > +
> > +static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> > +                     register_t v2, register_t v3, register_t v4, register_t v5,
> > +                     register_t v6, register_t v7)
> > +{
> > +        set_user_reg(regs, 0, v0);
> > +        set_user_reg(regs, 1, v1);
> > +        set_user_reg(regs, 2, v2);
> > +        set_user_reg(regs, 3, v3);
> > +        set_user_reg(regs, 4, v4);
> > +        set_user_reg(regs, 5, v5);
> > +        set_user_reg(regs, 6, v6);
> > +        set_user_reg(regs, 7, v7);
> > +}
> > +
> > +static void handle_version(struct cpu_user_regs *regs)
> > +{
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +    uint32_t vers = get_user_reg(regs, 1);
> > +
> > +    if ( vers < FFA_VERSION_1_1 )
> > +        vers = FFA_VERSION_1_0;
> > +    else
> > +        vers = FFA_VERSION_1_1;
> > +
> > +    ctx->guest_vers = vers;
> > +    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> > +}
> > +
> > +static bool ffa_handle_call(struct cpu_user_regs *regs)
> > +{
> > +    uint32_t fid = get_user_reg(regs, 0);
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +
> > +    if ( !ctx )
> > +        return false;
> > +
> > +    switch ( fid )
> > +    {
> > +    case FFA_VERSION:
> > +        handle_version(regs);
> > +        return true;
> > +
> > +    default:
> > +        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> > +        return false;
> > +    }
> > +}
> > +
> > +static int ffa_domain_init(struct domain *d)
> > +{
> > +    struct ffa_ctx *ctx;
> > +
> > +    if ( !ffa_version )
> > +        return -ENODEV;
> > +
> > +    ctx = xzalloc(struct ffa_ctx);
> > +    if ( !ctx )
> > +        return -ENOMEM;
> > +
> > +    d->arch.tee = ctx;
> > +
> > +    return 0;
> > +}
> > +
> > +/* This function is supposed to undo what ffa_domain_init() has done */
> > +static int ffa_relinquish_resources(struct domain *d)
> > +{
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +
> > +    if ( !ctx )
> > +        return 0;
> > +
> > +    XFREE(d->arch.tee);
> > +
> > +    return 0;
> > +}
> > +
> > +static bool ffa_probe(void)
> > +{
> > +    uint32_t vers;
> > +    unsigned int major_vers;
> > +    unsigned int minor_vers;
> > +
> > +    /*
> > +     * psci_init_smccc() updates this value with what's reported by EL-3
> > +     * or secure world.
> > +     */
> > +    if ( smccc_ver < ARM_SMCCC_VERSION_1_2 )
> > +    {
> > +        printk(XENLOG_ERR
> > +               "ffa: unsupported SMCCC version %#x (need at least %#x)\n",
> > +               smccc_ver, ARM_SMCCC_VERSION_1_2);
> > +        return false;
> > +    }
> > +
> > +    if ( !ffa_get_version(&vers) )
> > +        return false;
> > +
> > +    if ( vers < FFA_MIN_VERSION || vers > FFA_MY_VERSION )
> > +    {
> > +        printk(XENLOG_ERR "ffa: Incompatible version %#x found\n", vers);
> > +        return false;
> > +    }
> > +
> > +    major_vers = (vers >> FFA_VERSION_MAJOR_SHIFT) & FFA_VERSION_MAJOR_MASK;
> > +    minor_vers = vers & FFA_VERSION_MINOR_MASK;
> > +    printk(XENLOG_INFO "ARM FF-A Mediator version %u.%u\n",
> > +           FFA_MY_VERSION_MAJOR, FFA_MY_VERSION_MINOR);
> > +    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> > +           major_vers, minor_vers);
> > +
> > +    ffa_version = vers;
> > +
> > +    return true;
> > +}
> > +
> > +static const struct tee_mediator_ops ffa_ops =
> > +{
> > +    .probe = ffa_probe,
> > +    .domain_init = ffa_domain_init,
> > +    .relinquish_resources = ffa_relinquish_resources,
> > +    .handle_call = ffa_handle_call,
> > +};
> > +
> > +REGISTER_TEE_MEDIATOR(ffa, "FF-A", XEN_DOMCTL_CONFIG_TEE_FFA, &ffa_ops);
> > diff --git a/xen/arch/arm/vsmc.c b/xen/arch/arm/vsmc.c
> > index cd68fa80e98a..7f2f5eb9ce3d 100644
> > --- a/xen/arch/arm/vsmc.c
> > +++ b/xen/arch/arm/vsmc.c
> > @@ -15,6 +15,7 @@
> > #include <asm/monitor.h>
> > #include <asm/regs.h>
> > #include <asm/smccc.h>
> > +#include <asm/tee/ffa.h>
> > #include <asm/tee/tee.h>
> > #include <asm/traps.h>
> > #include <asm/vpsci.h>
> > @@ -24,7 +25,7 @@
> > #define XEN_SMCCC_FUNCTION_COUNT 3
> >
> > /* Number of functions currently supported by Standard Service Service Calls. */
> > -#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS)
> > +#define SSSC_SMCCC_FUNCTION_COUNT (3 + VPSCI_NR_FUNCS + FFA_NR_FUNCS)
> >
> > static bool fill_uid(struct cpu_user_regs *regs, xen_uuid_t uuid)
> > {
> > @@ -188,13 +189,23 @@ static bool handle_existing_apis(struct cpu_user_regs *regs)
> >     return do_vpsci_0_1_call(regs, fid);
> > }
> >
> > +static bool is_psci_fid(uint32_t fid)
> > +{
> > +    uint32_t fn = fid & ARM_SMCCC_FUNC_MASK;
> > +
> > +    return fn >= PSCI_FNUM_MIN_VALUE && fn <= PSCI_FNUM_MAX_VALUE;
> > +}
> > +
> > /* PSCI 0.2 interface and other Standard Secure Calls */
> > static bool handle_sssc(struct cpu_user_regs *regs)
> > {
> >     uint32_t fid = (uint32_t)get_user_reg(regs, 0);
> >
> > -    if ( do_vpsci_0_2_call(regs, fid) )
> > -        return true;
> > +    if ( is_psci_fid(fid) )
> > +        return do_vpsci_0_2_call(regs, fid);
> > +
> > +    if ( is_ffa_fid(fid) )
> > +        return tee_handle_call(regs);
> >
> >     switch ( fid )
> >     {
> > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> > index 1528ced5097a..92aff923056a 100644
> > --- a/xen/include/public/arch-arm.h
> > +++ b/xen/include/public/arch-arm.h
> > @@ -296,6 +296,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> >
> > #define XEN_DOMCTL_CONFIG_TEE_NONE      0
> > #define XEN_DOMCTL_CONFIG_TEE_OPTEE     1
> > +#define XEN_DOMCTL_CONFIG_TEE_FFA       2
> >
> > struct xen_arch_domainconfig {
> >     /* IN/OUT */
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 06/20] xen/arm: ffa: add flags for FFA_PARTITION_INFO_GET

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Defines flags used for the function FFA_PARTITION_INFO_GET.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 26 ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index aa6cdbe0a4f9..f4562ed2defc 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -56,6 +56,32 @@
> #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
>                                                  FFA_MY_VERSION_MINOR)
> 
> +/*
> + * Flags used for the FFA_PARTITION_INFO_GET return message:

This is somehow no completely precise.
Could I suggest to use what the doc says:
Flags to determine partition properties in FFA_PARTITION_INFO_GET return message


> + * BIT(0): Supports receipt of direct requests
> + * BIT(1): Can send direct requests
> + * BIT(2): Can send and receive indirect messages
> + * BIT(3): Supports receipt of notifications
> + * BIT(4-5): Partition ID is a PE endpoint ID

You describe all bits until 5 but not 6,7 and 8.
Please describe all of them to be coherent.

> + */
> +#define FFA_PART_PROP_DIRECT_REQ_RECV   BIT(0, U)
> +#define FFA_PART_PROP_DIRECT_REQ_SEND   BIT(1, U)
> +#define FFA_PART_PROP_INDIRECT_MSGS     BIT(2, U)
> +#define FFA_PART_PROP_RECV_NOTIF        BIT(3, U)
> +#define FFA_PART_PROP_IS_PE_ID          (0U << 4)
> +#define FFA_PART_PROP_IS_SEPID_INDEP    (1U << 4)
> +#define FFA_PART_PROP_IS_SEPID_DEP      (2U << 4)
> +#define FFA_PART_PROP_IS_AUX_ID         (3U << 4)
> +#define FFA_PART_PROP_NOTIF_CREATED     BIT(6, U)
> +#define FFA_PART_PROP_NOTIF_DESTROYED   BIT(7, U)
> +#define FFA_PART_PROP_AARCH64_STATE     BIT(8, U)

bits definitions are coherent with the standard

> +
> +/*
> + * Flag used as parameter to FFA_PARTITION_INFO_GET to return partition
> + * count only.
> + */
> +#define FFA_PARTITION_INFO_GET_COUNT_FLAG BIT(0, U)

same here.

Cheers
Bertrand

> +
> /* Function IDs */
> #define FFA_ERROR                       0x84000060U
> #define FFA_SUCCESS_32                  0x84000061U
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 07/20] xen/arm: ffa: add defines for framework direct request/response messages

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds defines for framework direct request/response messages.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index f4562ed2defc..d04bac9cc47f 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -56,6 +56,15 @@
> #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
>                                                  FFA_MY_VERSION_MINOR)
> 
> +/* Framework direct request/response */

In the previous patch you were more verbose in the comment which was nice.
I would suggest here to use the same "format":

Flags used for the MSG_SEND_DIRECT_REQ/RESP:
BIT(31): Framework or partition message
BIT(7-0): Message type for frameworks messages

> +#define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> +#define FFA_MSG_TYPE_MASK               0xFFU;

Maybe more coherent to name this FFA_MSG_FLAG_TYPE_MASK ?

I am a bit unsure here because we could also keep it like that and just
add _TYPE to other definitions after.

What do you think ?

> +#define FFA_MSG_PSCI                    0x0U
> +#define FFA_MSG_SEND_VM_CREATED         0x4U
> +#define FFA_MSG_RESP_VM_CREATED         0x5U
> +#define FFA_MSG_SEND_VM_DESTROYED       0x6U
> +#define FFA_MSG_RESP_VM_DESTROYED       0x7U
> +
> /*
>  * Flags used for the FFA_PARTITION_INFO_GET return message:
>  * BIT(0): Supports receipt of direct requests
> -- 
> 2.34.1
> 

Cheers
Bertrand

Re: [XEN PATCH v7 06/20] xen/arm: ffa: add flags for FFA_PARTITION_INFO_GET

[Jens Wiklander]

Hi Bertrand,

On Fri, Feb 24, 2023 at 10:30 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Defines flags used for the function FFA_PARTITION_INFO_GET.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 26 ++++++++++++++++++++++++++
> > 1 file changed, 26 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index aa6cdbe0a4f9..f4562ed2defc 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -56,6 +56,32 @@
> > #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
> >                                                  FFA_MY_VERSION_MINOR)
> >
> > +/*
> > + * Flags used for the FFA_PARTITION_INFO_GET return message:
>
> This is somehow no completely precise.
> Could I suggest to use what the doc says:
> Flags to determine partition properties in FFA_PARTITION_INFO_GET return message

OK

>
>
> > + * BIT(0): Supports receipt of direct requests
> > + * BIT(1): Can send direct requests
> > + * BIT(2): Can send and receive indirect messages
> > + * BIT(3): Supports receipt of notifications
> > + * BIT(4-5): Partition ID is a PE endpoint ID
>
> You describe all bits until 5 but not 6,7 and 8.
> Please describe all of them to be coherent.

OK

>
> > + */
> > +#define FFA_PART_PROP_DIRECT_REQ_RECV   BIT(0, U)
> > +#define FFA_PART_PROP_DIRECT_REQ_SEND   BIT(1, U)
> > +#define FFA_PART_PROP_INDIRECT_MSGS     BIT(2, U)
> > +#define FFA_PART_PROP_RECV_NOTIF        BIT(3, U)
> > +#define FFA_PART_PROP_IS_PE_ID          (0U << 4)
> > +#define FFA_PART_PROP_IS_SEPID_INDEP    (1U << 4)
> > +#define FFA_PART_PROP_IS_SEPID_DEP      (2U << 4)
> > +#define FFA_PART_PROP_IS_AUX_ID         (3U << 4)
> > +#define FFA_PART_PROP_NOTIF_CREATED     BIT(6, U)
> > +#define FFA_PART_PROP_NOTIF_DESTROYED   BIT(7, U)
> > +#define FFA_PART_PROP_AARCH64_STATE     BIT(8, U)
>
> bits definitions are coherent with the standard
>
> > +
> > +/*
> > + * Flag used as parameter to FFA_PARTITION_INFO_GET to return partition
> > + * count only.
> > + */
> > +#define FFA_PARTITION_INFO_GET_COUNT_FLAG BIT(0, U)
>
> same here.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > +
> > /* Function IDs */
> > #define FFA_ERROR                       0x84000060U
> > #define FFA_SUCCESS_32                  0x84000061U
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 08/20] xen/arm: ffa: note dependency on 4k pages

[Bertrand Marquis]

HI Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds a BUILD_BUG_ON() to assert the dependency on 4k pages in the FF-A
> mediator.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

NIT: I would s/note/enforce/ in the title:
xen/arm: ffa: enforce 4k pages

Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> xen/arch/arm/tee/ffa.c | 21 +++++++++++++++++++++
> 1 file changed, 21 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index d04bac9cc47f..8b0b80ce1ff5 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -56,6 +56,16 @@
> #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
>                                                  FFA_MY_VERSION_MINOR)
> 
> +/*
> + * The FF-A specification explicitly works with 4K pages as a measure of
> + * memory size, for example, FFA_RXTX_MAP takes one parameter "RX/TX page
> + * count" which is the number of contiguous 4K pages allocated. Xen may use
> + * a different page size depending on the configuration to avoid confusion
> + * with PAGE_SIZE use a special define when it's a page size as in the FF-A
> + * specification.
> + */
> +#define FFA_PAGE_SIZE                   SZ_4K
> +
> /* Framework direct request/response */
> #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> #define FFA_MSG_TYPE_MASK               0xFFU;
> @@ -242,6 +252,17 @@ static bool ffa_probe(void)
>     unsigned int major_vers;
>     unsigned int minor_vers;
> 
> +    /*
> +     * FF-A often works in units of 4K pages and currently it's assumed
> +     * that we can map memory using that granularity. See also the comment
> +     * above the FFA_PAGE_SIZE define.
> +     *
> +     * It is possible to support a PAGE_SIZE larger than 4K in Xen, but
> +     * until that is fully handled in this code make sure that we only use
> +     * 4K page sizes.
> +     */
> +    BUILD_BUG_ON(PAGE_SIZE != FFA_PAGE_SIZE);
> +
>     /*
>      * psci_init_smccc() updates this value with what's reported by EL-3
>      * or secure world.
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 09/20] xen/arm: ffa: add support for FFA_ID_GET

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support for the FF-A function FFA_ID_GET to return the ID of the
> calling client.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 21 ++++++++++++++++++++-
> 1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 8b0b80ce1ff5..463fd7730573 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -167,6 +167,12 @@ static bool ffa_get_version(uint32_t *vers)
>     return true;
> }
> 
> +static uint16_t get_vm_id(const struct domain *d)
> +{
> +    /* +1 since 0 is reserved for the hypervisor in FF-A */
> +    return d->domain_id + 1;
> +}
> +
> static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>                      register_t v2, register_t v3, register_t v4, register_t v5,
>                      register_t v6, register_t v7)
> @@ -181,6 +187,12 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>         set_user_reg(regs, 7, v7);
> }
> 
> +static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> +                             uint32_t w3)
> +{
> +    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
> +}
> +
> static void handle_version(struct cpu_user_regs *regs)
> {
>     struct domain *d = current->domain;
> @@ -210,6 +222,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>     case FFA_VERSION:
>         handle_version(regs);
>         return true;
> +    case FFA_ID_GET:
> +        set_regs_success(regs, get_vm_id(d), 0);
> +        return true;
> 
>     default:
>         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> @@ -221,7 +236,11 @@ static int ffa_domain_init(struct domain *d)
> {
>     struct ffa_ctx *ctx;
> 
> -    if ( !ffa_version )
> +     /*
> +      * We can't use that last possible domain ID or get_vm_id() would cause
> +      * an overflow.
> +      */
> +    if ( !ffa_version || d->domain_id == UINT16_MAX)
>         return -ENODEV;

In reality the overflow could only happen if this is called by the IDLE domain right now.
Anyway this could change and this is making the code more robust at no real cost.

I would just suggest here to return a different error code like ERANGE for example to
prevent missing ENODEV with other cases not related to FFA not being available.

Cheers
Bertrand

> 
>     ctx = xzalloc(struct ffa_ctx);
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 09/20] xen/arm: ffa: add support for FFA_ID_GET

[Julien Grall]

Hi,

On 27/02/2023 14:48, Bertrand Marquis wrote:
>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>
>> Adds support for the FF-A function FFA_ID_GET to return the ID of the
>> calling client.
>>
>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>> ---
>> xen/arch/arm/tee/ffa.c | 21 ++++++++++++++++++++-
>> 1 file changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>> index 8b0b80ce1ff5..463fd7730573 100644
>> --- a/xen/arch/arm/tee/ffa.c
>> +++ b/xen/arch/arm/tee/ffa.c
>> @@ -167,6 +167,12 @@ static bool ffa_get_version(uint32_t *vers)
>>      return true;
>> }
>>
>> +static uint16_t get_vm_id(const struct domain *d)
>> +{
>> +    /* +1 since 0 is reserved for the hypervisor in FF-A */
>> +    return d->domain_id + 1;
>> +}
>> +
>> static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>>                       register_t v2, register_t v3, register_t v4, register_t v5,
>>                       register_t v6, register_t v7)
>> @@ -181,6 +187,12 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>>          set_user_reg(regs, 7, v7);
>> }
>>
>> +static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
>> +                             uint32_t w3)
>> +{
>> +    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
>> +}
>> +
>> static void handle_version(struct cpu_user_regs *regs)
>> {
>>      struct domain *d = current->domain;
>> @@ -210,6 +222,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>      case FFA_VERSION:
>>          handle_version(regs);
>>          return true;
>> +    case FFA_ID_GET:
>> +        set_regs_success(regs, get_vm_id(d), 0);
>> +        return true;
>>
>>      default:
>>          gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
>> @@ -221,7 +236,11 @@ static int ffa_domain_init(struct domain *d)
>> {
>>      struct ffa_ctx *ctx;
>>
>> -    if ( !ffa_version )
>> +     /*
>> +      * We can't use that last possible domain ID or get_vm_id() would cause
>> +      * an overflow.
>> +      */
>> +    if ( !ffa_version || d->domain_id == UINT16_MAX)
>>          return -ENODEV;
> 
> In reality the overflow could only happen if this is called by the IDLE domain right now.
> Anyway this could change and this is making the code more robust at no real cost.
> 
> I would just suggest here to return a different error code like ERANGE for example to
> prevent missing ENODEV with other cases not related to FFA not being available.

+1. I would also like to suggest to use >= rather than == in case we 
decide to support more than 16-bit domid.

Cheers,

-- 
Julien Grall

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support for sending a FF-A direct request. Checks that the SP also
> supports handling a 32-bit direct request. 64-bit direct requests are
> not used by the mediator itself so there is not need to check for that.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 119 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 463fd7730573..a5d8a12635b6 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -142,6 +142,7 @@
> 
> struct ffa_ctx {
>     uint32_t guest_vers;
> +    bool interrupted;

This is added and set here for one special error code but is never used.
I would suggest to introduce this when there will be an action based on it.

> };
> 
> /* Negotiated FF-A version to use with the SPMC */
> @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
>     return true;
> }
> 
> +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
> +{
> +    switch ( resp->a0 )
> +    {
> +    case FFA_ERROR:
> +        if ( resp->a2 )
> +            return resp->a2;
> +        else
> +            return FFA_RET_NOT_SUPPORTED;
> +    case FFA_SUCCESS_32:
> +    case FFA_SUCCESS_64:
> +        return FFA_RET_OK;
> +    default:
> +        return FFA_RET_NOT_SUPPORTED;
> +    }
> +}
> +
> +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
> +                               register_t a3, register_t a4)
> +{
> +    const struct arm_smccc_1_2_regs arg = {
> +        .a0 = fid,
> +        .a1 = a1,
> +        .a2 = a2,
> +        .a3 = a3,
> +        .a4 = a4,
> +    };
> +    struct arm_smccc_1_2_regs resp;
> +
> +    arm_smccc_1_2_smc(&arg, &resp);
> +
> +    return get_ffa_ret_code(&resp);
> +}
> +
> +static int32_t ffa_features(uint32_t id)
> +{
> +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
> +}
> +
> +static bool check_mandatory_feature(uint32_t id)
> +{
> +    uint32_t ret = ffa_features(id);
> +
> +    if (ret)
> +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);

It might be useful here to actually print the error code.
Are we sure that all errors actually mean not supported ?

> +
> +    return !ret;
> +}
> +
> static uint16_t get_vm_id(const struct domain *d)
> {
>     /* +1 since 0 is reserved for the hypervisor in FF-A */
> @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
>     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> }
> 
> +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> +{
> +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> +    struct arm_smccc_1_2_regs resp = { };
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    uint32_t src_dst;
> +    uint64_t mask;
> +
> +    if ( smccc_is_conv_64(fid) )
> +        mask = GENMASK_ULL(63, 0);
> +    else
> +        mask = GENMASK_ULL(31, 0);
> +
> +    src_dst = get_user_reg(regs, 1);
> +    if ( (src_dst >> 16) != get_vm_id(d) )
> +    {
> +        resp.a0 = FFA_ERROR;
> +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
> +        goto out;
> +    }
> +
> +    arg.a1 = src_dst;
> +    arg.a2 = get_user_reg(regs, 2) & mask;
> +    arg.a3 = get_user_reg(regs, 3) & mask;
> +    arg.a4 = get_user_reg(regs, 4) & mask;
> +    arg.a5 = get_user_reg(regs, 5) & mask;
> +    arg.a6 = get_user_reg(regs, 6) & mask;
> +    arg.a7 = get_user_reg(regs, 7) & mask;
> +
> +    while ( true )
> +    {
> +        arm_smccc_1_2_smc(&arg, &resp);
> +
> +        switch ( resp.a0 )
> +        {
> +        case FFA_INTERRUPT:
> +            ctx->interrupted = true;
> +            goto out;
> +        case FFA_ERROR:
> +        case FFA_SUCCESS_32:
> +        case FFA_SUCCESS_64:
> +        case FFA_MSG_SEND_DIRECT_RESP_32:
> +        case FFA_MSG_SEND_DIRECT_RESP_64:
> +            goto out;
> +        default:
> +            /* Bad fid, report back. */
> +            memset(&arg, 0, sizeof(arg));
> +            arg.a0 = FFA_ERROR;
> +            arg.a1 = src_dst;
> +            arg.a2 = FFA_RET_NOT_SUPPORTED;
> +            continue;

There is a potential infinite loop here and i do not understand
why this needs to be done.
Here if something is returning a value that you do not understand
you send back an ERROR to it. I do not find in the spec where this
is supposed to be done.
Can you explain a bit here ?

> +        }
> +    }
> +
> +out:
> +    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
> +             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> +}
> +
> static bool ffa_handle_call(struct cpu_user_regs *regs)
> {
>     uint32_t fid = get_user_reg(regs, 0);
> @@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>     case FFA_ID_GET:
>         set_regs_success(regs, get_vm_id(d), 0);
>         return true;
> +    case FFA_MSG_SEND_DIRECT_REQ_32:
> +#ifdef CONFIG_ARM_64
> +    case FFA_MSG_SEND_DIRECT_REQ_64:
> +#endif
> +        handle_msg_send_direct_req(regs, fid);
> +        return true;
> 
>     default:
>         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> @@ -310,6 +426,9 @@ static bool ffa_probe(void)
>     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>            major_vers, minor_vers);
> 
> +    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> +        return false;

One could not need this feature and here this will make everything unavailable instead.
Why not just reporting back the unsupported error to clients using unsupported interfaces ?

Cheers
Bertrand

> +
>     ffa_version = vers;
> 
>     return true;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 11/20] xen/arm: ffa: map SPMC rx/tx buffers

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> When initializing the FF-A mediator map the RX and TX buffers shared with
> the SPMC.
> 
> These buffer are later used to to transmit data that cannot be passed in
> registers only.
> 
> Adds a check that the SP supports the needed FF-A features
> FFA_RXTX_MAP_64 / FFA_RXTX_MAP_32 and FFA_RXTX_UNMAP. In 64-bit mode we
> must use FFA_RXTX_MAP_64 since registers are used to transmit the
> physical addresses of the RX/TX buffers.

Right now, FFA on 32bit would only work correctly if LPAE is not used and only addresses
under 4G are used by Xen and by guests as addresses are transferred through a single register.

I think that we need for now to only enable FFA support on 64bit as the limitations we 
would need to enforce on 32bit are complex and the use case for FFA on 32bit platforms
is not that obvious now.

> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 57 +++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 56 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index a5d8a12635b6..07dd5c36d54b 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -148,6 +148,15 @@ struct ffa_ctx {
> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> +/*
> + * Our rx/tx buffers shared with the SPMC.
> + *
> + * ffa_page_count is the number of pages used in each of these buffers.
> + */
> +static void *ffa_rx __read_mostly;
> +static void *ffa_tx __read_mostly;
> +static unsigned int ffa_page_count __read_mostly;
> +
> static bool ffa_get_version(uint32_t *vers)
> {
>     const struct arm_smccc_1_2_regs arg = {
> @@ -217,6 +226,17 @@ static bool check_mandatory_feature(uint32_t id)
>     return !ret;
> }
> 
> +static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
> +                            uint32_t page_count)

Using register_t type here is doing an implicit cast when called and on
32bit this might later remove part of the address.
This function must take paddr_t as parameters.

> +{
> +    uint32_t fid = FFA_RXTX_MAP_32;
> +
> +    if ( IS_ENABLED(CONFIG_ARM_64) )
> +        fid = FFA_RXTX_MAP_64;
> +
> +    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);

simple call might not be suitable on 32bits due to the conversion.
As said earlier, it would make more sense to disable FFA on 32bit and
put some comments/build_bug_on in the code in places where there
would be something to fix.

> +}
> +
> static uint16_t get_vm_id(const struct domain *d)
> {
>     /* +1 since 0 is reserved for the hypervisor in FF-A */
> @@ -384,6 +404,7 @@ static int ffa_relinquish_resources(struct domain *d)
> static bool ffa_probe(void)
> {
>     uint32_t vers;
> +    int e;
>     unsigned int major_vers;
>     unsigned int minor_vers;
> 
> @@ -426,12 +447,46 @@ static bool ffa_probe(void)
>     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>            major_vers, minor_vers);
> 
> -    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> +    if (
> +#ifdef CONFIG_ARM_64
> +         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> +#endif
> +#ifdef CONFIG_ARM_32
> +         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> +#endif
> +         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> +         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>         return false;
> 
> +    ffa_rx = alloc_xenheap_pages(0, 0);
> +    if ( !ffa_rx )
> +        return false;
> +
> +    ffa_tx = alloc_xenheap_pages(0, 0);
> +    if ( !ffa_tx )
> +        goto err_free_ffa_rx;
> +
> +    e = ffa_rxtx_map(__pa(ffa_tx), __pa(ffa_rx), 1);
> +    if ( e )
> +    {
> +        printk(XENLOG_ERR "ffa: Failed to map rxtx: error %d\n", e);
> +        goto err_free_ffa_tx;
> +    }
> +    ffa_page_count = 1;

ffa_page_count is a constant here and is not used to do the allocation or
passed as parameter to ffa_rxtx_map.

Do you expect this value to be modified ? how ?

Please set it first and use it for allocation and as parameter to rxtx_map so
that a modification of the value would only have to be done in one place.

Please use a define if this is a constant.

As it is a global variable, does the parameter to rxtx_map make sense ?

Cheers
Bertrand

>     ffa_version = vers;
> 
>     return true;
> +
> +err_free_ffa_tx:
> +    free_xenheap_pages(ffa_tx, 0);
> +    ffa_tx = NULL;
> +err_free_ffa_rx:
> +    free_xenheap_pages(ffa_rx, 0);
> +    ffa_rx = NULL;
> +    ffa_page_count = 0;
> +    ffa_version = 0;
> +
> +    return false;
> }
> 
> static const struct tee_mediator_ops ffa_ops =
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 08/20] xen/arm: ffa: note dependency on 4k pages

[Jens Wiklander]

Hi Bertrand,

On Fri, Feb 24, 2023 at 4:27 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> HI Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds a BUILD_BUG_ON() to assert the dependency on 4k pages in the FF-A
> > mediator.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> NIT: I would s/note/enforce/ in the title:
> xen/arm: ffa: enforce 4k pages

OK, I'll fix it.

Thanks,
Jens

>
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
>
> Cheers
> Bertrand
>
> > ---
> > xen/arch/arm/tee/ffa.c | 21 +++++++++++++++++++++
> > 1 file changed, 21 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index d04bac9cc47f..8b0b80ce1ff5 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -56,6 +56,16 @@
> > #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
> >                                                  FFA_MY_VERSION_MINOR)
> >
> > +/*
> > + * The FF-A specification explicitly works with 4K pages as a measure of
> > + * memory size, for example, FFA_RXTX_MAP takes one parameter "RX/TX page
> > + * count" which is the number of contiguous 4K pages allocated. Xen may use
> > + * a different page size depending on the configuration to avoid confusion
> > + * with PAGE_SIZE use a special define when it's a page size as in the FF-A
> > + * specification.
> > + */
> > +#define FFA_PAGE_SIZE                   SZ_4K
> > +
> > /* Framework direct request/response */
> > #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> > #define FFA_MSG_TYPE_MASK               0xFFU;
> > @@ -242,6 +252,17 @@ static bool ffa_probe(void)
> >     unsigned int major_vers;
> >     unsigned int minor_vers;
> >
> > +    /*
> > +     * FF-A often works in units of 4K pages and currently it's assumed
> > +     * that we can map memory using that granularity. See also the comment
> > +     * above the FFA_PAGE_SIZE define.
> > +     *
> > +     * It is possible to support a PAGE_SIZE larger than 4K in Xen, but
> > +     * until that is fully handled in this code make sure that we only use
> > +     * 4K page sizes.
> > +     */
> > +    BUILD_BUG_ON(PAGE_SIZE != FFA_PAGE_SIZE);
> > +
> >     /*
> >      * psci_init_smccc() updates this value with what's reported by EL-3
> >      * or secure world.
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 09/20] xen/arm: ffa: add support for FFA_ID_GET

[Jens Wiklander]

Hi,

On Mon, Feb 27, 2023 at 4:00 PM Julien Grall <julien@xen.org> wrote:
>
> Hi,
>
> On 27/02/2023 14:48, Bertrand Marquis wrote:
> >> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>
> >> Adds support for the FF-A function FFA_ID_GET to return the ID of the
> >> calling client.
> >>
> >> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >> ---
> >> xen/arch/arm/tee/ffa.c | 21 ++++++++++++++++++++-
> >> 1 file changed, 20 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >> index 8b0b80ce1ff5..463fd7730573 100644
> >> --- a/xen/arch/arm/tee/ffa.c
> >> +++ b/xen/arch/arm/tee/ffa.c
> >> @@ -167,6 +167,12 @@ static bool ffa_get_version(uint32_t *vers)
> >>      return true;
> >> }
> >>
> >> +static uint16_t get_vm_id(const struct domain *d)
> >> +{
> >> +    /* +1 since 0 is reserved for the hypervisor in FF-A */
> >> +    return d->domain_id + 1;
> >> +}
> >> +
> >> static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> >>                       register_t v2, register_t v3, register_t v4, register_t v5,
> >>                       register_t v6, register_t v7)
> >> @@ -181,6 +187,12 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> >>          set_user_reg(regs, 7, v7);
> >> }
> >>
> >> +static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> >> +                             uint32_t w3)
> >> +{
> >> +    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
> >> +}
> >> +
> >> static void handle_version(struct cpu_user_regs *regs)
> >> {
> >>      struct domain *d = current->domain;
> >> @@ -210,6 +222,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>      case FFA_VERSION:
> >>          handle_version(regs);
> >>          return true;
> >> +    case FFA_ID_GET:
> >> +        set_regs_success(regs, get_vm_id(d), 0);
> >> +        return true;
> >>
> >>      default:
> >>          gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> >> @@ -221,7 +236,11 @@ static int ffa_domain_init(struct domain *d)
> >> {
> >>      struct ffa_ctx *ctx;
> >>
> >> -    if ( !ffa_version )
> >> +     /*
> >> +      * We can't use that last possible domain ID or get_vm_id() would cause
> >> +      * an overflow.
> >> +      */
> >> +    if ( !ffa_version || d->domain_id == UINT16_MAX)
> >>          return -ENODEV;
> >
> > In reality the overflow could only happen if this is called by the IDLE domain right now.
> > Anyway this could change and this is making the code more robust at no real cost.
> >
> > I would just suggest here to return a different error code like ERANGE for example to
> > prevent missing ENODEV with other cases not related to FFA not being available.
>
> +1. I would also like to suggest to use >= rather than == in case we
> decide to support more than 16-bit domid.

Makes sense, I'll fix it.

Thanks,
Jens

>
> Cheers,
>
> --
> Julien Grall

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> The FF-A specification defines framework messages sent as direct
> requests when certain events occurs. For instance when a VM (guest) is
> created or destroyed. Only SPs which have subscribed to these events
> will receive them. An SP can subscribe to these messages in its
> partition properties.
> 
> Adds a check that the SP supports the needed FF-A features
> FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
> 
> The partition properties of each SP is retrieved with
> FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> caller (us), so once we're done with the buffer it must be released
> using FFA_RX_RELEASE before another call can be made.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 190 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 07dd5c36d54b..f1b014b6c7f4 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -140,6 +140,14 @@
> #define FFA_MSG_SEND                    0x8400006EU
> #define FFA_MSG_POLL                    0x8400006AU
> 
> +/* Partition information descriptor */
> +struct ffa_partition_info_1_1 {
> +    uint16_t id;
> +    uint16_t execution_context;
> +    uint32_t partition_properties;
> +    uint8_t uuid[16];
> +};
> +
> struct ffa_ctx {
>     uint32_t guest_vers;
>     bool interrupted;
> @@ -148,6 +156,12 @@ struct ffa_ctx {
> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
> +static uint16_t *subscr_vm_created __read_mostly;
> +static unsigned int subscr_vm_created_count __read_mostly;

In the spec the number is returned in w2 so you should utse uint32_t here.

> +static uint16_t *subscr_vm_destroyed __read_mostly;
> +static unsigned int subscr_vm_destroyed_count __read_mostly;

Same here

> +
> /*
>  * Our rx/tx buffers shared with the SPMC.
>  *
> @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
>     return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
> }
> 
> +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> +                                      uint32_t w4, uint32_t w5,
> +                                      uint32_t *count)
> +{
> +    const struct arm_smccc_1_2_regs arg = {
> +        .a0 = FFA_PARTITION_INFO_GET,
> +        .a1 = w1,
> +        .a2 = w2,
> +        .a3 = w3,
> +        .a4 = w4,
> +        .a5 = w5,
> +    };
> +    struct arm_smccc_1_2_regs resp;
> +    uint32_t ret;
> +
> +    arm_smccc_1_2_smc(&arg, &resp);
> +
> +    ret = get_ffa_ret_code(&resp);
> +    if ( !ret )
> +        *count = resp.a2;
> +
> +    return ret;
> +}
> +
> +static int32_t ffa_rx_release(void)
> +{
> +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> +}
> +
> +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
> +                                      uint8_t msg)

This function is actually only useable to send framework created/destroyed
messages so the function name should be adapted to be less generic.

ffa_send_vm_events ?

unless you want to modify it later to send more framework messages ?

> +{
> +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
> +    int32_t res;
> +
> +    if ( msg == FFA_MSG_SEND_VM_CREATED )
> +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
> +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
> +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
> +    else
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    do {
> +        const struct arm_smccc_1_2_regs arg = {
> +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
> +            .a1 = sp_id,
> +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
> +            .a5 = vm_id,
> +        };
> +        struct arm_smccc_1_2_regs resp;
> +
> +        arm_smccc_1_2_smc(&arg, &resp);
> +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
> +        {
> +            /*
> +             * This is an invalid response, likely due to some error in the
> +             * implementation of the ABI.
> +             */
> +            return FFA_RET_INVALID_PARAMETERS;
> +        }
> +        res = resp.a3;
> +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );

We might end up in an infinite loop here or increase interrupt response time.
In the general case we could return that code directly to the VM but here we
are in the VM creation/destroy path so we cannot do that.

Maybe in debug mode at least we should have a retry counter here for now
with a print ?

> +
> +    return res;
> +}
> +
> static uint16_t get_vm_id(const struct domain *d)
> {
>     /* +1 since 0 is reserved for the hypervisor in FF-A */
> @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> static int ffa_domain_init(struct domain *d)
> {
>     struct ffa_ctx *ctx;
> +    unsigned int n;
> +    unsigned int m;
> +    unsigned int c_pos;
> +    int32_t res;
> 
>      /*
>       * We can't use that last possible domain ID or get_vm_id() would cause
> @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
>     if ( !ctx )
>         return -ENOMEM;
> 
> +    for ( n = 0; n < subscr_vm_created_count; n++ )
> +    {
> +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
> +                                     FFA_MSG_SEND_VM_CREATED);
> +        if ( res )
> +        {
> +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
> +                   get_vm_id(d), subscr_vm_created[n], res);

in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
instead of calling get_vm_id each time.

> +            c_pos = n;
> +            goto err;
> +        }
> +    }
> +
>     d->arch.tee = ctx;
> 
>     return 0;
> +
> +err:
> +    /* Undo any already sent vm created messaged */
> +    for ( n = 0; n < c_pos; n++ )
> +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
> +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
> +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> +                                       FFA_MSG_SEND_VM_DESTROYED);
> +
> +    return -ENOMEM;

The VM creation is not failing due to missing memory here.
We need to find a better error code.
Maybe ENOTCONN ?
I am open to ideas here :-)

> }
> 
> /* This function is supposed to undo what ffa_domain_init() has done */
> static int ffa_relinquish_resources(struct domain *d)
> {
>     struct ffa_ctx *ctx = d->arch.tee;
> +    unsigned int n;
> +    int32_t res;
> 
>     if ( !ctx )
>         return 0;
> 
> +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
> +    {
> +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> +                                     FFA_MSG_SEND_VM_DESTROYED);
> +
> +        if ( res )
> +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
> +                   get_vm_id(d), subscr_vm_destroyed[n], res);
> +    }
> +
>     XFREE(d->arch.tee);
> 
>     return 0;
> }
> 
> +static bool init_subscribers(void)
> +{
> +    struct ffa_partition_info_1_1 *fpi;
> +    bool ret = false;
> +    uint32_t count;
> +    int e;
> +    uint32_t n;
> +    uint32_t c_pos;
> +    uint32_t d_pos;
> +
> +    if ( ffa_version < FFA_VERSION_1_1 )
> +        return true;

Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
we could do the slow path and do a first loop on info_get until we get an error ?

> +
> +    e = ffa_partition_info_get(0, 0, 0, 0, 0, &count);
> +    if ( e )
> +    {
> +        printk(XENLOG_ERR "ffa: Failed to get list of SPs: %d\n", e);
> +        goto out;
> +    }
> +
> +    fpi = ffa_rx;
> +    subscr_vm_created_count = 0;
> +    subscr_vm_destroyed_count = 0;
> +    for ( n = 0; n < count; n++ )
> +    {
> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED)
> +            subscr_vm_created_count++;
> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED)
> +            subscr_vm_destroyed_count++;
> +    }
> +
> +    if ( subscr_vm_created_count )
> +        subscr_vm_created = xzalloc_array(uint16_t, subscr_vm_created_count);
> +    if ( subscr_vm_destroyed_count )
> +        subscr_vm_destroyed = xzalloc_array(uint16_t,
> +                                            subscr_vm_destroyed_count);
> +    if ( (subscr_vm_created_count && !subscr_vm_created) ||
> +         (subscr_vm_destroyed_count && !subscr_vm_destroyed) )
> +    {
> +        printk(XENLOG_ERR "ffa: Failed to allocate subscription lists\n");
> +        subscr_vm_created_count = 0;
> +        subscr_vm_destroyed_count = 0;
> +        XFREE(subscr_vm_created);
> +        XFREE(subscr_vm_destroyed);
> +        goto out;
> +    }
> +
> +    for ( c_pos = 0, d_pos = 0, n = 0; n < count; n++ )
> +    {
> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED )
> +            subscr_vm_created[c_pos++] = fpi[n].id;
> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED )
> +            subscr_vm_destroyed[d_pos++] = fpi[n].id;
> +    }
> +
> +    ret = true;
> +out:
> +    ffa_rx_release();
> +
> +    return ret;
> +}
> +
> static bool ffa_probe(void)
> {
>     uint32_t vers;
> @@ -447,7 +628,8 @@ static bool ffa_probe(void)
>     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>            major_vers, minor_vers);
> 
> -    if (
> +    if ( !check_mandatory_feature(FFA_PARTITION_INFO_GET) ||
> +         !check_mandatory_feature(FFA_RX_RELEASE) ||
> #ifdef CONFIG_ARM_64
>          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> #endif
> @@ -475,6 +657,9 @@ static bool ffa_probe(void)
>     ffa_page_count = 1;
>     ffa_version = vers;
> 
> +    if ( !init_subscribers() )
> +        goto err_free_ffa_tx;
> +
>     return true;
> 
> err_free_ffa_tx:
> @@ -485,6 +670,10 @@ err_free_ffa_rx:
>     ffa_rx = NULL;
>     ffa_page_count = 0;
>     ffa_version = 0;
> +    XFREE(subscr_vm_created);
> +    subscr_vm_created_count = 0;
> +    XFREE(subscr_vm_destroyed);
> +    subscr_vm_destroyed_count = 0;
> 
>     return false;
> }
> -- 
> 2.34.1
> 

Cheers
Bertrand

Re: [XEN PATCH v7 11/20] xen/arm: ffa: map SPMC rx/tx buffers

[Jens Wiklander]

Hi Bertrand,

On Tue, Feb 28, 2023 at 1:57 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > When initializing the FF-A mediator map the RX and TX buffers shared with
> > the SPMC.
> >
> > These buffer are later used to to transmit data that cannot be passed in
> > registers only.
> >
> > Adds a check that the SP supports the needed FF-A features
> > FFA_RXTX_MAP_64 / FFA_RXTX_MAP_32 and FFA_RXTX_UNMAP. In 64-bit mode we
> > must use FFA_RXTX_MAP_64 since registers are used to transmit the
> > physical addresses of the RX/TX buffers.
>
> Right now, FFA on 32bit would only work correctly if LPAE is not used and only addresses
> under 4G are used by Xen and by guests as addresses are transferred through a single register.
>
> I think that we need for now to only enable FFA support on 64bit as the limitations we
> would need to enforce on 32bit are complex and the use case for FFA on 32bit platforms
> is not that obvious now.

OK, I'll drop the #ifdef CONFIG_ARM_64 and #ifdef CONFIG_ARM_32 and
instead depend on ARM_64 in Kconfig.
If we ever want to use this on ARM_32 we'll have to go through
everything anyway.

>
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 57 +++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 56 insertions(+), 1 deletion(-)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index a5d8a12635b6..07dd5c36d54b 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -148,6 +148,15 @@ struct ffa_ctx {
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > +/*
> > + * Our rx/tx buffers shared with the SPMC.
> > + *
> > + * ffa_page_count is the number of pages used in each of these buffers.
> > + */
> > +static void *ffa_rx __read_mostly;
> > +static void *ffa_tx __read_mostly;
> > +static unsigned int ffa_page_count __read_mostly;
> > +
> > static bool ffa_get_version(uint32_t *vers)
> > {
> >     const struct arm_smccc_1_2_regs arg = {
> > @@ -217,6 +226,17 @@ static bool check_mandatory_feature(uint32_t id)
> >     return !ret;
> > }
> >
> > +static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
> > +                            uint32_t page_count)
>
> Using register_t type here is doing an implicit cast when called and on
> 32bit this might later remove part of the address.
> This function must take paddr_t as parameters.

I'll change to paddr_t for rx/tx.

>
> > +{
> > +    uint32_t fid = FFA_RXTX_MAP_32;
> > +
> > +    if ( IS_ENABLED(CONFIG_ARM_64) )
> > +        fid = FFA_RXTX_MAP_64;
> > +
> > +    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
>
> simple call might not be suitable on 32bits due to the conversion.
> As said earlier, it would make more sense to disable FFA on 32bit and
> put some comments/build_bug_on in the code in places where there
> would be something to fix.

I'm dropping the 32-bit support.

>
> > +}
> > +
> > static uint16_t get_vm_id(const struct domain *d)
> > {
> >     /* +1 since 0 is reserved for the hypervisor in FF-A */
> > @@ -384,6 +404,7 @@ static int ffa_relinquish_resources(struct domain *d)
> > static bool ffa_probe(void)
> > {
> >     uint32_t vers;
> > +    int e;
> >     unsigned int major_vers;
> >     unsigned int minor_vers;
> >
> > @@ -426,12 +447,46 @@ static bool ffa_probe(void)
> >     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> >            major_vers, minor_vers);
> >
> > -    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> > +    if (
> > +#ifdef CONFIG_ARM_64
> > +         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> > +#endif
> > +#ifdef CONFIG_ARM_32
> > +         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> > +#endif
> > +         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> > +         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >         return false;
> >
> > +    ffa_rx = alloc_xenheap_pages(0, 0);
> > +    if ( !ffa_rx )
> > +        return false;
> > +
> > +    ffa_tx = alloc_xenheap_pages(0, 0);
> > +    if ( !ffa_tx )
> > +        goto err_free_ffa_rx;
> > +
> > +    e = ffa_rxtx_map(__pa(ffa_tx), __pa(ffa_rx), 1);
> > +    if ( e )
> > +    {
> > +        printk(XENLOG_ERR "ffa: Failed to map rxtx: error %d\n", e);
> > +        goto err_free_ffa_tx;
> > +    }
> > +    ffa_page_count = 1;
>
> ffa_page_count is a constant here and is not used to do the allocation or
> passed as parameter to ffa_rxtx_map.
>
> Do you expect this value to be modified ? how ?

I expect this value to match how many FFA_PAGE_SIZE pages have been
mapped for the RX/TX buffers. Currently, this is only 1 but will have
to be changed later if PAGE_SIZE in Xen or in the secure world is
larger than FFA_PAGE_SIZE. We may also later add support
configurations where RX/TX buffers aren't mapped.

>
> Please set it first and use it for allocation and as parameter to rxtx_map so
> that a modification of the value would only have to be done in one place.
>
> Please use a define if this is a constant.

How about adding a define FFA_MIN_RXTX_PAGE_COUNT and giving that to
ffa_rxtx_map() and later assigning it to ffa_page_count if the call
succeeds?

>
> As it is a global variable, does the parameter to rxtx_map make sense ?

Yes, ffa_rxtx_map() is a dumb wrapper so it should have all the needed
parameters for the SMC provided.

Cheers,
Jens

>
> Cheers
> Bertrand
>
> >     ffa_version = vers;
> >
> >     return true;
> > +
> > +err_free_ffa_tx:
> > +    free_xenheap_pages(ffa_tx, 0);
> > +    ffa_tx = NULL;
> > +err_free_ffa_rx:
> > +    free_xenheap_pages(ffa_rx, 0);
> > +    ffa_rx = NULL;
> > +    ffa_page_count = 0;
> > +    ffa_version = 0;
> > +
> > +    return false;
> > }
> >
> > static const struct tee_mediator_ops ffa_ops =
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 11/20] xen/arm: ffa: map SPMC rx/tx buffers

[Bertrand Marquis]

Hi Jens,

> On 1 Mar 2023, at 10:30, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Tue, Feb 28, 2023 at 1:57 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> When initializing the FF-A mediator map the RX and TX buffers shared with
>>> the SPMC.
>>> 
>>> These buffer are later used to to transmit data that cannot be passed in
>>> registers only.
>>> 
>>> Adds a check that the SP supports the needed FF-A features
>>> FFA_RXTX_MAP_64 / FFA_RXTX_MAP_32 and FFA_RXTX_UNMAP. In 64-bit mode we
>>> must use FFA_RXTX_MAP_64 since registers are used to transmit the
>>> physical addresses of the RX/TX buffers.
>> 
>> Right now, FFA on 32bit would only work correctly if LPAE is not used and only addresses
>> under 4G are used by Xen and by guests as addresses are transferred through a single register.
>> 
>> I think that we need for now to only enable FFA support on 64bit as the limitations we
>> would need to enforce on 32bit are complex and the use case for FFA on 32bit platforms
>> is not that obvious now.
> 
> OK, I'll drop the #ifdef CONFIG_ARM_64 and #ifdef CONFIG_ARM_32 and
> instead depend on ARM_64 in Kconfig.
> If we ever want to use this on ARM_32 we'll have to go through
> everything anyway.

Yes this is the best solution for now.
And support.md patch is already saying already arm64.

> 
>> 
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 57 +++++++++++++++++++++++++++++++++++++++++-
>>> 1 file changed, 56 insertions(+), 1 deletion(-)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index a5d8a12635b6..07dd5c36d54b 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -148,6 +148,15 @@ struct ffa_ctx {
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> +/*
>>> + * Our rx/tx buffers shared with the SPMC.
>>> + *
>>> + * ffa_page_count is the number of pages used in each of these buffers.
>>> + */
>>> +static void *ffa_rx __read_mostly;
>>> +static void *ffa_tx __read_mostly;
>>> +static unsigned int ffa_page_count __read_mostly;
>>> +
>>> static bool ffa_get_version(uint32_t *vers)
>>> {
>>>    const struct arm_smccc_1_2_regs arg = {
>>> @@ -217,6 +226,17 @@ static bool check_mandatory_feature(uint32_t id)
>>>    return !ret;
>>> }
>>> 
>>> +static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
>>> +                            uint32_t page_count)
>> 
>> Using register_t type here is doing an implicit cast when called and on
>> 32bit this might later remove part of the address.
>> This function must take paddr_t as parameters.
> 
> I'll change to paddr_t for rx/tx.
> 
>> 
>>> +{
>>> +    uint32_t fid = FFA_RXTX_MAP_32;
>>> +
>>> +    if ( IS_ENABLED(CONFIG_ARM_64) )
>>> +        fid = FFA_RXTX_MAP_64;
>>> +
>>> +    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
>> 
>> simple call might not be suitable on 32bits due to the conversion.
>> As said earlier, it would make more sense to disable FFA on 32bit and
>> put some comments/build_bug_on in the code in places where there
>> would be something to fix.
> 
> I'm dropping the 32-bit support.
> 
>> 
>>> +}
>>> +
>>> static uint16_t get_vm_id(const struct domain *d)
>>> {
>>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
>>> @@ -384,6 +404,7 @@ static int ffa_relinquish_resources(struct domain *d)
>>> static bool ffa_probe(void)
>>> {
>>>    uint32_t vers;
>>> +    int e;
>>>    unsigned int major_vers;
>>>    unsigned int minor_vers;
>>> 
>>> @@ -426,12 +447,46 @@ static bool ffa_probe(void)
>>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>>>           major_vers, minor_vers);
>>> 
>>> -    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>> +    if (
>>> +#ifdef CONFIG_ARM_64
>>> +         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
>>> +#endif
>>> +#ifdef CONFIG_ARM_32
>>> +         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
>>> +#endif
>>> +         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
>>> +         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>>        return false;
>>> 
>>> +    ffa_rx = alloc_xenheap_pages(0, 0);
>>> +    if ( !ffa_rx )
>>> +        return false;
>>> +
>>> +    ffa_tx = alloc_xenheap_pages(0, 0);
>>> +    if ( !ffa_tx )
>>> +        goto err_free_ffa_rx;
>>> +
>>> +    e = ffa_rxtx_map(__pa(ffa_tx), __pa(ffa_rx), 1);
>>> +    if ( e )
>>> +    {
>>> +        printk(XENLOG_ERR "ffa: Failed to map rxtx: error %d\n", e);
>>> +        goto err_free_ffa_tx;
>>> +    }
>>> +    ffa_page_count = 1;
>> 
>> ffa_page_count is a constant here and is not used to do the allocation or
>> passed as parameter to ffa_rxtx_map.
>> 
>> Do you expect this value to be modified ? how ?
> 
> I expect this value to match how many FFA_PAGE_SIZE pages have been
> mapped for the RX/TX buffers. Currently, this is only 1 but will have
> to be changed later if PAGE_SIZE in Xen or in the secure world is
> larger than FFA_PAGE_SIZE. We may also later add support
> configurations where RX/TX buffers aren't mapped.

So it is a constant and the buffers are just mapped or not mapped.

> 
>> 
>> Please set it first and use it for allocation and as parameter to rxtx_map so
>> that a modification of the value would only have to be done in one place.
>> 
>> Please use a define if this is a constant.
> 
> How about adding a define FFA_MIN_RXTX_PAGE_COUNT and giving that to
> ffa_rxtx_map() and later assigning it to ffa_page_count if the call
> succeeds?

Why MIN ?

How about just using ffa_rx or ffa_tx being NULL or not to check if the buffers are
mapped and remove the count.

> 
>> 
>> As it is a global variable, does the parameter to rxtx_map make sense ?
> 
> Yes, ffa_rxtx_map() is a dumb wrapper so it should have all the needed
> parameters for the SMC provided.

Then passing FFA_MIN_RXTX_PAGE_COUNT should be enough.

Cheers
Bertrand

> 
> Cheers,
> Jens
> 
>> 
>> Cheers
>> Bertrand
>> 
>>>    ffa_version = vers;
>>> 
>>>    return true;
>>> +
>>> +err_free_ffa_tx:
>>> +    free_xenheap_pages(ffa_tx, 0);
>>> +    ffa_tx = NULL;
>>> +err_free_ffa_rx:
>>> +    free_xenheap_pages(ffa_rx, 0);
>>> +    ffa_rx = NULL;
>>> +    ffa_page_count = 0;
>>> +    ffa_version = 0;
>>> +
>>> +    return false;
>>> }
>>> 
>>> static const struct tee_mediator_ops ffa_ops =
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Jens Wiklander]

Hi Bertrand,

On Tue, Feb 28, 2023 at 5:49 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > The FF-A specification defines framework messages sent as direct
> > requests when certain events occurs. For instance when a VM (guest) is
> > created or destroyed. Only SPs which have subscribed to these events
> > will receive them. An SP can subscribe to these messages in its
> > partition properties.
> >
> > Adds a check that the SP supports the needed FF-A features
> > FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
> >
> > The partition properties of each SP is retrieved with
> > FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
> > Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> > caller (us), so once we're done with the buffer it must be released
> > using FFA_RX_RELEASE before another call can be made.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 190 insertions(+), 1 deletion(-)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 07dd5c36d54b..f1b014b6c7f4 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -140,6 +140,14 @@
> > #define FFA_MSG_SEND                    0x8400006EU
> > #define FFA_MSG_POLL                    0x8400006AU
> >
> > +/* Partition information descriptor */
> > +struct ffa_partition_info_1_1 {
> > +    uint16_t id;
> > +    uint16_t execution_context;
> > +    uint32_t partition_properties;
> > +    uint8_t uuid[16];
> > +};
> > +
> > struct ffa_ctx {
> >     uint32_t guest_vers;
> >     bool interrupted;
> > @@ -148,6 +156,12 @@ struct ffa_ctx {
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
> > +static uint16_t *subscr_vm_created __read_mostly;
> > +static unsigned int subscr_vm_created_count __read_mostly;
>
> In the spec the number is returned in w2 so you should utse uint32_t here.

I don't understand. This value is increased for each SP which has the
property set in the Partition information descriptor.

>
> > +static uint16_t *subscr_vm_destroyed __read_mostly;
> > +static unsigned int subscr_vm_destroyed_count __read_mostly;
>
> Same here
>
> > +
> > /*
> >  * Our rx/tx buffers shared with the SPMC.
> >  *
> > @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
> >     return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
> > }
> >
> > +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> > +                                      uint32_t w4, uint32_t w5,
> > +                                      uint32_t *count)
> > +{
> > +    const struct arm_smccc_1_2_regs arg = {
> > +        .a0 = FFA_PARTITION_INFO_GET,
> > +        .a1 = w1,
> > +        .a2 = w2,
> > +        .a3 = w3,
> > +        .a4 = w4,
> > +        .a5 = w5,
> > +    };
> > +    struct arm_smccc_1_2_regs resp;
> > +    uint32_t ret;
> > +
> > +    arm_smccc_1_2_smc(&arg, &resp);
> > +
> > +    ret = get_ffa_ret_code(&resp);
> > +    if ( !ret )
> > +        *count = resp.a2;
> > +
> > +    return ret;
> > +}
> > +
> > +static int32_t ffa_rx_release(void)
> > +{
> > +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> > +}
> > +
> > +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
> > +                                      uint8_t msg)
>
> This function is actually only useable to send framework created/destroyed
> messages so the function name should be adapted to be less generic.
>
> ffa_send_vm_events ?
>
> unless you want to modify it later to send more framework messages ?

That was the plan, but that may never happen. I'll rename it to
ffa_send_vm_event() since we're only sending one event at a time.

>
> > +{
> > +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
> > +    int32_t res;
> > +
> > +    if ( msg == FFA_MSG_SEND_VM_CREATED )
> > +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
> > +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
> > +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
> > +    else
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    do {
> > +        const struct arm_smccc_1_2_regs arg = {
> > +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
> > +            .a1 = sp_id,
> > +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
> > +            .a5 = vm_id,
> > +        };
> > +        struct arm_smccc_1_2_regs resp;
> > +
> > +        arm_smccc_1_2_smc(&arg, &resp);
> > +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
> > +        {
> > +            /*
> > +             * This is an invalid response, likely due to some error in the
> > +             * implementation of the ABI.
> > +             */
> > +            return FFA_RET_INVALID_PARAMETERS;
> > +        }
> > +        res = resp.a3;
> > +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
>
> We might end up in an infinite loop here or increase interrupt response time.
> In the general case we could return that code directly to the VM but here we
> are in the VM creation/destroy path so we cannot do that.
>
> Maybe in debug mode at least we should have a retry counter here for now
> with a print ?

OK, I'll add something.

>
> > +
> > +    return res;
> > +}
> > +
> > static uint16_t get_vm_id(const struct domain *d)
> > {
> >     /* +1 since 0 is reserved for the hypervisor in FF-A */
> > @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> > static int ffa_domain_init(struct domain *d)
> > {
> >     struct ffa_ctx *ctx;
> > +    unsigned int n;
> > +    unsigned int m;
> > +    unsigned int c_pos;
> > +    int32_t res;
> >
> >      /*
> >       * We can't use that last possible domain ID or get_vm_id() would cause
> > @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
> >     if ( !ctx )
> >         return -ENOMEM;
> >
> > +    for ( n = 0; n < subscr_vm_created_count; n++ )
> > +    {
> > +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
> > +                                     FFA_MSG_SEND_VM_CREATED);
> > +        if ( res )
> > +        {
> > +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
> > +                   get_vm_id(d), subscr_vm_created[n], res);
>
> in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
> instead of calling get_vm_id each time.

OK

>
> > +            c_pos = n;
> > +            goto err;
> > +        }
> > +    }
> > +
> >     d->arch.tee = ctx;
> >
> >     return 0;
> > +
> > +err:
> > +    /* Undo any already sent vm created messaged */
> > +    for ( n = 0; n < c_pos; n++ )
> > +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
> > +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
> > +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> > +                                       FFA_MSG_SEND_VM_DESTROYED);
> > +
> > +    return -ENOMEM;
>
> The VM creation is not failing due to missing memory here.
> We need to find a better error code.
> Maybe ENOTCONN ?
> I am open to ideas here :-)

That makes sense, I'll change it to ENOTCONN.

>
> > }
> >
> > /* This function is supposed to undo what ffa_domain_init() has done */
> > static int ffa_relinquish_resources(struct domain *d)
> > {
> >     struct ffa_ctx *ctx = d->arch.tee;
> > +    unsigned int n;
> > +    int32_t res;
> >
> >     if ( !ctx )
> >         return 0;
> >
> > +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
> > +    {
> > +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> > +                                     FFA_MSG_SEND_VM_DESTROYED);
> > +
> > +        if ( res )
> > +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
> > +                   get_vm_id(d), subscr_vm_destroyed[n], res);
> > +    }
> > +
> >     XFREE(d->arch.tee);
> >
> >     return 0;
> > }
> >
> > +static bool init_subscribers(void)
> > +{
> > +    struct ffa_partition_info_1_1 *fpi;
> > +    bool ret = false;
> > +    uint32_t count;
> > +    int e;
> > +    uint32_t n;
> > +    uint32_t c_pos;
> > +    uint32_t d_pos;
> > +
> > +    if ( ffa_version < FFA_VERSION_1_1 )
> > +        return true;
>
> Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
> we could do the slow path and do a first loop on info_get until we get an error ?

Sending the events is not supported in 1.0 so there's nothing to
record in that case.

Thanks,
Jens

>
> > +
> > +    e = ffa_partition_info_get(0, 0, 0, 0, 0, &count);
> > +    if ( e )
> > +    {
> > +        printk(XENLOG_ERR "ffa: Failed to get list of SPs: %d\n", e);
> > +        goto out;
> > +    }
> > +
> > +    fpi = ffa_rx;
> > +    subscr_vm_created_count = 0;
> > +    subscr_vm_destroyed_count = 0;
> > +    for ( n = 0; n < count; n++ )
> > +    {
> > +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED)
> > +            subscr_vm_created_count++;
> > +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED)
> > +            subscr_vm_destroyed_count++;
> > +    }
> > +
> > +    if ( subscr_vm_created_count )
> > +        subscr_vm_created = xzalloc_array(uint16_t, subscr_vm_created_count);
> > +    if ( subscr_vm_destroyed_count )
> > +        subscr_vm_destroyed = xzalloc_array(uint16_t,
> > +                                            subscr_vm_destroyed_count);
> > +    if ( (subscr_vm_created_count && !subscr_vm_created) ||
> > +         (subscr_vm_destroyed_count && !subscr_vm_destroyed) )
> > +    {
> > +        printk(XENLOG_ERR "ffa: Failed to allocate subscription lists\n");
> > +        subscr_vm_created_count = 0;
> > +        subscr_vm_destroyed_count = 0;
> > +        XFREE(subscr_vm_created);
> > +        XFREE(subscr_vm_destroyed);
> > +        goto out;
> > +    }
> > +
> > +    for ( c_pos = 0, d_pos = 0, n = 0; n < count; n++ )
> > +    {
> > +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED )
> > +            subscr_vm_created[c_pos++] = fpi[n].id;
> > +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED )
> > +            subscr_vm_destroyed[d_pos++] = fpi[n].id;
> > +    }
> > +
> > +    ret = true;
> > +out:
> > +    ffa_rx_release();
> > +
> > +    return ret;
> > +}
> > +
> > static bool ffa_probe(void)
> > {
> >     uint32_t vers;
> > @@ -447,7 +628,8 @@ static bool ffa_probe(void)
> >     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> >            major_vers, minor_vers);
> >
> > -    if (
> > +    if ( !check_mandatory_feature(FFA_PARTITION_INFO_GET) ||
> > +         !check_mandatory_feature(FFA_RX_RELEASE) ||
> > #ifdef CONFIG_ARM_64
> >          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> > #endif
> > @@ -475,6 +657,9 @@ static bool ffa_probe(void)
> >     ffa_page_count = 1;
> >     ffa_version = vers;
> >
> > +    if ( !init_subscribers() )
> > +        goto err_free_ffa_tx;
> > +
> >     return true;
> >
> > err_free_ffa_tx:
> > @@ -485,6 +670,10 @@ err_free_ffa_rx:
> >     ffa_rx = NULL;
> >     ffa_page_count = 0;
> >     ffa_version = 0;
> > +    XFREE(subscr_vm_created);
> > +    subscr_vm_created_count = 0;
> > +    XFREE(subscr_vm_destroyed);
> > +    subscr_vm_destroyed_count = 0;
> >
> >     return false;
> > }
> > --
> > 2.34.1
> >
>
> Cheers
> Bertrand
>
>

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Jens Wiklander]

Hi Bertrand,

On Mon, Feb 27, 2023 at 4:28 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds support for sending a FF-A direct request. Checks that the SP also
> > supports handling a 32-bit direct request. 64-bit direct requests are
> > not used by the mediator itself so there is not need to check for that.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 119 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 463fd7730573..a5d8a12635b6 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -142,6 +142,7 @@
> >
> > struct ffa_ctx {
> >     uint32_t guest_vers;
> > +    bool interrupted;
>
> This is added and set here for one special error code but is never used.
> I would suggest to introduce this when there will be an action based on it.

I'm sorry, I forgot about completing this. I'll add code to deal with
FFA_INTERRUPT. This will be tricky to test though since we don't use
FFA_INTERRUPT like this with OP-TEE. The Hypervisor is required by the
FF-A standard to support it so I better add something.

>
> > };
> >
> > /* Negotiated FF-A version to use with the SPMC */
> > @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
> >     return true;
> > }
> >
> > +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
> > +{
> > +    switch ( resp->a0 )
> > +    {
> > +    case FFA_ERROR:
> > +        if ( resp->a2 )
> > +            return resp->a2;
> > +        else
> > +            return FFA_RET_NOT_SUPPORTED;
> > +    case FFA_SUCCESS_32:
> > +    case FFA_SUCCESS_64:
> > +        return FFA_RET_OK;
> > +    default:
> > +        return FFA_RET_NOT_SUPPORTED;
> > +    }
> > +}
> > +
> > +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
> > +                               register_t a3, register_t a4)
> > +{
> > +    const struct arm_smccc_1_2_regs arg = {
> > +        .a0 = fid,
> > +        .a1 = a1,
> > +        .a2 = a2,
> > +        .a3 = a3,
> > +        .a4 = a4,
> > +    };
> > +    struct arm_smccc_1_2_regs resp;
> > +
> > +    arm_smccc_1_2_smc(&arg, &resp);
> > +
> > +    return get_ffa_ret_code(&resp);
> > +}
> > +
> > +static int32_t ffa_features(uint32_t id)
> > +{
> > +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
> > +}
> > +
> > +static bool check_mandatory_feature(uint32_t id)
> > +{
> > +    uint32_t ret = ffa_features(id);
> > +
> > +    if (ret)
> > +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
>
> It might be useful here to actually print the error code.
> Are we sure that all errors actually mean not supported ?

Yes, that's what the standard says.

>
> > +
> > +    return !ret;
> > +}
> > +
> > static uint16_t get_vm_id(const struct domain *d)
> > {
> >     /* +1 since 0 is reserved for the hypervisor in FF-A */
> > @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
> >     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> > }
> >
> > +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> > +{
> > +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> > +    struct arm_smccc_1_2_regs resp = { };
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +    uint32_t src_dst;
> > +    uint64_t mask;
> > +
> > +    if ( smccc_is_conv_64(fid) )
> > +        mask = GENMASK_ULL(63, 0);
> > +    else
> > +        mask = GENMASK_ULL(31, 0);
> > +
> > +    src_dst = get_user_reg(regs, 1);
> > +    if ( (src_dst >> 16) != get_vm_id(d) )
> > +    {
> > +        resp.a0 = FFA_ERROR;
> > +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
> > +        goto out;
> > +    }
> > +
> > +    arg.a1 = src_dst;
> > +    arg.a2 = get_user_reg(regs, 2) & mask;
> > +    arg.a3 = get_user_reg(regs, 3) & mask;
> > +    arg.a4 = get_user_reg(regs, 4) & mask;
> > +    arg.a5 = get_user_reg(regs, 5) & mask;
> > +    arg.a6 = get_user_reg(regs, 6) & mask;
> > +    arg.a7 = get_user_reg(regs, 7) & mask;
> > +
> > +    while ( true )
> > +    {
> > +        arm_smccc_1_2_smc(&arg, &resp);
> > +
> > +        switch ( resp.a0 )
> > +        {
> > +        case FFA_INTERRUPT:
> > +            ctx->interrupted = true;
> > +            goto out;
> > +        case FFA_ERROR:
> > +        case FFA_SUCCESS_32:
> > +        case FFA_SUCCESS_64:
> > +        case FFA_MSG_SEND_DIRECT_RESP_32:
> > +        case FFA_MSG_SEND_DIRECT_RESP_64:
> > +            goto out;
> > +        default:
> > +            /* Bad fid, report back. */
> > +            memset(&arg, 0, sizeof(arg));
> > +            arg.a0 = FFA_ERROR;
> > +            arg.a1 = src_dst;
> > +            arg.a2 = FFA_RET_NOT_SUPPORTED;
> > +            continue;
>
> There is a potential infinite loop here and i do not understand
> why this needs to be done.
> Here if something is returning a value that you do not understand
> you send back an ERROR to it. I do not find in the spec where this
> is supposed to be done.
> Can you explain a bit here ?

This should normally not happen, but the SP/SPMC is responding with a
request that we don't know what to do with. The standard doesn't say
how to handle that as far as I understand. However, returning back to
the VM at this point with an error may leave the SP/SPMC in a strange
state. So I think it's better to report back to the SP/SPMC that the
request isn't understood and hopefully it can at least return back
with an error in a sane state.

I'll add something to the comment.

>
> > +        }
> > +    }
> > +
> > +out:
> > +    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
> > +             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> > +}
> > +
> > static bool ffa_handle_call(struct cpu_user_regs *regs)
> > {
> >     uint32_t fid = get_user_reg(regs, 0);
> > @@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >     case FFA_ID_GET:
> >         set_regs_success(regs, get_vm_id(d), 0);
> >         return true;
> > +    case FFA_MSG_SEND_DIRECT_REQ_32:
> > +#ifdef CONFIG_ARM_64
> > +    case FFA_MSG_SEND_DIRECT_REQ_64:
> > +#endif
> > +        handle_msg_send_direct_req(regs, fid);
> > +        return true;
> >
> >     default:
> >         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> > @@ -310,6 +426,9 @@ static bool ffa_probe(void)
> >     printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> >            major_vers, minor_vers);
> >
> > +    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> > +        return false;
>
> One could not need this feature and here this will make everything unavailable instead.
> Why not just reporting back the unsupported error to clients using unsupported interfaces ?

One could perhaps argue that this check should be moved to a later
patch in this series. Perhaps there's some future configuration that
might make sense without this feature, but for now, it doesn't make
sense to initialize without it.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > +
> >     ffa_version = vers;
> >
> >     return true;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 11/20] xen/arm: ffa: map SPMC rx/tx buffers

[Jens Wiklander]

Hi,

On Wed, Mar 1, 2023 at 10:56 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 1 Mar 2023, at 10:30, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Tue, Feb 28, 2023 at 1:57 PM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> When initializing the FF-A mediator map the RX and TX buffers shared with
> >>> the SPMC.
> >>>
> >>> These buffer are later used to to transmit data that cannot be passed in
> >>> registers only.
> >>>
> >>> Adds a check that the SP supports the needed FF-A features
> >>> FFA_RXTX_MAP_64 / FFA_RXTX_MAP_32 and FFA_RXTX_UNMAP. In 64-bit mode we
> >>> must use FFA_RXTX_MAP_64 since registers are used to transmit the
> >>> physical addresses of the RX/TX buffers.
> >>
> >> Right now, FFA on 32bit would only work correctly if LPAE is not used and only addresses
> >> under 4G are used by Xen and by guests as addresses are transferred through a single register.
> >>
> >> I think that we need for now to only enable FFA support on 64bit as the limitations we
> >> would need to enforce on 32bit are complex and the use case for FFA on 32bit platforms
> >> is not that obvious now.
> >
> > OK, I'll drop the #ifdef CONFIG_ARM_64 and #ifdef CONFIG_ARM_32 and
> > instead depend on ARM_64 in Kconfig.
> > If we ever want to use this on ARM_32 we'll have to go through
> > everything anyway.
>
> Yes this is the best solution for now.
> And support.md patch is already saying already arm64.
>
> >
> >>
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 57 +++++++++++++++++++++++++++++++++++++++++-
> >>> 1 file changed, 56 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index a5d8a12635b6..07dd5c36d54b 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -148,6 +148,15 @@ struct ffa_ctx {
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> +/*
> >>> + * Our rx/tx buffers shared with the SPMC.
> >>> + *
> >>> + * ffa_page_count is the number of pages used in each of these buffers.
> >>> + */
> >>> +static void *ffa_rx __read_mostly;
> >>> +static void *ffa_tx __read_mostly;
> >>> +static unsigned int ffa_page_count __read_mostly;
> >>> +
> >>> static bool ffa_get_version(uint32_t *vers)
> >>> {
> >>>    const struct arm_smccc_1_2_regs arg = {
> >>> @@ -217,6 +226,17 @@ static bool check_mandatory_feature(uint32_t id)
> >>>    return !ret;
> >>> }
> >>>
> >>> +static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
> >>> +                            uint32_t page_count)
> >>
> >> Using register_t type here is doing an implicit cast when called and on
> >> 32bit this might later remove part of the address.
> >> This function must take paddr_t as parameters.
> >
> > I'll change to paddr_t for rx/tx.
> >
> >>
> >>> +{
> >>> +    uint32_t fid = FFA_RXTX_MAP_32;
> >>> +
> >>> +    if ( IS_ENABLED(CONFIG_ARM_64) )
> >>> +        fid = FFA_RXTX_MAP_64;
> >>> +
> >>> +    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
> >>
> >> simple call might not be suitable on 32bits due to the conversion.
> >> As said earlier, it would make more sense to disable FFA on 32bit and
> >> put some comments/build_bug_on in the code in places where there
> >> would be something to fix.
> >
> > I'm dropping the 32-bit support.
> >
> >>
> >>> +}
> >>> +
> >>> static uint16_t get_vm_id(const struct domain *d)
> >>> {
> >>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
> >>> @@ -384,6 +404,7 @@ static int ffa_relinquish_resources(struct domain *d)
> >>> static bool ffa_probe(void)
> >>> {
> >>>    uint32_t vers;
> >>> +    int e;
> >>>    unsigned int major_vers;
> >>>    unsigned int minor_vers;
> >>>
> >>> @@ -426,12 +447,46 @@ static bool ffa_probe(void)
> >>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> >>>           major_vers, minor_vers);
> >>>
> >>> -    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >>> +    if (
> >>> +#ifdef CONFIG_ARM_64
> >>> +         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> >>> +#endif
> >>> +#ifdef CONFIG_ARM_32
> >>> +         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> >>> +#endif
> >>> +         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> >>> +         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >>>        return false;
> >>>
> >>> +    ffa_rx = alloc_xenheap_pages(0, 0);
> >>> +    if ( !ffa_rx )
> >>> +        return false;
> >>> +
> >>> +    ffa_tx = alloc_xenheap_pages(0, 0);
> >>> +    if ( !ffa_tx )
> >>> +        goto err_free_ffa_rx;
> >>> +
> >>> +    e = ffa_rxtx_map(__pa(ffa_tx), __pa(ffa_rx), 1);
> >>> +    if ( e )
> >>> +    {
> >>> +        printk(XENLOG_ERR "ffa: Failed to map rxtx: error %d\n", e);
> >>> +        goto err_free_ffa_tx;
> >>> +    }
> >>> +    ffa_page_count = 1;
> >>
> >> ffa_page_count is a constant here and is not used to do the allocation or
> >> passed as parameter to ffa_rxtx_map.
> >>
> >> Do you expect this value to be modified ? how ?
> >
> > I expect this value to match how many FFA_PAGE_SIZE pages have been
> > mapped for the RX/TX buffers. Currently, this is only 1 but will have
> > to be changed later if PAGE_SIZE in Xen or in the secure world is
> > larger than FFA_PAGE_SIZE. We may also later add support
> > configurations where RX/TX buffers aren't mapped.
>
> So it is a constant and the buffers are just mapped or not mapped.

Correct

>
> >
> >>
> >> Please set it first and use it for allocation and as parameter to rxtx_map so
> >> that a modification of the value would only have to be done in one place.
> >>
> >> Please use a define if this is a constant.
> >
> > How about adding a define FFA_MIN_RXTX_PAGE_COUNT and giving that to
> > ffa_rxtx_map() and later assigning it to ffa_page_count if the call
> > succeeds?
>
> Why MIN ?

I was trying to prepare a bit for the future with a minimum value that
would be rounded up to a larger granule as needed depending on
PAGE_SIZE in Xen and what might be discovered about the secure world.

>
> How about just using ffa_rx or ffa_tx being NULL or not to check if the buffers are
> mapped and remove the count.

Sure, I'll drop the _MIN_ part of the define then.

>
> >
> >>
> >> As it is a global variable, does the parameter to rxtx_map make sense ?
> >
> > Yes, ffa_rxtx_map() is a dumb wrapper so it should have all the needed
> > parameters for the SMC provided.
>
> Then passing FFA_MIN_RXTX_PAGE_COUNT should be enough.

OK.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> >
> > Cheers,
> > Jens
> >
> >>
> >> Cheers
> >> Bertrand
> >>
> >>>    ffa_version = vers;
> >>>
> >>>    return true;
> >>> +
> >>> +err_free_ffa_tx:
> >>> +    free_xenheap_pages(ffa_tx, 0);
> >>> +    ffa_tx = NULL;
> >>> +err_free_ffa_rx:
> >>> +    free_xenheap_pages(ffa_rx, 0);
> >>> +    ffa_rx = NULL;
> >>> +    ffa_page_count = 0;
> >>> +    ffa_version = 0;
> >>> +
> >>> +    return false;
> >>> }
> >>>
> >>> static const struct tee_mediator_ops ffa_ops =
> >>> --
> >>> 2.34.1
>
>

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Bertrand Marquis]

Hi Jens,

> On 1 Mar 2023, at 11:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Tue, Feb 28, 2023 at 5:49 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> The FF-A specification defines framework messages sent as direct
>>> requests when certain events occurs. For instance when a VM (guest) is
>>> created or destroyed. Only SPs which have subscribed to these events
>>> will receive them. An SP can subscribe to these messages in its
>>> partition properties.
>>> 
>>> Adds a check that the SP supports the needed FF-A features
>>> FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
>>> 
>>> The partition properties of each SP is retrieved with
>>> FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
>>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
>>> caller (us), so once we're done with the buffer it must be released
>>> using FFA_RX_RELEASE before another call can be made.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
>>> 1 file changed, 190 insertions(+), 1 deletion(-)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 07dd5c36d54b..f1b014b6c7f4 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -140,6 +140,14 @@
>>> #define FFA_MSG_SEND                    0x8400006EU
>>> #define FFA_MSG_POLL                    0x8400006AU
>>> 
>>> +/* Partition information descriptor */
>>> +struct ffa_partition_info_1_1 {
>>> +    uint16_t id;
>>> +    uint16_t execution_context;
>>> +    uint32_t partition_properties;
>>> +    uint8_t uuid[16];
>>> +};
>>> +
>>> struct ffa_ctx {
>>>    uint32_t guest_vers;
>>>    bool interrupted;
>>> @@ -148,6 +156,12 @@ struct ffa_ctx {
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
>>> +static uint16_t *subscr_vm_created __read_mostly;
>>> +static unsigned int subscr_vm_created_count __read_mostly;
>> 
>> In the spec the number is returned in w2 so you should utse uint32_t here.
> 
> I don't understand. This value is increased for each SP which has the
> property set in the Partition information descriptor.

Using generic types should be prevented when possible.
Here this is a subset of the number of partition which is uint32_t (wX reg) so
i think this would be the logical type for this.

> 
>> 
>>> +static uint16_t *subscr_vm_destroyed __read_mostly;
>>> +static unsigned int subscr_vm_destroyed_count __read_mostly;
>> 
>> Same here
>> 
>>> +
>>> /*
>>> * Our rx/tx buffers shared with the SPMC.
>>> *
>>> @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
>>>    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
>>> }
>>> 
>>> +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
>>> +                                      uint32_t w4, uint32_t w5,
>>> +                                      uint32_t *count)
>>> +{
>>> +    const struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = FFA_PARTITION_INFO_GET,
>>> +        .a1 = w1,
>>> +        .a2 = w2,
>>> +        .a3 = w3,
>>> +        .a4 = w4,
>>> +        .a5 = w5,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +    uint32_t ret;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    ret = get_ffa_ret_code(&resp);
>>> +    if ( !ret )
>>> +        *count = resp.a2;
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +static int32_t ffa_rx_release(void)
>>> +{
>>> +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
>>> +}
>>> +
>>> +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>>> +                                      uint8_t msg)
>> 
>> This function is actually only useable to send framework created/destroyed
>> messages so the function name should be adapted to be less generic.
>> 
>> ffa_send_vm_events ?
>> 
>> unless you want to modify it later to send more framework messages ?
> 
> That was the plan, but that may never happen. I'll rename it to
> ffa_send_vm_event() since we're only sending one event at a time.
> 
>> 
>>> +{
>>> +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
>>> +    int32_t res;
>>> +
>>> +    if ( msg == FFA_MSG_SEND_VM_CREATED )
>>> +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
>>> +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
>>> +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
>>> +    else
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    do {
>>> +        const struct arm_smccc_1_2_regs arg = {
>>> +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
>>> +            .a1 = sp_id,
>>> +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
>>> +            .a5 = vm_id,
>>> +        };
>>> +        struct arm_smccc_1_2_regs resp;
>>> +
>>> +        arm_smccc_1_2_smc(&arg, &resp);
>>> +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
>>> +        {
>>> +            /*
>>> +             * This is an invalid response, likely due to some error in the
>>> +             * implementation of the ABI.
>>> +             */
>>> +            return FFA_RET_INVALID_PARAMETERS;
>>> +        }
>>> +        res = resp.a3;
>>> +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
>> 
>> We might end up in an infinite loop here or increase interrupt response time.
>> In the general case we could return that code directly to the VM but here we
>> are in the VM creation/destroy path so we cannot do that.
>> 
>> Maybe in debug mode at least we should have a retry counter here for now
>> with a print ?
> 
> OK, I'll add something.
> 
>> 
>>> +
>>> +    return res;
>>> +}
>>> +
>>> static uint16_t get_vm_id(const struct domain *d)
>>> {
>>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
>>> @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> static int ffa_domain_init(struct domain *d)
>>> {
>>>    struct ffa_ctx *ctx;
>>> +    unsigned int n;
>>> +    unsigned int m;
>>> +    unsigned int c_pos;
>>> +    int32_t res;
>>> 
>>>     /*
>>>      * We can't use that last possible domain ID or get_vm_id() would cause
>>> @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
>>>    if ( !ctx )
>>>        return -ENOMEM;
>>> 
>>> +    for ( n = 0; n < subscr_vm_created_count; n++ )
>>> +    {
>>> +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
>>> +                                     FFA_MSG_SEND_VM_CREATED);
>>> +        if ( res )
>>> +        {
>>> +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
>>> +                   get_vm_id(d), subscr_vm_created[n], res);
>> 
>> in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
>> instead of calling get_vm_id each time.
> 
> OK
> 
>> 
>>> +            c_pos = n;
>>> +            goto err;
>>> +        }
>>> +    }
>>> +
>>>    d->arch.tee = ctx;
>>> 
>>>    return 0;
>>> +
>>> +err:
>>> +    /* Undo any already sent vm created messaged */
>>> +    for ( n = 0; n < c_pos; n++ )
>>> +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
>>> +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
>>> +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>> +                                       FFA_MSG_SEND_VM_DESTROYED);
>>> +
>>> +    return -ENOMEM;
>> 
>> The VM creation is not failing due to missing memory here.
>> We need to find a better error code.
>> Maybe ENOTCONN ?
>> I am open to ideas here :-)
> 
> That makes sense, I'll change it to ENOTCONN.
> 
>> 
>>> }
>>> 
>>> /* This function is supposed to undo what ffa_domain_init() has done */
>>> static int ffa_relinquish_resources(struct domain *d)
>>> {
>>>    struct ffa_ctx *ctx = d->arch.tee;
>>> +    unsigned int n;
>>> +    int32_t res;
>>> 
>>>    if ( !ctx )
>>>        return 0;
>>> 
>>> +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
>>> +    {
>>> +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>> +                                     FFA_MSG_SEND_VM_DESTROYED);
>>> +
>>> +        if ( res )
>>> +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
>>> +                   get_vm_id(d), subscr_vm_destroyed[n], res);
>>> +    }
>>> +
>>>    XFREE(d->arch.tee);
>>> 
>>>    return 0;
>>> }
>>> 
>>> +static bool init_subscribers(void)
>>> +{
>>> +    struct ffa_partition_info_1_1 *fpi;
>>> +    bool ret = false;
>>> +    uint32_t count;
>>> +    int e;
>>> +    uint32_t n;
>>> +    uint32_t c_pos;
>>> +    uint32_t d_pos;
>>> +
>>> +    if ( ffa_version < FFA_VERSION_1_1 )
>>> +        return true;
>> 
>> Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
>> we could do the slow path and do a first loop on info_get until we get an error ?
> 
> Sending the events is not supported in 1.0 so there's nothing to
> record in that case.

Please add a comment here to say that subscribers are only supported after 1.1
and also mention it in the commit message.

Cheers
Bertrand

> 
> Thanks,
> Jens
> 
>> 
>>> +
>>> +    e = ffa_partition_info_get(0, 0, 0, 0, 0, &count);
>>> +    if ( e )
>>> +    {
>>> +        printk(XENLOG_ERR "ffa: Failed to get list of SPs: %d\n", e);
>>> +        goto out;
>>> +    }
>>> +
>>> +    fpi = ffa_rx;
>>> +    subscr_vm_created_count = 0;
>>> +    subscr_vm_destroyed_count = 0;
>>> +    for ( n = 0; n < count; n++ )
>>> +    {
>>> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED)
>>> +            subscr_vm_created_count++;
>>> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED)
>>> +            subscr_vm_destroyed_count++;
>>> +    }
>>> +
>>> +    if ( subscr_vm_created_count )
>>> +        subscr_vm_created = xzalloc_array(uint16_t, subscr_vm_created_count);
>>> +    if ( subscr_vm_destroyed_count )
>>> +        subscr_vm_destroyed = xzalloc_array(uint16_t,
>>> +                                            subscr_vm_destroyed_count);
>>> +    if ( (subscr_vm_created_count && !subscr_vm_created) ||
>>> +         (subscr_vm_destroyed_count && !subscr_vm_destroyed) )
>>> +    {
>>> +        printk(XENLOG_ERR "ffa: Failed to allocate subscription lists\n");
>>> +        subscr_vm_created_count = 0;
>>> +        subscr_vm_destroyed_count = 0;
>>> +        XFREE(subscr_vm_created);
>>> +        XFREE(subscr_vm_destroyed);
>>> +        goto out;
>>> +    }
>>> +
>>> +    for ( c_pos = 0, d_pos = 0, n = 0; n < count; n++ )
>>> +    {
>>> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED )
>>> +            subscr_vm_created[c_pos++] = fpi[n].id;
>>> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED )
>>> +            subscr_vm_destroyed[d_pos++] = fpi[n].id;
>>> +    }
>>> +
>>> +    ret = true;
>>> +out:
>>> +    ffa_rx_release();
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> static bool ffa_probe(void)
>>> {
>>>    uint32_t vers;
>>> @@ -447,7 +628,8 @@ static bool ffa_probe(void)
>>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>>>           major_vers, minor_vers);
>>> 
>>> -    if (
>>> +    if ( !check_mandatory_feature(FFA_PARTITION_INFO_GET) ||
>>> +         !check_mandatory_feature(FFA_RX_RELEASE) ||
>>> #ifdef CONFIG_ARM_64
>>>         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
>>> #endif
>>> @@ -475,6 +657,9 @@ static bool ffa_probe(void)
>>>    ffa_page_count = 1;
>>>    ffa_version = vers;
>>> 
>>> +    if ( !init_subscribers() )
>>> +        goto err_free_ffa_tx;
>>> +
>>>    return true;
>>> 
>>> err_free_ffa_tx:
>>> @@ -485,6 +670,10 @@ err_free_ffa_rx:
>>>    ffa_rx = NULL;
>>>    ffa_page_count = 0;
>>>    ffa_version = 0;
>>> +    XFREE(subscr_vm_created);
>>> +    subscr_vm_created_count = 0;
>>> +    XFREE(subscr_vm_destroyed);
>>> +    subscr_vm_destroyed_count = 0;
>>> 
>>>    return false;
>>> }
>>> --
>>> 2.34.1
>>> 
>> 
>> Cheers
>> Bertrand

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Bertrand Marquis]

HI Jens,

> On 1 Mar 2023, at 11:55, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Mon, Feb 27, 2023 at 4:28 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support for sending a FF-A direct request. Checks that the SP also
>>> supports handling a 32-bit direct request. 64-bit direct requests are
>>> not used by the mediator itself so there is not need to check for that.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 119 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 463fd7730573..a5d8a12635b6 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -142,6 +142,7 @@
>>> 
>>> struct ffa_ctx {
>>>    uint32_t guest_vers;
>>> +    bool interrupted;
>> 
>> This is added and set here for one special error code but is never used.
>> I would suggest to introduce this when there will be an action based on it.
> 
> I'm sorry, I forgot about completing this. I'll add code to deal with
> FFA_INTERRUPT. This will be tricky to test though since we don't use
> FFA_INTERRUPT like this with OP-TEE. The Hypervisor is required by the
> FF-A standard to support it so I better add something.

You can do that in a different patch then and just remove this from this patch ?

> 
>> 
>>> };
>>> 
>>> /* Negotiated FF-A version to use with the SPMC */
>>> @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
>>>    return true;
>>> }
>>> 
>>> +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
>>> +{
>>> +    switch ( resp->a0 )
>>> +    {
>>> +    case FFA_ERROR:
>>> +        if ( resp->a2 )
>>> +            return resp->a2;
>>> +        else
>>> +            return FFA_RET_NOT_SUPPORTED;
>>> +    case FFA_SUCCESS_32:
>>> +    case FFA_SUCCESS_64:
>>> +        return FFA_RET_OK;
>>> +    default:
>>> +        return FFA_RET_NOT_SUPPORTED;
>>> +    }
>>> +}
>>> +
>>> +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
>>> +                               register_t a3, register_t a4)
>>> +{
>>> +    const struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = fid,
>>> +        .a1 = a1,
>>> +        .a2 = a2,
>>> +        .a3 = a3,
>>> +        .a4 = a4,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    return get_ffa_ret_code(&resp);
>>> +}
>>> +
>>> +static int32_t ffa_features(uint32_t id)
>>> +{
>>> +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
>>> +}
>>> +
>>> +static bool check_mandatory_feature(uint32_t id)
>>> +{
>>> +    uint32_t ret = ffa_features(id);
>>> +
>>> +    if (ret)
>>> +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
>> 
>> It might be useful here to actually print the error code.
>> Are we sure that all errors actually mean not supported ?
> 
> Yes, that's what the standard says.

The error code might still be useful in the print.

> 
>> 
>>> +
>>> +    return !ret;
>>> +}
>>> +
>>> static uint16_t get_vm_id(const struct domain *d)
>>> {
>>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
>>> @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
>>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
>>> }
>>> 
>>> +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
>>> +{
>>> +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
>>> +    struct arm_smccc_1_2_regs resp = { };
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +    uint32_t src_dst;
>>> +    uint64_t mask;
>>> +
>>> +    if ( smccc_is_conv_64(fid) )
>>> +        mask = GENMASK_ULL(63, 0);
>>> +    else
>>> +        mask = GENMASK_ULL(31, 0);
>>> +
>>> +    src_dst = get_user_reg(regs, 1);
>>> +    if ( (src_dst >> 16) != get_vm_id(d) )
>>> +    {
>>> +        resp.a0 = FFA_ERROR;
>>> +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +
>>> +    arg.a1 = src_dst;
>>> +    arg.a2 = get_user_reg(regs, 2) & mask;
>>> +    arg.a3 = get_user_reg(regs, 3) & mask;
>>> +    arg.a4 = get_user_reg(regs, 4) & mask;
>>> +    arg.a5 = get_user_reg(regs, 5) & mask;
>>> +    arg.a6 = get_user_reg(regs, 6) & mask;
>>> +    arg.a7 = get_user_reg(regs, 7) & mask;
>>> +
>>> +    while ( true )
>>> +    {
>>> +        arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +        switch ( resp.a0 )
>>> +        {
>>> +        case FFA_INTERRUPT:
>>> +            ctx->interrupted = true;
>>> +            goto out;
>>> +        case FFA_ERROR:
>>> +        case FFA_SUCCESS_32:
>>> +        case FFA_SUCCESS_64:
>>> +        case FFA_MSG_SEND_DIRECT_RESP_32:
>>> +        case FFA_MSG_SEND_DIRECT_RESP_64:
>>> +            goto out;
>>> +        default:
>>> +            /* Bad fid, report back. */
>>> +            memset(&arg, 0, sizeof(arg));
>>> +            arg.a0 = FFA_ERROR;
>>> +            arg.a1 = src_dst;
>>> +            arg.a2 = FFA_RET_NOT_SUPPORTED;
>>> +            continue;
>> 
>> There is a potential infinite loop here and i do not understand
>> why this needs to be done.
>> Here if something is returning a value that you do not understand
>> you send back an ERROR to it. I do not find in the spec where this
>> is supposed to be done.
>> Can you explain a bit here ?
> 
> This should normally not happen, but the SP/SPMC is responding with a
> request that we don't know what to do with. The standard doesn't say
> how to handle that as far as I understand. However, returning back to
> the VM at this point with an error may leave the SP/SPMC in a strange
> state. So I think it's better to report back to the SP/SPMC that the
> request isn't understood and hopefully it can at least return back
> with an error in a sane state.
> 
> I'll add something to the comment.

Please also make sure that the code is not looping infinitely on this.

> 
>> 
>>> +        }
>>> +    }
>>> +
>>> +out:
>>> +    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
>>> +             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
>>> +}
>>> +
>>> static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> {
>>>    uint32_t fid = get_user_reg(regs, 0);
>>> @@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>    case FFA_ID_GET:
>>>        set_regs_success(regs, get_vm_id(d), 0);
>>>        return true;
>>> +    case FFA_MSG_SEND_DIRECT_REQ_32:
>>> +#ifdef CONFIG_ARM_64
>>> +    case FFA_MSG_SEND_DIRECT_REQ_64:
>>> +#endif
>>> +        handle_msg_send_direct_req(regs, fid);
>>> +        return true;
>>> 
>>>    default:
>>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
>>> @@ -310,6 +426,9 @@ static bool ffa_probe(void)
>>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>>>           major_vers, minor_vers);
>>> 
>>> +    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>> +        return false;
>> 
>> One could not need this feature and here this will make everything unavailable instead.
>> Why not just reporting back the unsupported error to clients using unsupported interfaces ?
> 
> One could perhaps argue that this check should be moved to a later
> patch in this series. Perhaps there's some future configuration that
> might make sense without this feature, but for now, it doesn't make
> sense to initialize without it.

I am starting to wonder if we should not at boot scan for available features, save them
somewhere and then accept/reject calls depending on the supported features.

Maybe just add a TODO here so that we remember that this is something that could be
checked/modified one day. That would also give an insight if someone has such a usecase
one day.

Cheers
Bertrand

> 
> Thanks,
> Jens
> 
>> 
>> Cheers
>> Bertrand
>> 
>>> +
>>>    ffa_version = vers;
>>> 
>>>    return true;
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Bertrand Marquis]

Hi Jens,

> On 1 Mar 2023, at 11:55, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Mon, Feb 27, 2023 at 4:28 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support for sending a FF-A direct request. Checks that the SP also
>>> supports handling a 32-bit direct request. 64-bit direct requests are
>>> not used by the mediator itself so there is not need to check for that.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 119 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 463fd7730573..a5d8a12635b6 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -142,6 +142,7 @@
>>> 
>>> struct ffa_ctx {
>>>    uint32_t guest_vers;
>>> +    bool interrupted;
>> 
>> This is added and set here for one special error code but is never used.
>> I would suggest to introduce this when there will be an action based on it.
> 
> I'm sorry, I forgot about completing this. I'll add code to deal with
> FFA_INTERRUPT. This will be tricky to test though since we don't use
> FFA_INTERRUPT like this with OP-TEE. The Hypervisor is required by the
> FF-A standard to support it so I better add something.
> 
>> 
>>> };
>>> 
>>> /* Negotiated FF-A version to use with the SPMC */
>>> @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
>>>    return true;
>>> }
>>> 
>>> +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
>>> +{
>>> +    switch ( resp->a0 )
>>> +    {
>>> +    case FFA_ERROR:
>>> +        if ( resp->a2 )
>>> +            return resp->a2;
>>> +        else
>>> +            return FFA_RET_NOT_SUPPORTED;
>>> +    case FFA_SUCCESS_32:
>>> +    case FFA_SUCCESS_64:
>>> +        return FFA_RET_OK;
>>> +    default:
>>> +        return FFA_RET_NOT_SUPPORTED;
>>> +    }
>>> +}
>>> +
>>> +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
>>> +                               register_t a3, register_t a4)
>>> +{
>>> +    const struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = fid,
>>> +        .a1 = a1,
>>> +        .a2 = a2,
>>> +        .a3 = a3,
>>> +        .a4 = a4,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    return get_ffa_ret_code(&resp);
>>> +}
>>> +
>>> +static int32_t ffa_features(uint32_t id)
>>> +{
>>> +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
>>> +}
>>> +
>>> +static bool check_mandatory_feature(uint32_t id)
>>> +{
>>> +    uint32_t ret = ffa_features(id);
>>> +
>>> +    if (ret)
>>> +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
>> 
>> It might be useful here to actually print the error code.
>> Are we sure that all errors actually mean not supported ?
> 
> Yes, that's what the standard says.
> 
>> 
>>> +
>>> +    return !ret;
>>> +}
>>> +
>>> static uint16_t get_vm_id(const struct domain *d)
>>> {
>>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
>>> @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
>>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
>>> }
>>> 
>>> +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
>>> +{
>>> +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
>>> +    struct arm_smccc_1_2_regs resp = { };
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +    uint32_t src_dst;
>>> +    uint64_t mask;
>>> +
>>> +    if ( smccc_is_conv_64(fid) )
>>> +        mask = GENMASK_ULL(63, 0);
>>> +    else
>>> +        mask = GENMASK_ULL(31, 0);
>>> +
>>> +    src_dst = get_user_reg(regs, 1);
>>> +    if ( (src_dst >> 16) != get_vm_id(d) )
>>> +    {
>>> +        resp.a0 = FFA_ERROR;
>>> +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +
>>> +    arg.a1 = src_dst;
>>> +    arg.a2 = get_user_reg(regs, 2) & mask;
>>> +    arg.a3 = get_user_reg(regs, 3) & mask;
>>> +    arg.a4 = get_user_reg(regs, 4) & mask;
>>> +    arg.a5 = get_user_reg(regs, 5) & mask;
>>> +    arg.a6 = get_user_reg(regs, 6) & mask;
>>> +    arg.a7 = get_user_reg(regs, 7) & mask;
>>> +
>>> +    while ( true )
>>> +    {
>>> +        arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +        switch ( resp.a0 )
>>> +        {
>>> +        case FFA_INTERRUPT:
>>> +            ctx->interrupted = true;
>>> +            goto out;
>>> +        case FFA_ERROR:
>>> +        case FFA_SUCCESS_32:
>>> +        case FFA_SUCCESS_64:
>>> +        case FFA_MSG_SEND_DIRECT_RESP_32:
>>> +        case FFA_MSG_SEND_DIRECT_RESP_64:
>>> +            goto out;
>>> +        default:
>>> +            /* Bad fid, report back. */
>>> +            memset(&arg, 0, sizeof(arg));
>>> +            arg.a0 = FFA_ERROR;
>>> +            arg.a1 = src_dst;
>>> +            arg.a2 = FFA_RET_NOT_SUPPORTED;
>>> +            continue;
>> 
>> There is a potential infinite loop here and i do not understand
>> why this needs to be done.
>> Here if something is returning a value that you do not understand
>> you send back an ERROR to it. I do not find in the spec where this
>> is supposed to be done.
>> Can you explain a bit here ?
> 
> This should normally not happen, but the SP/SPMC is responding with a
> request that we don't know what to do with. The standard doesn't say
> how to handle that as far as I understand. However, returning back to
> the VM at this point with an error may leave the SP/SPMC in a strange
> state. So I think it's better to report back to the SP/SPMC that the
> request isn't understood and hopefully it can at least return back
> with an error in a sane state.
> 
> I'll add something to the comment.

I discussed that with Achin and Marc today at Arm and if we get an invalid
fid we do not need to report it back like you did.
We should instead report this as an error to the requester.

This is good as it will remove the :-)

Cheers
Bertrand

> 
>> 
>>> +        }
>>> +    }
>>> +
>>> +out:
>>> +    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
>>> +             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
>>> +}
>>> +
>>> static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> {
>>>    uint32_t fid = get_user_reg(regs, 0);
>>> @@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>    case FFA_ID_GET:
>>>        set_regs_success(regs, get_vm_id(d), 0);
>>>        return true;
>>> +    case FFA_MSG_SEND_DIRECT_REQ_32:
>>> +#ifdef CONFIG_ARM_64
>>> +    case FFA_MSG_SEND_DIRECT_REQ_64:
>>> +#endif
>>> +        handle_msg_send_direct_req(regs, fid);
>>> +        return true;
>>> 
>>>    default:
>>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
>>> @@ -310,6 +426,9 @@ static bool ffa_probe(void)
>>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>>>           major_vers, minor_vers);
>>> 
>>> +    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>> +        return false;
>> 
>> One could not need this feature and here this will make everything unavailable instead.
>> Why not just reporting back the unsupported error to clients using unsupported interfaces ?
> 
> One could perhaps argue that this check should be moved to a later
> patch in this series. Perhaps there's some future configuration that
> might make sense without this feature, but for now, it doesn't make
> sense to initialize without it.
> 
> Thanks,
> Jens
> 
>> 
>> Cheers
>> Bertrand
>> 
>>> +
>>>    ffa_version = vers;
>>> 
>>>    return true;
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Bertrand Marquis]

Hi Jens,

> On 1 Mar 2023, at 11:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Tue, Feb 28, 2023 at 5:49 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> The FF-A specification defines framework messages sent as direct
>>> requests when certain events occurs. For instance when a VM (guest) is
>>> created or destroyed. Only SPs which have subscribed to these events
>>> will receive them. An SP can subscribe to these messages in its
>>> partition properties.
>>> 
>>> Adds a check that the SP supports the needed FF-A features
>>> FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
>>> 
>>> The partition properties of each SP is retrieved with
>>> FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
>>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
>>> caller (us), so once we're done with the buffer it must be released
>>> using FFA_RX_RELEASE before another call can be made.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
>>> 1 file changed, 190 insertions(+), 1 deletion(-)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 07dd5c36d54b..f1b014b6c7f4 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -140,6 +140,14 @@
>>> #define FFA_MSG_SEND                    0x8400006EU
>>> #define FFA_MSG_POLL                    0x8400006AU
>>> 
>>> +/* Partition information descriptor */
>>> +struct ffa_partition_info_1_1 {
>>> +    uint16_t id;
>>> +    uint16_t execution_context;
>>> +    uint32_t partition_properties;
>>> +    uint8_t uuid[16];
>>> +};
>>> +
>>> struct ffa_ctx {
>>>    uint32_t guest_vers;
>>>    bool interrupted;
>>> @@ -148,6 +156,12 @@ struct ffa_ctx {
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
>>> +static uint16_t *subscr_vm_created __read_mostly;
>>> +static unsigned int subscr_vm_created_count __read_mostly;
>> 
>> In the spec the number is returned in w2 so you should utse uint32_t here.
> 
> I don't understand. This value is increased for each SP which has the
> property set in the Partition information descriptor.
> 
>> 
>>> +static uint16_t *subscr_vm_destroyed __read_mostly;
>>> +static unsigned int subscr_vm_destroyed_count __read_mostly;
>> 
>> Same here
>> 
>>> +
>>> /*
>>> * Our rx/tx buffers shared with the SPMC.
>>> *
>>> @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
>>>    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
>>> }
>>> 
>>> +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
>>> +                                      uint32_t w4, uint32_t w5,
>>> +                                      uint32_t *count)
>>> +{
>>> +    const struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = FFA_PARTITION_INFO_GET,
>>> +        .a1 = w1,
>>> +        .a2 = w2,
>>> +        .a3 = w3,
>>> +        .a4 = w4,
>>> +        .a5 = w5,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +    uint32_t ret;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    ret = get_ffa_ret_code(&resp);
>>> +    if ( !ret )
>>> +        *count = resp.a2;
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +static int32_t ffa_rx_release(void)
>>> +{
>>> +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
>>> +}
>>> +
>>> +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>>> +                                      uint8_t msg)
>> 
>> This function is actually only useable to send framework created/destroyed
>> messages so the function name should be adapted to be less generic.
>> 
>> ffa_send_vm_events ?
>> 
>> unless you want to modify it later to send more framework messages ?
> 
> That was the plan, but that may never happen. I'll rename it to
> ffa_send_vm_event() since we're only sending one event at a time.
> 
>> 
>>> +{
>>> +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
>>> +    int32_t res;
>>> +
>>> +    if ( msg == FFA_MSG_SEND_VM_CREATED )
>>> +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
>>> +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
>>> +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
>>> +    else
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    do {
>>> +        const struct arm_smccc_1_2_regs arg = {
>>> +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
>>> +            .a1 = sp_id,
>>> +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
>>> +            .a5 = vm_id,
>>> +        };
>>> +        struct arm_smccc_1_2_regs resp;
>>> +
>>> +        arm_smccc_1_2_smc(&arg, &resp);
>>> +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
>>> +        {
>>> +            /*
>>> +             * This is an invalid response, likely due to some error in the
>>> +             * implementation of the ABI.
>>> +             */
>>> +            return FFA_RET_INVALID_PARAMETERS;
>>> +        }
>>> +        res = resp.a3;
>>> +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
>> 
>> We might end up in an infinite loop here or increase interrupt response time.
>> In the general case we could return that code directly to the VM but here we
>> are in the VM creation/destroy path so we cannot do that.
>> 
>> Maybe in debug mode at least we should have a retry counter here for now
>> with a print ?
> 
> OK, I'll add something.

In fact here we should always count and fail if we do it more than once or twice.

In a normal case, we would return the INTERRUPTED or RETRY to the caller so
that he would do the call again.
Here this is something done by Xen internally so we cannot do that (we could make
the VM creation fail but destroying we cannot) so we must limit the number of retry
and continue if this fails.

Cheers
Bertrand

> 
>> 
>>> +
>>> +    return res;
>>> +}
>>> +
>>> static uint16_t get_vm_id(const struct domain *d)
>>> {
>>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
>>> @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> static int ffa_domain_init(struct domain *d)
>>> {
>>>    struct ffa_ctx *ctx;
>>> +    unsigned int n;
>>> +    unsigned int m;
>>> +    unsigned int c_pos;
>>> +    int32_t res;
>>> 
>>>     /*
>>>      * We can't use that last possible domain ID or get_vm_id() would cause
>>> @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
>>>    if ( !ctx )
>>>        return -ENOMEM;
>>> 
>>> +    for ( n = 0; n < subscr_vm_created_count; n++ )
>>> +    {
>>> +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
>>> +                                     FFA_MSG_SEND_VM_CREATED);
>>> +        if ( res )
>>> +        {
>>> +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
>>> +                   get_vm_id(d), subscr_vm_created[n], res);
>> 
>> in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
>> instead of calling get_vm_id each time.
> 
> OK
> 
>> 
>>> +            c_pos = n;
>>> +            goto err;
>>> +        }
>>> +    }
>>> +
>>>    d->arch.tee = ctx;
>>> 
>>>    return 0;
>>> +
>>> +err:
>>> +    /* Undo any already sent vm created messaged */
>>> +    for ( n = 0; n < c_pos; n++ )
>>> +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
>>> +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
>>> +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>> +                                       FFA_MSG_SEND_VM_DESTROYED);
>>> +
>>> +    return -ENOMEM;
>> 
>> The VM creation is not failing due to missing memory here.
>> We need to find a better error code.
>> Maybe ENOTCONN ?
>> I am open to ideas here :-)
> 
> That makes sense, I'll change it to ENOTCONN.
> 
>> 
>>> }
>>> 
>>> /* This function is supposed to undo what ffa_domain_init() has done */
>>> static int ffa_relinquish_resources(struct domain *d)
>>> {
>>>    struct ffa_ctx *ctx = d->arch.tee;
>>> +    unsigned int n;
>>> +    int32_t res;
>>> 
>>>    if ( !ctx )
>>>        return 0;
>>> 
>>> +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
>>> +    {
>>> +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>> +                                     FFA_MSG_SEND_VM_DESTROYED);
>>> +
>>> +        if ( res )
>>> +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
>>> +                   get_vm_id(d), subscr_vm_destroyed[n], res);
>>> +    }
>>> +
>>>    XFREE(d->arch.tee);
>>> 
>>>    return 0;
>>> }
>>> 
>>> +static bool init_subscribers(void)
>>> +{
>>> +    struct ffa_partition_info_1_1 *fpi;
>>> +    bool ret = false;
>>> +    uint32_t count;
>>> +    int e;
>>> +    uint32_t n;
>>> +    uint32_t c_pos;
>>> +    uint32_t d_pos;
>>> +
>>> +    if ( ffa_version < FFA_VERSION_1_1 )
>>> +        return true;
>> 
>> Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
>> we could do the slow path and do a first loop on info_get until we get an error ?
> 
> Sending the events is not supported in 1.0 so there's nothing to
> record in that case.
> 
> Thanks,
> Jens
> 
>> 
>>> +
>>> +    e = ffa_partition_info_get(0, 0, 0, 0, 0, &count);
>>> +    if ( e )
>>> +    {
>>> +        printk(XENLOG_ERR "ffa: Failed to get list of SPs: %d\n", e);
>>> +        goto out;
>>> +    }
>>> +
>>> +    fpi = ffa_rx;
>>> +    subscr_vm_created_count = 0;
>>> +    subscr_vm_destroyed_count = 0;
>>> +    for ( n = 0; n < count; n++ )
>>> +    {
>>> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED)
>>> +            subscr_vm_created_count++;
>>> +        if (fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED)
>>> +            subscr_vm_destroyed_count++;
>>> +    }
>>> +
>>> +    if ( subscr_vm_created_count )
>>> +        subscr_vm_created = xzalloc_array(uint16_t, subscr_vm_created_count);
>>> +    if ( subscr_vm_destroyed_count )
>>> +        subscr_vm_destroyed = xzalloc_array(uint16_t,
>>> +                                            subscr_vm_destroyed_count);
>>> +    if ( (subscr_vm_created_count && !subscr_vm_created) ||
>>> +         (subscr_vm_destroyed_count && !subscr_vm_destroyed) )
>>> +    {
>>> +        printk(XENLOG_ERR "ffa: Failed to allocate subscription lists\n");
>>> +        subscr_vm_created_count = 0;
>>> +        subscr_vm_destroyed_count = 0;
>>> +        XFREE(subscr_vm_created);
>>> +        XFREE(subscr_vm_destroyed);
>>> +        goto out;
>>> +    }
>>> +
>>> +    for ( c_pos = 0, d_pos = 0, n = 0; n < count; n++ )
>>> +    {
>>> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_CREATED )
>>> +            subscr_vm_created[c_pos++] = fpi[n].id;
>>> +        if ( fpi[n].partition_properties & FFA_PART_PROP_NOTIF_DESTROYED )
>>> +            subscr_vm_destroyed[d_pos++] = fpi[n].id;
>>> +    }
>>> +
>>> +    ret = true;
>>> +out:
>>> +    ffa_rx_release();
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> static bool ffa_probe(void)
>>> {
>>>    uint32_t vers;
>>> @@ -447,7 +628,8 @@ static bool ffa_probe(void)
>>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
>>>           major_vers, minor_vers);
>>> 
>>> -    if (
>>> +    if ( !check_mandatory_feature(FFA_PARTITION_INFO_GET) ||
>>> +         !check_mandatory_feature(FFA_RX_RELEASE) ||
>>> #ifdef CONFIG_ARM_64
>>>         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
>>> #endif
>>> @@ -475,6 +657,9 @@ static bool ffa_probe(void)
>>>    ffa_page_count = 1;
>>>    ffa_version = vers;
>>> 
>>> +    if ( !init_subscribers() )
>>> +        goto err_free_ffa_tx;
>>> +
>>>    return true;
>>> 
>>> err_free_ffa_tx:
>>> @@ -485,6 +670,10 @@ err_free_ffa_rx:
>>>    ffa_rx = NULL;
>>>    ffa_page_count = 0;
>>>    ffa_version = 0;
>>> +    XFREE(subscr_vm_created);
>>> +    subscr_vm_created_count = 0;
>>> +    XFREE(subscr_vm_destroyed);
>>> +    subscr_vm_destroyed_count = 0;
>>> 
>>>    return false;
>>> }
>>> --
>>> 2.34.1
>>> 
>> 
>> Cheers
>> Bertrand

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Jens Wiklander]

On Wed, Mar 1, 2023 at 4:50 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 1 Mar 2023, at 11:55, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Mon, Feb 27, 2023 at 4:28 PM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support for sending a FF-A direct request. Checks that the SP also
> >>> supports handling a 32-bit direct request. 64-bit direct requests are
> >>> not used by the mediator itself so there is not need to check for that.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
> >>> 1 file changed, 119 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 463fd7730573..a5d8a12635b6 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -142,6 +142,7 @@
> >>>
> >>> struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>> +    bool interrupted;
> >>
> >> This is added and set here for one special error code but is never used.
> >> I would suggest to introduce this when there will be an action based on it.
> >
> > I'm sorry, I forgot about completing this. I'll add code to deal with
> > FFA_INTERRUPT. This will be tricky to test though since we don't use
> > FFA_INTERRUPT like this with OP-TEE. The Hypervisor is required by the
> > FF-A standard to support it so I better add something.
> >
> >>
> >>> };
> >>>
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
> >>>    return true;
> >>> }
> >>>
> >>> +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
> >>> +{
> >>> +    switch ( resp->a0 )
> >>> +    {
> >>> +    case FFA_ERROR:
> >>> +        if ( resp->a2 )
> >>> +            return resp->a2;
> >>> +        else
> >>> +            return FFA_RET_NOT_SUPPORTED;
> >>> +    case FFA_SUCCESS_32:
> >>> +    case FFA_SUCCESS_64:
> >>> +        return FFA_RET_OK;
> >>> +    default:
> >>> +        return FFA_RET_NOT_SUPPORTED;
> >>> +    }
> >>> +}
> >>> +
> >>> +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
> >>> +                               register_t a3, register_t a4)
> >>> +{
> >>> +    const struct arm_smccc_1_2_regs arg = {
> >>> +        .a0 = fid,
> >>> +        .a1 = a1,
> >>> +        .a2 = a2,
> >>> +        .a3 = a3,
> >>> +        .a4 = a4,
> >>> +    };
> >>> +    struct arm_smccc_1_2_regs resp;
> >>> +
> >>> +    arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +    return get_ffa_ret_code(&resp);
> >>> +}
> >>> +
> >>> +static int32_t ffa_features(uint32_t id)
> >>> +{
> >>> +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
> >>> +}
> >>> +
> >>> +static bool check_mandatory_feature(uint32_t id)
> >>> +{
> >>> +    uint32_t ret = ffa_features(id);
> >>> +
> >>> +    if (ret)
> >>> +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
> >>
> >> It might be useful here to actually print the error code.
> >> Are we sure that all errors actually mean not supported ?
> >
> > Yes, that's what the standard says.
> >
> >>
> >>> +
> >>> +    return !ret;
> >>> +}
> >>> +
> >>> static uint16_t get_vm_id(const struct domain *d)
> >>> {
> >>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
> >>> @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
> >>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> >>> }
> >>>
> >>> +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> >>> +{
> >>> +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> >>> +    struct arm_smccc_1_2_regs resp = { };
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    uint32_t src_dst;
> >>> +    uint64_t mask;
> >>> +
> >>> +    if ( smccc_is_conv_64(fid) )
> >>> +        mask = GENMASK_ULL(63, 0);
> >>> +    else
> >>> +        mask = GENMASK_ULL(31, 0);
> >>> +
> >>> +    src_dst = get_user_reg(regs, 1);
> >>> +    if ( (src_dst >> 16) != get_vm_id(d) )
> >>> +    {
> >>> +        resp.a0 = FFA_ERROR;
> >>> +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +
> >>> +    arg.a1 = src_dst;
> >>> +    arg.a2 = get_user_reg(regs, 2) & mask;
> >>> +    arg.a3 = get_user_reg(regs, 3) & mask;
> >>> +    arg.a4 = get_user_reg(regs, 4) & mask;
> >>> +    arg.a5 = get_user_reg(regs, 5) & mask;
> >>> +    arg.a6 = get_user_reg(regs, 6) & mask;
> >>> +    arg.a7 = get_user_reg(regs, 7) & mask;
> >>> +
> >>> +    while ( true )
> >>> +    {
> >>> +        arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +        switch ( resp.a0 )
> >>> +        {
> >>> +        case FFA_INTERRUPT:
> >>> +            ctx->interrupted = true;
> >>> +            goto out;
> >>> +        case FFA_ERROR:
> >>> +        case FFA_SUCCESS_32:
> >>> +        case FFA_SUCCESS_64:
> >>> +        case FFA_MSG_SEND_DIRECT_RESP_32:
> >>> +        case FFA_MSG_SEND_DIRECT_RESP_64:
> >>> +            goto out;
> >>> +        default:
> >>> +            /* Bad fid, report back. */
> >>> +            memset(&arg, 0, sizeof(arg));
> >>> +            arg.a0 = FFA_ERROR;
> >>> +            arg.a1 = src_dst;
> >>> +            arg.a2 = FFA_RET_NOT_SUPPORTED;
> >>> +            continue;
> >>
> >> There is a potential infinite loop here and i do not understand
> >> why this needs to be done.
> >> Here if something is returning a value that you do not understand
> >> you send back an ERROR to it. I do not find in the spec where this
> >> is supposed to be done.
> >> Can you explain a bit here ?
> >
> > This should normally not happen, but the SP/SPMC is responding with a
> > request that we don't know what to do with. The standard doesn't say
> > how to handle that as far as I understand. However, returning back to
> > the VM at this point with an error may leave the SP/SPMC in a strange
> > state. So I think it's better to report back to the SP/SPMC that the
> > request isn't understood and hopefully it can at least return back
> > with an error in a sane state.
> >
> > I'll add something to the comment.
>
> I discussed that with Achin and Marc today at Arm and if we get an invalid
> fid we do not need to report it back like you did.
> We should instead report this as an error to the requester.
>
> This is good as it will remove the :-)

Great, thanks.
/Jens

Re: [XEN PATCH v7 10/20] xen/arm: ffa: add direct request support

[Jens Wiklander]

Hi,

On Wed, Mar 1, 2023 at 2:06 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> HI Jens,
>
> > On 1 Mar 2023, at 11:55, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Mon, Feb 27, 2023 at 4:28 PM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support for sending a FF-A direct request. Checks that the SP also
> >>> supports handling a 32-bit direct request. 64-bit direct requests are
> >>> not used by the mediator itself so there is not need to check for that.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 119 +++++++++++++++++++++++++++++++++++++++++
> >>> 1 file changed, 119 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 463fd7730573..a5d8a12635b6 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -142,6 +142,7 @@
> >>>
> >>> struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>> +    bool interrupted;
> >>
> >> This is added and set here for one special error code but is never used.
> >> I would suggest to introduce this when there will be an action based on it.
> >
> > I'm sorry, I forgot about completing this. I'll add code to deal with
> > FFA_INTERRUPT. This will be tricky to test though since we don't use
> > FFA_INTERRUPT like this with OP-TEE. The Hypervisor is required by the
> > FF-A standard to support it so I better add something.
>
> You can do that in a different patch then and just remove this from this patch ?

OK, I'll do that.

>
> >
> >>
> >>> };
> >>>
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> @@ -167,6 +168,55 @@ static bool ffa_get_version(uint32_t *vers)
> >>>    return true;
> >>> }
> >>>
> >>> +static int32_t get_ffa_ret_code(const struct arm_smccc_1_2_regs *resp)
> >>> +{
> >>> +    switch ( resp->a0 )
> >>> +    {
> >>> +    case FFA_ERROR:
> >>> +        if ( resp->a2 )
> >>> +            return resp->a2;
> >>> +        else
> >>> +            return FFA_RET_NOT_SUPPORTED;
> >>> +    case FFA_SUCCESS_32:
> >>> +    case FFA_SUCCESS_64:
> >>> +        return FFA_RET_OK;
> >>> +    default:
> >>> +        return FFA_RET_NOT_SUPPORTED;
> >>> +    }
> >>> +}
> >>> +
> >>> +static int32_t ffa_simple_call(uint32_t fid, register_t a1, register_t a2,
> >>> +                               register_t a3, register_t a4)
> >>> +{
> >>> +    const struct arm_smccc_1_2_regs arg = {
> >>> +        .a0 = fid,
> >>> +        .a1 = a1,
> >>> +        .a2 = a2,
> >>> +        .a3 = a3,
> >>> +        .a4 = a4,
> >>> +    };
> >>> +    struct arm_smccc_1_2_regs resp;
> >>> +
> >>> +    arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +    return get_ffa_ret_code(&resp);
> >>> +}
> >>> +
> >>> +static int32_t ffa_features(uint32_t id)
> >>> +{
> >>> +    return ffa_simple_call(FFA_FEATURES, id, 0, 0, 0);
> >>> +}
> >>> +
> >>> +static bool check_mandatory_feature(uint32_t id)
> >>> +{
> >>> +    uint32_t ret = ffa_features(id);
> >>> +
> >>> +    if (ret)
> >>> +        printk(XENLOG_ERR "ffa: mandatory feature id %#x missing\n", id);
> >>
> >> It might be useful here to actually print the error code.
> >> Are we sure that all errors actually mean not supported ?
> >
> > Yes, that's what the standard says.
>
> The error code might still be useful in the print.

OK, I'll add it.

>
> >
> >>
> >>> +
> >>> +    return !ret;
> >>> +}
> >>> +
> >>> static uint16_t get_vm_id(const struct domain *d)
> >>> {
> >>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
> >>> @@ -208,6 +258,66 @@ static void handle_version(struct cpu_user_regs *regs)
> >>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> >>> }
> >>>
> >>> +static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> >>> +{
> >>> +    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> >>> +    struct arm_smccc_1_2_regs resp = { };
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    uint32_t src_dst;
> >>> +    uint64_t mask;
> >>> +
> >>> +    if ( smccc_is_conv_64(fid) )
> >>> +        mask = GENMASK_ULL(63, 0);
> >>> +    else
> >>> +        mask = GENMASK_ULL(31, 0);
> >>> +
> >>> +    src_dst = get_user_reg(regs, 1);
> >>> +    if ( (src_dst >> 16) != get_vm_id(d) )
> >>> +    {
> >>> +        resp.a0 = FFA_ERROR;
> >>> +        resp.a2 = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +
> >>> +    arg.a1 = src_dst;
> >>> +    arg.a2 = get_user_reg(regs, 2) & mask;
> >>> +    arg.a3 = get_user_reg(regs, 3) & mask;
> >>> +    arg.a4 = get_user_reg(regs, 4) & mask;
> >>> +    arg.a5 = get_user_reg(regs, 5) & mask;
> >>> +    arg.a6 = get_user_reg(regs, 6) & mask;
> >>> +    arg.a7 = get_user_reg(regs, 7) & mask;
> >>> +
> >>> +    while ( true )
> >>> +    {
> >>> +        arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +        switch ( resp.a0 )
> >>> +        {
> >>> +        case FFA_INTERRUPT:
> >>> +            ctx->interrupted = true;
> >>> +            goto out;
> >>> +        case FFA_ERROR:
> >>> +        case FFA_SUCCESS_32:
> >>> +        case FFA_SUCCESS_64:
> >>> +        case FFA_MSG_SEND_DIRECT_RESP_32:
> >>> +        case FFA_MSG_SEND_DIRECT_RESP_64:
> >>> +            goto out;
> >>> +        default:
> >>> +            /* Bad fid, report back. */
> >>> +            memset(&arg, 0, sizeof(arg));
> >>> +            arg.a0 = FFA_ERROR;
> >>> +            arg.a1 = src_dst;
> >>> +            arg.a2 = FFA_RET_NOT_SUPPORTED;
> >>> +            continue;
> >>
> >> There is a potential infinite loop here and i do not understand
> >> why this needs to be done.
> >> Here if something is returning a value that you do not understand
> >> you send back an ERROR to it. I do not find in the spec where this
> >> is supposed to be done.
> >> Can you explain a bit here ?
> >
> > This should normally not happen, but the SP/SPMC is responding with a
> > request that we don't know what to do with. The standard doesn't say
> > how to handle that as far as I understand. However, returning back to
> > the VM at this point with an error may leave the SP/SPMC in a strange
> > state. So I think it's better to report back to the SP/SPMC that the
> > request isn't understood and hopefully it can at least return back
> > with an error in a sane state.
> >
> > I'll add something to the comment.
>
> Please also make sure that the code is not looping infinitely on this.

I'll remove the loop as agreed in the other mail.

>
> >
> >>
> >>> +        }
> >>> +    }
> >>> +
> >>> +out:
> >>> +    set_regs(regs, resp.a0, resp.a1 & mask, resp.a2 & mask, resp.a3 & mask,
> >>> +             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> >>> +}
> >>> +
> >>> static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>> {
> >>>    uint32_t fid = get_user_reg(regs, 0);
> >>> @@ -225,6 +335,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>    case FFA_ID_GET:
> >>>        set_regs_success(regs, get_vm_id(d), 0);
> >>>        return true;
> >>> +    case FFA_MSG_SEND_DIRECT_REQ_32:
> >>> +#ifdef CONFIG_ARM_64
> >>> +    case FFA_MSG_SEND_DIRECT_REQ_64:
> >>> +#endif
> >>> +        handle_msg_send_direct_req(regs, fid);
> >>> +        return true;
> >>>
> >>>    default:
> >>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> >>> @@ -310,6 +426,9 @@ static bool ffa_probe(void)
> >>>    printk(XENLOG_INFO "ARM FF-A Firmware version %u.%u\n",
> >>>           major_vers, minor_vers);
> >>>
> >>> +    if ( !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >>> +        return false;
> >>
> >> One could not need this feature and here this will make everything unavailable instead.
> >> Why not just reporting back the unsupported error to clients using unsupported interfaces ?
> >
> > One could perhaps argue that this check should be moved to a later
> > patch in this series. Perhaps there's some future configuration that
> > might make sense without this feature, but for now, it doesn't make
> > sense to initialize without it.
>
> I am starting to wonder if we should not at boot scan for available features, save them
> somewhere and then accept/reject calls depending on the supported features.
>
> Maybe just add a TODO here so that we remember that this is something that could be
> checked/modified one day. That would also give an insight if someone has such a usecase
> one day.

I'll add a comment.

Cheers,
Jens


>
> Cheers
> Bertrand
>
> >
> > Thanks,
> > Jens
> >
> >>
> >> Cheers
> >> Bertrand
> >>
> >>> +
> >>>    ffa_version = vers;
> >>>
> >>>    return true;
> >>> --
> >>> 2.34.1
>
>

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Jens Wiklander]

Hi,

On Wed, Mar 1, 2023 at 1:58 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 1 Mar 2023, at 11:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Tue, Feb 28, 2023 at 5:49 PM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> The FF-A specification defines framework messages sent as direct
> >>> requests when certain events occurs. For instance when a VM (guest) is
> >>> created or destroyed. Only SPs which have subscribed to these events
> >>> will receive them. An SP can subscribe to these messages in its
> >>> partition properties.
> >>>
> >>> Adds a check that the SP supports the needed FF-A features
> >>> FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
> >>>
> >>> The partition properties of each SP is retrieved with
> >>> FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
> >>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> >>> caller (us), so once we're done with the buffer it must be released
> >>> using FFA_RX_RELEASE before another call can be made.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
> >>> 1 file changed, 190 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 07dd5c36d54b..f1b014b6c7f4 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -140,6 +140,14 @@
> >>> #define FFA_MSG_SEND                    0x8400006EU
> >>> #define FFA_MSG_POLL                    0x8400006AU
> >>>
> >>> +/* Partition information descriptor */
> >>> +struct ffa_partition_info_1_1 {
> >>> +    uint16_t id;
> >>> +    uint16_t execution_context;
> >>> +    uint32_t partition_properties;
> >>> +    uint8_t uuid[16];
> >>> +};
> >>> +
> >>> struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>>    bool interrupted;
> >>> @@ -148,6 +156,12 @@ struct ffa_ctx {
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
> >>> +static uint16_t *subscr_vm_created __read_mostly;
> >>> +static unsigned int subscr_vm_created_count __read_mostly;
> >>
> >> In the spec the number is returned in w2 so you should utse uint32_t here.
> >
> > I don't understand. This value is increased for each SP which has the
> > property set in the Partition information descriptor.
>
> Using generic types should be prevented when possible.

I'm usually of the opposite opinion, fixed width integers should only
be used when there's a good reason to do so. However, if this is the
Xen philosophy I can replace all those unsigned int with uint32_t if
that's preferable.

> Here this is a subset of the number of partition which is uint32_t (wX reg) so
> i think this would be the logical type for this.

The maximum number is actually UINT16_MAX so an observant reader might
wonder why the uint32_t type was used here.

>
> >
> >>
> >>> +static uint16_t *subscr_vm_destroyed __read_mostly;
> >>> +static unsigned int subscr_vm_destroyed_count __read_mostly;
> >>
> >> Same here
> >>
> >>> +
> >>> /*
> >>> * Our rx/tx buffers shared with the SPMC.
> >>> *
> >>> @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
> >>>    return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
> >>> }
> >>>
> >>> +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> >>> +                                      uint32_t w4, uint32_t w5,
> >>> +                                      uint32_t *count)
> >>> +{
> >>> +    const struct arm_smccc_1_2_regs arg = {
> >>> +        .a0 = FFA_PARTITION_INFO_GET,
> >>> +        .a1 = w1,
> >>> +        .a2 = w2,
> >>> +        .a3 = w3,
> >>> +        .a4 = w4,
> >>> +        .a5 = w5,
> >>> +    };
> >>> +    struct arm_smccc_1_2_regs resp;
> >>> +    uint32_t ret;
> >>> +
> >>> +    arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +    ret = get_ffa_ret_code(&resp);
> >>> +    if ( !ret )
> >>> +        *count = resp.a2;
> >>> +
> >>> +    return ret;
> >>> +}
> >>> +
> >>> +static int32_t ffa_rx_release(void)
> >>> +{
> >>> +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> >>> +}
> >>> +
> >>> +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
> >>> +                                      uint8_t msg)
> >>
> >> This function is actually only useable to send framework created/destroyed
> >> messages so the function name should be adapted to be less generic.
> >>
> >> ffa_send_vm_events ?
> >>
> >> unless you want to modify it later to send more framework messages ?
> >
> > That was the plan, but that may never happen. I'll rename it to
> > ffa_send_vm_event() since we're only sending one event at a time.
> >
> >>
> >>> +{
> >>> +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
> >>> +    int32_t res;
> >>> +
> >>> +    if ( msg == FFA_MSG_SEND_VM_CREATED )
> >>> +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
> >>> +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
> >>> +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
> >>> +    else
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    do {
> >>> +        const struct arm_smccc_1_2_regs arg = {
> >>> +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
> >>> +            .a1 = sp_id,
> >>> +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
> >>> +            .a5 = vm_id,
> >>> +        };
> >>> +        struct arm_smccc_1_2_regs resp;
> >>> +
> >>> +        arm_smccc_1_2_smc(&arg, &resp);
> >>> +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
> >>> +        {
> >>> +            /*
> >>> +             * This is an invalid response, likely due to some error in the
> >>> +             * implementation of the ABI.
> >>> +             */
> >>> +            return FFA_RET_INVALID_PARAMETERS;
> >>> +        }
> >>> +        res = resp.a3;
> >>> +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
> >>
> >> We might end up in an infinite loop here or increase interrupt response time.
> >> In the general case we could return that code directly to the VM but here we
> >> are in the VM creation/destroy path so we cannot do that.
> >>
> >> Maybe in debug mode at least we should have a retry counter here for now
> >> with a print ?
> >
> > OK, I'll add something.
> >
> >>
> >>> +
> >>> +    return res;
> >>> +}
> >>> +
> >>> static uint16_t get_vm_id(const struct domain *d)
> >>> {
> >>>    /* +1 since 0 is reserved for the hypervisor in FF-A */
> >>> @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>> static int ffa_domain_init(struct domain *d)
> >>> {
> >>>    struct ffa_ctx *ctx;
> >>> +    unsigned int n;
> >>> +    unsigned int m;
> >>> +    unsigned int c_pos;
> >>> +    int32_t res;
> >>>
> >>>     /*
> >>>      * We can't use that last possible domain ID or get_vm_id() would cause
> >>> @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
> >>>    if ( !ctx )
> >>>        return -ENOMEM;
> >>>
> >>> +    for ( n = 0; n < subscr_vm_created_count; n++ )
> >>> +    {
> >>> +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
> >>> +                                     FFA_MSG_SEND_VM_CREATED);
> >>> +        if ( res )
> >>> +        {
> >>> +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
> >>> +                   get_vm_id(d), subscr_vm_created[n], res);
> >>
> >> in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
> >> instead of calling get_vm_id each time.
> >
> > OK
> >
> >>
> >>> +            c_pos = n;
> >>> +            goto err;
> >>> +        }
> >>> +    }
> >>> +
> >>>    d->arch.tee = ctx;
> >>>
> >>>    return 0;
> >>> +
> >>> +err:
> >>> +    /* Undo any already sent vm created messaged */
> >>> +    for ( n = 0; n < c_pos; n++ )
> >>> +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
> >>> +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
> >>> +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> >>> +                                       FFA_MSG_SEND_VM_DESTROYED);
> >>> +
> >>> +    return -ENOMEM;
> >>
> >> The VM creation is not failing due to missing memory here.
> >> We need to find a better error code.
> >> Maybe ENOTCONN ?
> >> I am open to ideas here :-)
> >
> > That makes sense, I'll change it to ENOTCONN.
> >
> >>
> >>> }
> >>>
> >>> /* This function is supposed to undo what ffa_domain_init() has done */
> >>> static int ffa_relinquish_resources(struct domain *d)
> >>> {
> >>>    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    unsigned int n;
> >>> +    int32_t res;
> >>>
> >>>    if ( !ctx )
> >>>        return 0;
> >>>
> >>> +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
> >>> +    {
> >>> +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
> >>> +                                     FFA_MSG_SEND_VM_DESTROYED);
> >>> +
> >>> +        if ( res )
> >>> +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
> >>> +                   get_vm_id(d), subscr_vm_destroyed[n], res);
> >>> +    }
> >>> +
> >>>    XFREE(d->arch.tee);
> >>>
> >>>    return 0;
> >>> }
> >>>
> >>> +static bool init_subscribers(void)
> >>> +{
> >>> +    struct ffa_partition_info_1_1 *fpi;
> >>> +    bool ret = false;
> >>> +    uint32_t count;
> >>> +    int e;
> >>> +    uint32_t n;
> >>> +    uint32_t c_pos;
> >>> +    uint32_t d_pos;
> >>> +
> >>> +    if ( ffa_version < FFA_VERSION_1_1 )
> >>> +        return true;
> >>
> >> Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
> >> we could do the slow path and do a first loop on info_get until we get an error ?
> >
> > Sending the events is not supported in 1.0 so there's nothing to
> > record in that case.
>
> Please add a comment here to say that subscribers are only supported after 1.1
> and also mention it in the commit message.

OK.

Thanks,
Jens

Re: [XEN PATCH v7 12/20] xen/arm: ffa: send guest events to Secure Partitions

[Bertrand Marquis]

Hi Jens,

> On 1 Mar 2023, at 17:45, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi,
> 
> On Wed, Mar 1, 2023 at 1:58 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 1 Mar 2023, at 11:16, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Hi Bertrand,
>>> 
>>> On Tue, Feb 28, 2023 at 5:49 PM Bertrand Marquis
>>> <Bertrand.Marquis@arm.com> wrote:
>>>> 
>>>> Hi Jens,
>>>> 
>>>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>> 
>>>>> The FF-A specification defines framework messages sent as direct
>>>>> requests when certain events occurs. For instance when a VM (guest) is
>>>>> created or destroyed. Only SPs which have subscribed to these events
>>>>> will receive them. An SP can subscribe to these messages in its
>>>>> partition properties.
>>>>> 
>>>>> Adds a check that the SP supports the needed FF-A features
>>>>> FFA_PARTITION_INFO_GET and FFA_RX_RELEASE.
>>>>> 
>>>>> The partition properties of each SP is retrieved with
>>>>> FFA_PARTITION_INFO_GET which returns the information in our RX buffer.
>>>>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
>>>>> caller (us), so once we're done with the buffer it must be released
>>>>> using FFA_RX_RELEASE before another call can be made.
>>>>> 
>>>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>>>> ---
>>>>> xen/arch/arm/tee/ffa.c | 191 ++++++++++++++++++++++++++++++++++++++++-
>>>>> 1 file changed, 190 insertions(+), 1 deletion(-)
>>>>> 
>>>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>>>> index 07dd5c36d54b..f1b014b6c7f4 100644
>>>>> --- a/xen/arch/arm/tee/ffa.c
>>>>> +++ b/xen/arch/arm/tee/ffa.c
>>>>> @@ -140,6 +140,14 @@
>>>>> #define FFA_MSG_SEND                    0x8400006EU
>>>>> #define FFA_MSG_POLL                    0x8400006AU
>>>>> 
>>>>> +/* Partition information descriptor */
>>>>> +struct ffa_partition_info_1_1 {
>>>>> +    uint16_t id;
>>>>> +    uint16_t execution_context;
>>>>> +    uint32_t partition_properties;
>>>>> +    uint8_t uuid[16];
>>>>> +};
>>>>> +
>>>>> struct ffa_ctx {
>>>>>   uint32_t guest_vers;
>>>>>   bool interrupted;
>>>>> @@ -148,6 +156,12 @@ struct ffa_ctx {
>>>>> /* Negotiated FF-A version to use with the SPMC */
>>>>> static uint32_t ffa_version __ro_after_init;
>>>>> 
>>>>> +/* SPs subscribing to VM_CREATE and VM_DESTROYED events */
>>>>> +static uint16_t *subscr_vm_created __read_mostly;
>>>>> +static unsigned int subscr_vm_created_count __read_mostly;
>>>> 
>>>> In the spec the number is returned in w2 so you should utse uint32_t here.
>>> 
>>> I don't understand. This value is increased for each SP which has the
>>> property set in the Partition information descriptor.
>> 
>> Using generic types should be prevented when possible.
> 
> I'm usually of the opposite opinion, fixed width integers should only
> be used when there's a good reason to do so. However, if this is the
> Xen philosophy I can replace all those unsigned int with uint32_t if
> that's preferable.

Safety standards usually enforce to use explicit size types to prevent
compiler dependencies.

> 
>> Here this is a subset of the number of partition which is uint32_t (wX reg) so
>> i think this would be the logical type for this.
> 
> The maximum number is actually UINT16_MAX so an observant reader might
> wonder why the uint32_t type was used here.

Switching to uint16_t might make sense then but you will have to check that you
are not going over UINT16_MAX in the code as you get a uint32_t back from the call.


Cheers
Bertrand

> 
>> 
>>> 
>>>> 
>>>>> +static uint16_t *subscr_vm_destroyed __read_mostly;
>>>>> +static unsigned int subscr_vm_destroyed_count __read_mostly;
>>>> 
>>>> Same here
>>>> 
>>>>> +
>>>>> /*
>>>>> * Our rx/tx buffers shared with the SPMC.
>>>>> *
>>>>> @@ -237,6 +251,72 @@ static int32_t ffa_rxtx_map(register_t tx_addr, register_t rx_addr,
>>>>>   return ffa_simple_call(fid, tx_addr, rx_addr, page_count, 0);
>>>>> }
>>>>> 
>>>>> +static int32_t ffa_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
>>>>> +                                      uint32_t w4, uint32_t w5,
>>>>> +                                      uint32_t *count)
>>>>> +{
>>>>> +    const struct arm_smccc_1_2_regs arg = {
>>>>> +        .a0 = FFA_PARTITION_INFO_GET,
>>>>> +        .a1 = w1,
>>>>> +        .a2 = w2,
>>>>> +        .a3 = w3,
>>>>> +        .a4 = w4,
>>>>> +        .a5 = w5,
>>>>> +    };
>>>>> +    struct arm_smccc_1_2_regs resp;
>>>>> +    uint32_t ret;
>>>>> +
>>>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>>>> +
>>>>> +    ret = get_ffa_ret_code(&resp);
>>>>> +    if ( !ret )
>>>>> +        *count = resp.a2;
>>>>> +
>>>>> +    return ret;
>>>>> +}
>>>>> +
>>>>> +static int32_t ffa_rx_release(void)
>>>>> +{
>>>>> +    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
>>>>> +}
>>>>> +
>>>>> +static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>>>>> +                                      uint8_t msg)
>>>> 
>>>> This function is actually only useable to send framework created/destroyed
>>>> messages so the function name should be adapted to be less generic.
>>>> 
>>>> ffa_send_vm_events ?
>>>> 
>>>> unless you want to modify it later to send more framework messages ?
>>> 
>>> That was the plan, but that may never happen. I'll rename it to
>>> ffa_send_vm_event() since we're only sending one event at a time.
>>> 
>>>> 
>>>>> +{
>>>>> +    uint32_t exp_resp = FFA_MSG_FLAG_FRAMEWORK;
>>>>> +    int32_t res;
>>>>> +
>>>>> +    if ( msg == FFA_MSG_SEND_VM_CREATED )
>>>>> +        exp_resp |= FFA_MSG_RESP_VM_CREATED;
>>>>> +    else if ( msg == FFA_MSG_SEND_VM_DESTROYED )
>>>>> +        exp_resp |= FFA_MSG_RESP_VM_DESTROYED;
>>>>> +    else
>>>>> +        return FFA_RET_INVALID_PARAMETERS;
>>>>> +
>>>>> +    do {
>>>>> +        const struct arm_smccc_1_2_regs arg = {
>>>>> +            .a0 = FFA_MSG_SEND_DIRECT_REQ_32,
>>>>> +            .a1 = sp_id,
>>>>> +            .a2 = FFA_MSG_FLAG_FRAMEWORK | msg,
>>>>> +            .a5 = vm_id,
>>>>> +        };
>>>>> +        struct arm_smccc_1_2_regs resp;
>>>>> +
>>>>> +        arm_smccc_1_2_smc(&arg, &resp);
>>>>> +        if ( resp.a0 != FFA_MSG_SEND_DIRECT_RESP_32 || resp.a2 != exp_resp )
>>>>> +        {
>>>>> +            /*
>>>>> +             * This is an invalid response, likely due to some error in the
>>>>> +             * implementation of the ABI.
>>>>> +             */
>>>>> +            return FFA_RET_INVALID_PARAMETERS;
>>>>> +        }
>>>>> +        res = resp.a3;
>>>>> +    } while ( res == FFA_RET_INTERRUPTED || res == FFA_RET_RETRY );
>>>> 
>>>> We might end up in an infinite loop here or increase interrupt response time.
>>>> In the general case we could return that code directly to the VM but here we
>>>> are in the VM creation/destroy path so we cannot do that.
>>>> 
>>>> Maybe in debug mode at least we should have a retry counter here for now
>>>> with a print ?
>>> 
>>> OK, I'll add something.
>>> 
>>>> 
>>>>> +
>>>>> +    return res;
>>>>> +}
>>>>> +
>>>>> static uint16_t get_vm_id(const struct domain *d)
>>>>> {
>>>>>   /* +1 since 0 is reserved for the hypervisor in FF-A */
>>>>> @@ -371,6 +451,10 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>>> static int ffa_domain_init(struct domain *d)
>>>>> {
>>>>>   struct ffa_ctx *ctx;
>>>>> +    unsigned int n;
>>>>> +    unsigned int m;
>>>>> +    unsigned int c_pos;
>>>>> +    int32_t res;
>>>>> 
>>>>>    /*
>>>>>     * We can't use that last possible domain ID or get_vm_id() would cause
>>>>> @@ -383,24 +467,121 @@ static int ffa_domain_init(struct domain *d)
>>>>>   if ( !ctx )
>>>>>       return -ENOMEM;
>>>>> 
>>>>> +    for ( n = 0; n < subscr_vm_created_count; n++ )
>>>>> +    {
>>>>> +        res = ffa_direct_req_send_vm(subscr_vm_created[n], get_vm_id(d),
>>>>> +                                     FFA_MSG_SEND_VM_CREATED);
>>>>> +        if ( res )
>>>>> +        {
>>>>> +            printk(XENLOG_ERR "ffa: Failed to report creation of vm_id %u to  %u: res %d\n",
>>>>> +                   get_vm_id(d), subscr_vm_created[n], res);
>>>> 
>>>> in this function, get_vm_id(d) will not change so i would suggest to store it in a local variable
>>>> instead of calling get_vm_id each time.
>>> 
>>> OK
>>> 
>>>> 
>>>>> +            c_pos = n;
>>>>> +            goto err;
>>>>> +        }
>>>>> +    }
>>>>> +
>>>>>   d->arch.tee = ctx;
>>>>> 
>>>>>   return 0;
>>>>> +
>>>>> +err:
>>>>> +    /* Undo any already sent vm created messaged */
>>>>> +    for ( n = 0; n < c_pos; n++ )
>>>>> +        for ( m = 0; m < subscr_vm_destroyed_count; m++ )
>>>>> +            if ( subscr_vm_destroyed[m] == subscr_vm_created[n] )
>>>>> +                ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>>>> +                                       FFA_MSG_SEND_VM_DESTROYED);
>>>>> +
>>>>> +    return -ENOMEM;
>>>> 
>>>> The VM creation is not failing due to missing memory here.
>>>> We need to find a better error code.
>>>> Maybe ENOTCONN ?
>>>> I am open to ideas here :-)
>>> 
>>> That makes sense, I'll change it to ENOTCONN.
>>> 
>>>> 
>>>>> }
>>>>> 
>>>>> /* This function is supposed to undo what ffa_domain_init() has done */
>>>>> static int ffa_relinquish_resources(struct domain *d)
>>>>> {
>>>>>   struct ffa_ctx *ctx = d->arch.tee;
>>>>> +    unsigned int n;
>>>>> +    int32_t res;
>>>>> 
>>>>>   if ( !ctx )
>>>>>       return 0;
>>>>> 
>>>>> +    for ( n = 0; n < subscr_vm_destroyed_count; n++ )
>>>>> +    {
>>>>> +        res = ffa_direct_req_send_vm(subscr_vm_destroyed[n], get_vm_id(d),
>>>>> +                                     FFA_MSG_SEND_VM_DESTROYED);
>>>>> +
>>>>> +        if ( res )
>>>>> +            printk(XENLOG_ERR "ffa: Failed to report destruction of vm_id %u to  %u: res %d\n",
>>>>> +                   get_vm_id(d), subscr_vm_destroyed[n], res);
>>>>> +    }
>>>>> +
>>>>>   XFREE(d->arch.tee);
>>>>> 
>>>>>   return 0;
>>>>> }
>>>>> 
>>>>> +static bool init_subscribers(void)
>>>>> +{
>>>>> +    struct ffa_partition_info_1_1 *fpi;
>>>>> +    bool ret = false;
>>>>> +    uint32_t count;
>>>>> +    int e;
>>>>> +    uint32_t n;
>>>>> +    uint32_t c_pos;
>>>>> +    uint32_t d_pos;
>>>>> +
>>>>> +    if ( ffa_version < FFA_VERSION_1_1 )
>>>>> +        return true;
>>>> 
>>>> Correct me if i am wrong but on 1.0 version we cannot retrieve the count but
>>>> we could do the slow path and do a first loop on info_get until we get an error ?
>>> 
>>> Sending the events is not supported in 1.0 so there's nothing to
>>> record in that case.
>> 
>> Please add a comment here to say that subscribers are only supported after 1.1
>> and also mention it in the commit message.
> 
> OK.
> 
> Thanks,
> Jens

Re: [XEN PATCH v7 13/20] xen/arm: ffa: support mapping guest RX/TX buffers

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support in the mediator to map and unmap the RX and TX buffers
> provided by the guest using the two FF-A functions FFA_RXTX_MAP and
> FFA_RXTX_UNMAP.
> 
> These buffer are later used to to transmit data that cannot be passed in
> registers only.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 127 +++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 127 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index f1b014b6c7f4..953b6dfd5eca 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -149,10 +149,17 @@ struct ffa_partition_info_1_1 {
> };
> 
> struct ffa_ctx {
> +    void *rx;
> +    const void *tx;
> +    struct page_info *rx_pg;
> +    struct page_info *tx_pg;
> +    unsigned int page_count;
>     uint32_t guest_vers;
> +    bool tx_is_mine;
>     bool interrupted;
> };
> 
> +
Newline probably added by mistake.

> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> @@ -337,6 +344,11 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>         set_user_reg(regs, 7, v7);
> }
> 
> +static void set_regs_error(struct cpu_user_regs *regs, uint32_t error_code)
> +{
> +    set_regs(regs, FFA_ERROR, 0, error_code, 0, 0, 0, 0, 0);
> +}
> +
> static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
>                              uint32_t w3)
> {
> @@ -358,6 +370,99 @@ static void handle_version(struct cpu_user_regs *regs)
>     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> }
> 
> +static uint32_t handle_rxtx_map(uint32_t fid, register_t tx_addr,
> +                                register_t rx_addr, uint32_t page_count)
> +{
> +    uint32_t ret = FFA_RET_INVALID_PARAMETERS;
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    struct page_info *tx_pg;
> +    struct page_info *rx_pg;
> +    p2m_type_t t;
> +    void *rx;
> +    void *tx;
> +
> +    if ( !smccc_is_conv_64(fid) )
> +    {
> +        tx_addr &= UINT32_MAX;
> +        rx_addr &= UINT32_MAX;
> +    }

I am bit wondering here what we should do:
- we could just say that 32bit version of the call is not allowed for non 32bit guests
- we could check that the highest bits are 0 for 64bit guests and return an error if not
- we can just mask hopping that if there was a mistake the address after the mask
does not exist in the guest space

At the end nothing in the spec is preventing a 64bit guest to use the 32bit so it might
be a good idea to return an error if the highest 32bit are not 0 here ?

> +
> +    /* For now to keep things simple, only deal with a single page */
> +    if ( page_count != 1 )
> +        return FFA_RET_NOT_SUPPORTED;

Please add a TODO here and a print as this is a limitation we will probably have to
work on soon.


> +
> +    /* Already mapped */
> +    if ( ctx->rx )
> +        return FFA_RET_DENIED;
> +
> +    tx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(tx_addr)), &t, P2M_ALLOC);
> +    if ( !tx_pg )
> +        return FFA_RET_INVALID_PARAMETERS;
> +    /* Only normal RAM for now */
> +    if ( !p2m_is_ram(t) )
> +        goto err_put_tx_pg;
> +
> +    rx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(rx_addr)), &t, P2M_ALLOC);
> +    if ( !tx_pg )
> +        goto err_put_tx_pg;
> +    /* Only normal RAM for now */
> +    if ( !p2m_is_ram(t) )
> +        goto err_put_rx_pg;
> +
> +    tx = __map_domain_page_global(tx_pg);
> +    if ( !tx )
> +        goto err_put_rx_pg;
> +
> +    rx = __map_domain_page_global(rx_pg);
> +    if ( !rx )
> +        goto err_unmap_tx;
> +
> +    ctx->rx = rx;
> +    ctx->tx = tx;
> +    ctx->rx_pg = rx_pg;
> +    ctx->tx_pg = tx_pg;
> +    ctx->page_count = 1;

please use page_count here instead of 1 so that this is not forgotten once
we add support for more pages.


Cheers
Bertrand

> +    ctx->tx_is_mine = true;
> +    return FFA_RET_OK;
> +
> +err_unmap_tx:
> +    unmap_domain_page_global(tx);
> +err_put_rx_pg:
> +    put_page(rx_pg);
> +err_put_tx_pg:
> +    put_page(tx_pg);
> +
> +    return ret;
> +}
> +
> +static void rxtx_unmap(struct ffa_ctx *ctx)
> +{
> +    unmap_domain_page_global(ctx->rx);
> +    unmap_domain_page_global(ctx->tx);
> +    put_page(ctx->rx_pg);
> +    put_page(ctx->tx_pg);
> +    ctx->rx = NULL;
> +    ctx->tx = NULL;
> +    ctx->rx_pg = NULL;
> +    ctx->tx_pg = NULL;
> +    ctx->page_count = 0;
> +    ctx->tx_is_mine = false;
> +}
> +
> +static uint32_t handle_rxtx_unmap(void)
> +{
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +
> +    if ( !ctx->rx )
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    rxtx_unmap(ctx);
> +
> +    return FFA_RET_OK;
> +}
> +
> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> {
>     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> @@ -423,6 +528,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>     uint32_t fid = get_user_reg(regs, 0);
>     struct domain *d = current->domain;
>     struct ffa_ctx *ctx = d->arch.tee;
> +    int e;
> 
>     if ( !ctx )
>         return false;
> @@ -435,6 +541,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>     case FFA_ID_GET:
>         set_regs_success(regs, get_vm_id(d), 0);
>         return true;
> +    case FFA_RXTX_MAP_32:
> +#ifdef CONFIG_ARM_64
> +    case FFA_RXTX_MAP_64:
> +#endif
> +        e = handle_rxtx_map(fid, get_user_reg(regs, 1), get_user_reg(regs, 2),
> +                            get_user_reg(regs, 3));
> +        if ( e )
> +            set_regs_error(regs, e);
> +        else
> +            set_regs_success(regs, 0, 0);
> +        return true;
> +    case FFA_RXTX_UNMAP:
> +        e = handle_rxtx_unmap();
> +        if ( e )
> +            set_regs_error(regs, e);
> +        else
> +            set_regs_success(regs, 0, 0);
> +        return true;
>     case FFA_MSG_SEND_DIRECT_REQ_32:
> #ifdef CONFIG_ARM_64
>     case FFA_MSG_SEND_DIRECT_REQ_64:
> @@ -515,6 +639,9 @@ static int ffa_relinquish_resources(struct domain *d)
>                    get_vm_id(d), subscr_vm_destroyed[n], res);
>     }
> 
> +    if ( ctx->rx )
> +        rxtx_unmap(ctx);
> +
>     XFREE(d->arch.tee);
> 
>     return 0;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 07/20] xen/arm: ffa: add defines for framework direct request/response messages

[Jens Wiklander]

Hi Bertrand,

On Fri, Feb 24, 2023 at 10:39 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds defines for framework direct request/response messages.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 9 +++++++++
> > 1 file changed, 9 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index f4562ed2defc..d04bac9cc47f 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -56,6 +56,15 @@
> > #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
> >                                                  FFA_MY_VERSION_MINOR)
> >
> > +/* Framework direct request/response */
>
> In the previous patch you were more verbose in the comment which was nice.
> I would suggest here to use the same "format":
>
> Flags used for the MSG_SEND_DIRECT_REQ/RESP:
> BIT(31): Framework or partition message
> BIT(7-0): Message type for frameworks messages

OK, I'll update.

>
> > +#define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> > +#define FFA_MSG_TYPE_MASK               0xFFU;
>
> Maybe more coherent to name this FFA_MSG_FLAG_TYPE_MASK ?

This is a balancing act, in this case, I don't think that adding FLAG_
helps much.

>
> I am a bit unsure here because we could also keep it like that and just
> add _TYPE to other definitions after.
>
> What do you think ?

I think the defines are long enough as they are.

Cheers,
Jens

>
> > +#define FFA_MSG_PSCI                    0x0U
> > +#define FFA_MSG_SEND_VM_CREATED         0x4U
> > +#define FFA_MSG_RESP_VM_CREATED         0x5U
> > +#define FFA_MSG_SEND_VM_DESTROYED       0x6U
> > +#define FFA_MSG_RESP_VM_DESTROYED       0x7U
> > +
> > /*
> >  * Flags used for the FFA_PARTITION_INFO_GET return message:
> >  * BIT(0): Supports receipt of direct requests
> > --
> > 2.34.1
> >
>
> Cheers
> Bertrand
>

Re: [XEN PATCH v7 13/20] xen/arm: ffa: support mapping guest RX/TX buffers

[Jens Wiklander]

Hi Bertrand,

On Thu, Mar 2, 2023 at 4:05 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds support in the mediator to map and unmap the RX and TX buffers
> > provided by the guest using the two FF-A functions FFA_RXTX_MAP and
> > FFA_RXTX_UNMAP.
> >
> > These buffer are later used to to transmit data that cannot be passed in
> > registers only.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 127 +++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 127 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index f1b014b6c7f4..953b6dfd5eca 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -149,10 +149,17 @@ struct ffa_partition_info_1_1 {
> > };
> >
> > struct ffa_ctx {
> > +    void *rx;
> > +    const void *tx;
> > +    struct page_info *rx_pg;
> > +    struct page_info *tx_pg;
> > +    unsigned int page_count;
> >     uint32_t guest_vers;
> > +    bool tx_is_mine;
> >     bool interrupted;
> > };
> >
> > +
> Newline probably added by mistake.

Yes, I'll remove it.

>
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > @@ -337,6 +344,11 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> >         set_user_reg(regs, 7, v7);
> > }
> >
> > +static void set_regs_error(struct cpu_user_regs *regs, uint32_t error_code)
> > +{
> > +    set_regs(regs, FFA_ERROR, 0, error_code, 0, 0, 0, 0, 0);
> > +}
> > +
> > static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> >                              uint32_t w3)
> > {
> > @@ -358,6 +370,99 @@ static void handle_version(struct cpu_user_regs *regs)
> >     set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> > }
> >
> > +static uint32_t handle_rxtx_map(uint32_t fid, register_t tx_addr,
> > +                                register_t rx_addr, uint32_t page_count)
> > +{
> > +    uint32_t ret = FFA_RET_INVALID_PARAMETERS;
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +    struct page_info *tx_pg;
> > +    struct page_info *rx_pg;
> > +    p2m_type_t t;
> > +    void *rx;
> > +    void *tx;
> > +
> > +    if ( !smccc_is_conv_64(fid) )
> > +    {
> > +        tx_addr &= UINT32_MAX;
> > +        rx_addr &= UINT32_MAX;
> > +    }
>
> I am bit wondering here what we should do:
> - we could just say that 32bit version of the call is not allowed for non 32bit guests
> - we could check that the highest bits are 0 for 64bit guests and return an error if not
> - we can just mask hopping that if there was a mistake the address after the mask
> does not exist in the guest space
>
> At the end nothing in the spec is preventing a 64bit guest to use the 32bit so it might
> be a good idea to return an error if the highest 32bit are not 0 here ?

The SMC Calling Convention says:
When an SMC32/HVC32 call is made from AArch64:
- A Function Identifier is passed in register W0.
- Arguments are passed in registers W1-W7.

So masking off the higher bits is all that should be done.

>
> > +
> > +    /* For now to keep things simple, only deal with a single page */
> > +    if ( page_count != 1 )
> > +        return FFA_RET_NOT_SUPPORTED;
>
> Please add a TODO here and a print as this is a limitation we will probably have to
> work on soon.

I'll add an arbitrary upper limit and a print if it's exceeded.

>
>
> > +
> > +    /* Already mapped */
> > +    if ( ctx->rx )
> > +        return FFA_RET_DENIED;
> > +
> > +    tx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(tx_addr)), &t, P2M_ALLOC);
> > +    if ( !tx_pg )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +    /* Only normal RAM for now */
> > +    if ( !p2m_is_ram(t) )
> > +        goto err_put_tx_pg;
> > +
> > +    rx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(rx_addr)), &t, P2M_ALLOC);
> > +    if ( !tx_pg )
> > +        goto err_put_tx_pg;
> > +    /* Only normal RAM for now */
> > +    if ( !p2m_is_ram(t) )
> > +        goto err_put_rx_pg;
> > +
> > +    tx = __map_domain_page_global(tx_pg);
> > +    if ( !tx )
> > +        goto err_put_rx_pg;
> > +
> > +    rx = __map_domain_page_global(rx_pg);
> > +    if ( !rx )
> > +        goto err_unmap_tx;
> > +
> > +    ctx->rx = rx;
> > +    ctx->tx = tx;
> > +    ctx->rx_pg = rx_pg;
> > +    ctx->tx_pg = tx_pg;
> > +    ctx->page_count = 1;
>
> please use page_count here instead of 1 so that this is not forgotten once
> we add support for more pages.

OK

Cheers,
Jens

>
>
> Cheers
> Bertrand
>
> > +    ctx->tx_is_mine = true;
> > +    return FFA_RET_OK;
> > +
> > +err_unmap_tx:
> > +    unmap_domain_page_global(tx);
> > +err_put_rx_pg:
> > +    put_page(rx_pg);
> > +err_put_tx_pg:
> > +    put_page(tx_pg);
> > +
> > +    return ret;
> > +}
> > +
> > +static void rxtx_unmap(struct ffa_ctx *ctx)
> > +{
> > +    unmap_domain_page_global(ctx->rx);
> > +    unmap_domain_page_global(ctx->tx);
> > +    put_page(ctx->rx_pg);
> > +    put_page(ctx->tx_pg);
> > +    ctx->rx = NULL;
> > +    ctx->tx = NULL;
> > +    ctx->rx_pg = NULL;
> > +    ctx->tx_pg = NULL;
> > +    ctx->page_count = 0;
> > +    ctx->tx_is_mine = false;
> > +}
> > +
> > +static uint32_t handle_rxtx_unmap(void)
> > +{
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +
> > +    if ( !ctx->rx )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    rxtx_unmap(ctx);
> > +
> > +    return FFA_RET_OK;
> > +}
> > +
> > static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> > {
> >     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> > @@ -423,6 +528,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >     uint32_t fid = get_user_reg(regs, 0);
> >     struct domain *d = current->domain;
> >     struct ffa_ctx *ctx = d->arch.tee;
> > +    int e;
> >
> >     if ( !ctx )
> >         return false;
> > @@ -435,6 +541,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >     case FFA_ID_GET:
> >         set_regs_success(regs, get_vm_id(d), 0);
> >         return true;
> > +    case FFA_RXTX_MAP_32:
> > +#ifdef CONFIG_ARM_64
> > +    case FFA_RXTX_MAP_64:
> > +#endif
> > +        e = handle_rxtx_map(fid, get_user_reg(regs, 1), get_user_reg(regs, 2),
> > +                            get_user_reg(regs, 3));
> > +        if ( e )
> > +            set_regs_error(regs, e);
> > +        else
> > +            set_regs_success(regs, 0, 0);
> > +        return true;
> > +    case FFA_RXTX_UNMAP:
> > +        e = handle_rxtx_unmap();
> > +        if ( e )
> > +            set_regs_error(regs, e);
> > +        else
> > +            set_regs_success(regs, 0, 0);
> > +        return true;
> >     case FFA_MSG_SEND_DIRECT_REQ_32:
> > #ifdef CONFIG_ARM_64
> >     case FFA_MSG_SEND_DIRECT_REQ_64:
> > @@ -515,6 +639,9 @@ static int ffa_relinquish_resources(struct domain *d)
> >                    get_vm_id(d), subscr_vm_destroyed[n], res);
> >     }
> >
> > +    if ( ctx->rx )
> > +        rxtx_unmap(ctx);
> > +
> >     XFREE(d->arch.tee);
> >
> >     return 0;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 07/20] xen/arm: ffa: add defines for framework direct request/response messages

[Bertrand Marquis]

HI Jens,

> On 3 Mar 2023, at 08:01, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Fri, Feb 24, 2023 at 10:39 AM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds defines for framework direct request/response messages.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 9 +++++++++
>>> 1 file changed, 9 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index f4562ed2defc..d04bac9cc47f 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -56,6 +56,15 @@
>>> #define FFA_MY_VERSION          MAKE_FFA_VERSION(FFA_MY_VERSION_MAJOR, \
>>>                                                 FFA_MY_VERSION_MINOR)
>>> 
>>> +/* Framework direct request/response */
>> 
>> In the previous patch you were more verbose in the comment which was nice.
>> I would suggest here to use the same "format":
>> 
>> Flags used for the MSG_SEND_DIRECT_REQ/RESP:
>> BIT(31): Framework or partition message
>> BIT(7-0): Message type for frameworks messages
> 
> OK, I'll update.
> 
>> 
>>> +#define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
>>> +#define FFA_MSG_TYPE_MASK               0xFFU;
>> 
>> Maybe more coherent to name this FFA_MSG_FLAG_TYPE_MASK ?
> 
> This is a balancing act, in this case, I don't think that adding FLAG_
> helps much.

Agree you can remove flag.

> 
>> 
>> I am a bit unsure here because we could also keep it like that and just
>> add _TYPE to other definitions after.
>> 
>> What do you think ?
> 
> I think the defines are long enough as they are.

Right.

Cheers
Bertrand

> 
> Cheers,
> Jens
> 
>> 
>>> +#define FFA_MSG_PSCI                    0x0U
>>> +#define FFA_MSG_SEND_VM_CREATED         0x4U
>>> +#define FFA_MSG_RESP_VM_CREATED         0x5U
>>> +#define FFA_MSG_SEND_VM_DESTROYED       0x6U
>>> +#define FFA_MSG_RESP_VM_DESTROYED       0x7U
>>> +
>>> /*
>>> * Flags used for the FFA_PARTITION_INFO_GET return message:
>>> * BIT(0): Supports receipt of direct requests
>>> --
>>> 2.34.1
>>> 
>> 
>> Cheers
>> Bertrand

Re: [XEN PATCH v7 13/20] xen/arm: ffa: support mapping guest RX/TX buffers

[Bertrand Marquis]

Hi Jens,

> On 3 Mar 2023, at 08:41, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Thu, Mar 2, 2023 at 4:05 PM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support in the mediator to map and unmap the RX and TX buffers
>>> provided by the guest using the two FF-A functions FFA_RXTX_MAP and
>>> FFA_RXTX_UNMAP.
>>> 
>>> These buffer are later used to to transmit data that cannot be passed in
>>> registers only.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 127 +++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 127 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index f1b014b6c7f4..953b6dfd5eca 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -149,10 +149,17 @@ struct ffa_partition_info_1_1 {
>>> };
>>> 
>>> struct ffa_ctx {
>>> +    void *rx;
>>> +    const void *tx;
>>> +    struct page_info *rx_pg;
>>> +    struct page_info *tx_pg;
>>> +    unsigned int page_count;
>>>    uint32_t guest_vers;
>>> +    bool tx_is_mine;
>>>    bool interrupted;
>>> };
>>> 
>>> +
>> Newline probably added by mistake.
> 
> Yes, I'll remove it.
> 
>> 
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> @@ -337,6 +344,11 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
>>>        set_user_reg(regs, 7, v7);
>>> }
>>> 
>>> +static void set_regs_error(struct cpu_user_regs *regs, uint32_t error_code)
>>> +{
>>> +    set_regs(regs, FFA_ERROR, 0, error_code, 0, 0, 0, 0, 0);
>>> +}
>>> +
>>> static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
>>>                             uint32_t w3)
>>> {
>>> @@ -358,6 +370,99 @@ static void handle_version(struct cpu_user_regs *regs)
>>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
>>> }
>>> 
>>> +static uint32_t handle_rxtx_map(uint32_t fid, register_t tx_addr,
>>> +                                register_t rx_addr, uint32_t page_count)
>>> +{
>>> +    uint32_t ret = FFA_RET_INVALID_PARAMETERS;
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +    struct page_info *tx_pg;
>>> +    struct page_info *rx_pg;
>>> +    p2m_type_t t;
>>> +    void *rx;
>>> +    void *tx;
>>> +
>>> +    if ( !smccc_is_conv_64(fid) )
>>> +    {
>>> +        tx_addr &= UINT32_MAX;
>>> +        rx_addr &= UINT32_MAX;
>>> +    }
>> 
>> I am bit wondering here what we should do:
>> - we could just say that 32bit version of the call is not allowed for non 32bit guests
>> - we could check that the highest bits are 0 for 64bit guests and return an error if not
>> - we can just mask hopping that if there was a mistake the address after the mask
>> does not exist in the guest space
>> 
>> At the end nothing in the spec is preventing a 64bit guest to use the 32bit so it might
>> be a good idea to return an error if the highest 32bit are not 0 here ?
> 
> The SMC Calling Convention says:
> When an SMC32/HVC32 call is made from AArch64:
> - A Function Identifier is passed in register W0.
> - Arguments are passed in registers W1-W7.
> 
> So masking off the higher bits is all that should be done.

Please put a comment saying that in 32 bit convention higher bits should be ignored.

> 
>> 
>>> +
>>> +    /* For now to keep things simple, only deal with a single page */
>>> +    if ( page_count != 1 )
>>> +        return FFA_RET_NOT_SUPPORTED;
>> 
>> Please add a TODO here and a print as this is a limitation we will probably have to
>> work on soon.
> 
> I'll add an arbitrary upper limit and a print if it's exceeded.

thanks

> 
>> 
>> 
>>> +
>>> +    /* Already mapped */
>>> +    if ( ctx->rx )
>>> +        return FFA_RET_DENIED;
>>> +
>>> +    tx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(tx_addr)), &t, P2M_ALLOC);
>>> +    if ( !tx_pg )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +    /* Only normal RAM for now */
>>> +    if ( !p2m_is_ram(t) )
>>> +        goto err_put_tx_pg;
>>> +
>>> +    rx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(rx_addr)), &t, P2M_ALLOC);
>>> +    if ( !tx_pg )
>>> +        goto err_put_tx_pg;
>>> +    /* Only normal RAM for now */
>>> +    if ( !p2m_is_ram(t) )
>>> +        goto err_put_rx_pg;
>>> +
>>> +    tx = __map_domain_page_global(tx_pg);
>>> +    if ( !tx )
>>> +        goto err_put_rx_pg;
>>> +
>>> +    rx = __map_domain_page_global(rx_pg);
>>> +    if ( !rx )
>>> +        goto err_unmap_tx;
>>> +
>>> +    ctx->rx = rx;
>>> +    ctx->tx = tx;
>>> +    ctx->rx_pg = rx_pg;
>>> +    ctx->tx_pg = tx_pg;
>>> +    ctx->page_count = 1;
>> 
>> please use page_count here instead of 1 so that this is not forgotten once
>> we add support for more pages.
> 
> OK

Cheers
Bertrand

> 
> Cheers,
> Jens
> 
>> 
>> 
>> Cheers
>> Bertrand
>> 
>>> +    ctx->tx_is_mine = true;
>>> +    return FFA_RET_OK;
>>> +
>>> +err_unmap_tx:
>>> +    unmap_domain_page_global(tx);
>>> +err_put_rx_pg:
>>> +    put_page(rx_pg);
>>> +err_put_tx_pg:
>>> +    put_page(tx_pg);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +static void rxtx_unmap(struct ffa_ctx *ctx)
>>> +{
>>> +    unmap_domain_page_global(ctx->rx);
>>> +    unmap_domain_page_global(ctx->tx);
>>> +    put_page(ctx->rx_pg);
>>> +    put_page(ctx->tx_pg);
>>> +    ctx->rx = NULL;
>>> +    ctx->tx = NULL;
>>> +    ctx->rx_pg = NULL;
>>> +    ctx->tx_pg = NULL;
>>> +    ctx->page_count = 0;
>>> +    ctx->tx_is_mine = false;
>>> +}
>>> +
>>> +static uint32_t handle_rxtx_unmap(void)
>>> +{
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +
>>> +    if ( !ctx->rx )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    rxtx_unmap(ctx);
>>> +
>>> +    return FFA_RET_OK;
>>> +}
>>> +
>>> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
>>> {
>>>    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
>>> @@ -423,6 +528,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>    uint32_t fid = get_user_reg(regs, 0);
>>>    struct domain *d = current->domain;
>>>    struct ffa_ctx *ctx = d->arch.tee;
>>> +    int e;
>>> 
>>>    if ( !ctx )
>>>        return false;
>>> @@ -435,6 +541,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>    case FFA_ID_GET:
>>>        set_regs_success(regs, get_vm_id(d), 0);
>>>        return true;
>>> +    case FFA_RXTX_MAP_32:
>>> +#ifdef CONFIG_ARM_64
>>> +    case FFA_RXTX_MAP_64:
>>> +#endif
>>> +        e = handle_rxtx_map(fid, get_user_reg(regs, 1), get_user_reg(regs, 2),
>>> +                            get_user_reg(regs, 3));
>>> +        if ( e )
>>> +            set_regs_error(regs, e);
>>> +        else
>>> +            set_regs_success(regs, 0, 0);
>>> +        return true;
>>> +    case FFA_RXTX_UNMAP:
>>> +        e = handle_rxtx_unmap();
>>> +        if ( e )
>>> +            set_regs_error(regs, e);
>>> +        else
>>> +            set_regs_success(regs, 0, 0);
>>> +        return true;
>>>    case FFA_MSG_SEND_DIRECT_REQ_32:
>>> #ifdef CONFIG_ARM_64
>>>    case FFA_MSG_SEND_DIRECT_REQ_64:
>>> @@ -515,6 +639,9 @@ static int ffa_relinquish_resources(struct domain *d)
>>>                   get_vm_id(d), subscr_vm_destroyed[n], res);
>>>    }
>>> 
>>> +    if ( ctx->rx )
>>> +        rxtx_unmap(ctx);
>>> +
>>>    XFREE(d->arch.tee);
>>> 
>>>    return 0;
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 14/20] xen/arm: ffa: support guest FFA_PARTITION_INFO_GET

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support in the mediator to handle FFA_PARTITION_INFO_GET requests
> from a guest. The requests are forwarded to the SPMC and the response is
> translated according to the FF-A version in use by the guest.
> 
> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> caller (the guest in this case), so once it is done with the buffer it
> must be released using FFA_RX_RELEASE before another call can be made.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 126 ++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 124 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 953b6dfd5eca..3571817c0bcd 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -141,6 +141,12 @@
> #define FFA_MSG_POLL                    0x8400006AU
> 
> /* Partition information descriptor */
> +struct ffa_partition_info_1_0 {
> +    uint16_t id;
> +    uint16_t execution_context;
> +    uint32_t partition_properties;
> +};
> +
> struct ffa_partition_info_1_1 {
>     uint16_t id;
>     uint16_t execution_context;
> @@ -157,9 +163,8 @@ struct ffa_ctx {
>     uint32_t guest_vers;
>     bool tx_is_mine;
>     bool interrupted;
> +    spinlock_t lock;
> };
> -
> -

This is removing 2 empty lines (previous patch was wrongly adding one)
but one empty line is required here.

> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> @@ -173,10 +178,16 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
>  * Our rx/tx buffers shared with the SPMC.
>  *
>  * ffa_page_count is the number of pages used in each of these buffers.
> + *
> + * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
> + * Note that the SPMC is also tracking the ownership of our RX buffer so
> + * for calls which uses our RX buffer to deliver a result we must call
> + * ffa_rx_release() to let the SPMC know that we're done with the buffer.
>  */
> static void *ffa_rx __read_mostly;
> static void *ffa_tx __read_mostly;
> static unsigned int ffa_page_count __read_mostly;
> +static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> 
> static bool ffa_get_version(uint32_t *vers)
> {
> @@ -463,6 +474,98 @@ static uint32_t handle_rxtx_unmap(void)
>     return FFA_RET_OK;
> }
> 
> +static uint32_t handle_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> +                                          uint32_t w4, uint32_t w5,
> +                                          uint32_t *count)
> +{
> +    bool query_count_only = w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG;
> +    uint32_t w5_mask = 0;
> +    uint32_t ret = FFA_RET_DENIED;
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +
> +    /*
> +     * FF-A v1.0 has w5 MBZ while v1.1 allows
> +     * FFA_PARTITION_INFO_GET_COUNT_FLAG to be non-zero.
> +     */
> +    if ( ctx->guest_vers == FFA_VERSION_1_1 )
> +        w5_mask = FFA_PARTITION_INFO_GET_COUNT_FLAG;
> +    if ( w5 & ~w5_mask )
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    if ( query_count_only )
> +        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);

This code seems a bit to complex.
I would suggest the following:

if (w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG)
{
     if ( ctx->guest_vers == FFA_VERSION_1_1 )
	return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
     else
        return FFA_RET_INVALID_PARAMETERS;
}

> +
> +    if ( !ffa_page_count )
> +        return FFA_RET_DENIED;
> +
> +    spin_lock(&ctx->lock);
> +    spin_lock(&ffa_rx_buffer_lock);
> +    if ( !ctx->page_count || !ctx->tx_is_mine )

If i understand correctly tx_is_mine is protecting the guest rx
buffer until rx_release is called by the guest so that we do not
write in it before the guest has retrieved the data from it.

The name is very misleading, maybe rx_is_writeable or free would be better ?

Also it would be more optimal to test it before taking ffa_rx_buffer_lock.


> +        goto out;
> +    ret = ffa_partition_info_get(w1, w2, w3, w4, w5, count);
> +    if ( ret )
> +        goto out;
> +
> +    if ( ctx->guest_vers == FFA_VERSION_1_0 )
> +    {
> +        size_t n;
> +        struct ffa_partition_info_1_1 *src = ffa_rx;
> +        struct ffa_partition_info_1_0 *dst = ctx->rx;
> +
> +        if ( ctx->page_count * FFA_PAGE_SIZE < *count * sizeof(*dst) )
> +        {
> +            ret = FFA_RET_NO_MEMORY;
> +            goto out_rx_release;
> +        }
> +
> +        for ( n = 0; n < *count; n++ )
> +        {
> +            dst[n].id = src[n].id;
> +            dst[n].execution_context = src[n].execution_context;
> +            dst[n].partition_properties = src[n].partition_properties;
> +        }
> +    }
> +    else
> +    {
> +        size_t sz = *count * sizeof(struct ffa_partition_info_1_1);
> +
> +        if ( ctx->page_count * FFA_PAGE_SIZE < sz )
> +        {
> +            ret = FFA_RET_NO_MEMORY;
> +            goto out_rx_release;
> +        }
> +
> +
> +        memcpy(ctx->rx, ffa_rx, sz);
> +    }
> +    ctx->tx_is_mine = false;

at this point we have no reason to hold ctx->lock

> +out_rx_release:
> +    ffa_rx_release();

There should be no case where do release without unlocking.

It might be cleaner to have 2 functions ffa_rx_get and ffa_rx_release
 handling both the lock and the rx_release message.

> +out:
> +    spin_unlock(&ffa_rx_buffer_lock);

this should stay with ffa_rx_release

Cheers
Bertrand

> +    spin_unlock(&ctx->lock);
> +
> +    return ret;
> +}
> +
> +static uint32_t handle_rx_release(void)
> +{
> +    uint32_t ret = FFA_RET_DENIED;
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +
> +    spin_lock(&ctx->lock);
> +    if ( !ctx->page_count || ctx->tx_is_mine )
> +        goto out;
> +    ret = FFA_RET_OK;
> +    ctx->tx_is_mine = true;
> +out:
> +    spin_unlock(&ctx->lock);
> +
> +    return ret;
> +}
> +
> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> {
>     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> @@ -528,6 +631,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>     uint32_t fid = get_user_reg(regs, 0);
>     struct domain *d = current->domain;
>     struct ffa_ctx *ctx = d->arch.tee;
> +    uint32_t count;
>     int e;
> 
>     if ( !ctx )
> @@ -559,6 +663,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>         else
>             set_regs_success(regs, 0, 0);
>         return true;
> +    case FFA_PARTITION_INFO_GET:
> +        e = handle_partition_info_get(get_user_reg(regs, 1),
> +                                      get_user_reg(regs, 2),
> +                                      get_user_reg(regs, 3),
> +                                      get_user_reg(regs, 4),
> +                                      get_user_reg(regs, 5), &count);
> +        if ( e )
> +            set_regs_error(regs, e);
> +        else
> +            set_regs_success(regs, count, 0);
> +        return true;
> +    case FFA_RX_RELEASE:
> +        e = handle_rx_release();
> +        if ( e )
> +            set_regs_error(regs, e);
> +        else
> +            set_regs_success(regs, 0, 0);
> +        return true;
>     case FFA_MSG_SEND_DIRECT_REQ_32:
> #ifdef CONFIG_ARM_64
>     case FFA_MSG_SEND_DIRECT_REQ_64:
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 13/20] xen/arm: ffa: support mapping guest RX/TX buffers

[Jens Wiklander]

Hi,

On Fri, Mar 3, 2023 at 9:16 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 3 Mar 2023, at 08:41, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Thu, Mar 2, 2023 at 4:05 PM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support in the mediator to map and unmap the RX and TX buffers
> >>> provided by the guest using the two FF-A functions FFA_RXTX_MAP and
> >>> FFA_RXTX_UNMAP.
> >>>
> >>> These buffer are later used to to transmit data that cannot be passed in
> >>> registers only.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 127 +++++++++++++++++++++++++++++++++++++++++
> >>> 1 file changed, 127 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index f1b014b6c7f4..953b6dfd5eca 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -149,10 +149,17 @@ struct ffa_partition_info_1_1 {
> >>> };
> >>>
> >>> struct ffa_ctx {
> >>> +    void *rx;
> >>> +    const void *tx;
> >>> +    struct page_info *rx_pg;
> >>> +    struct page_info *tx_pg;
> >>> +    unsigned int page_count;
> >>>    uint32_t guest_vers;
> >>> +    bool tx_is_mine;
> >>>    bool interrupted;
> >>> };
> >>>
> >>> +
> >> Newline probably added by mistake.
> >
> > Yes, I'll remove it.
> >
> >>
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> @@ -337,6 +344,11 @@ static void set_regs(struct cpu_user_regs *regs, register_t v0, register_t v1,
> >>>        set_user_reg(regs, 7, v7);
> >>> }
> >>>
> >>> +static void set_regs_error(struct cpu_user_regs *regs, uint32_t error_code)
> >>> +{
> >>> +    set_regs(regs, FFA_ERROR, 0, error_code, 0, 0, 0, 0, 0);
> >>> +}
> >>> +
> >>> static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> >>>                             uint32_t w3)
> >>> {
> >>> @@ -358,6 +370,99 @@ static void handle_version(struct cpu_user_regs *regs)
> >>>    set_regs(regs, vers, 0, 0, 0, 0, 0, 0, 0);
> >>> }
> >>>
> >>> +static uint32_t handle_rxtx_map(uint32_t fid, register_t tx_addr,
> >>> +                                register_t rx_addr, uint32_t page_count)
> >>> +{
> >>> +    uint32_t ret = FFA_RET_INVALID_PARAMETERS;
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    struct page_info *tx_pg;
> >>> +    struct page_info *rx_pg;
> >>> +    p2m_type_t t;
> >>> +    void *rx;
> >>> +    void *tx;
> >>> +
> >>> +    if ( !smccc_is_conv_64(fid) )
> >>> +    {
> >>> +        tx_addr &= UINT32_MAX;
> >>> +        rx_addr &= UINT32_MAX;
> >>> +    }
> >>
> >> I am bit wondering here what we should do:
> >> - we could just say that 32bit version of the call is not allowed for non 32bit guests
> >> - we could check that the highest bits are 0 for 64bit guests and return an error if not
> >> - we can just mask hopping that if there was a mistake the address after the mask
> >> does not exist in the guest space
> >>
> >> At the end nothing in the spec is preventing a 64bit guest to use the 32bit so it might
> >> be a good idea to return an error if the highest 32bit are not 0 here ?
> >
> > The SMC Calling Convention says:
> > When an SMC32/HVC32 call is made from AArch64:
> > - A Function Identifier is passed in register W0.
> > - Arguments are passed in registers W1-W7.
> >
> > So masking off the higher bits is all that should be done.
>
> Please put a comment saying that in 32 bit convention higher bits should be ignored.

OK.

Thanks,
Jens

>
> >
> >>
> >>> +
> >>> +    /* For now to keep things simple, only deal with a single page */
> >>> +    if ( page_count != 1 )
> >>> +        return FFA_RET_NOT_SUPPORTED;
> >>
> >> Please add a TODO here and a print as this is a limitation we will probably have to
> >> work on soon.
> >
> > I'll add an arbitrary upper limit and a print if it's exceeded.
>
> thanks
>
> >
> >>
> >>
> >>> +
> >>> +    /* Already mapped */
> >>> +    if ( ctx->rx )
> >>> +        return FFA_RET_DENIED;
> >>> +
> >>> +    tx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(tx_addr)), &t, P2M_ALLOC);
> >>> +    if ( !tx_pg )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +    /* Only normal RAM for now */
> >>> +    if ( !p2m_is_ram(t) )
> >>> +        goto err_put_tx_pg;
> >>> +
> >>> +    rx_pg = get_page_from_gfn(d, gfn_x(gaddr_to_gfn(rx_addr)), &t, P2M_ALLOC);
> >>> +    if ( !tx_pg )
> >>> +        goto err_put_tx_pg;
> >>> +    /* Only normal RAM for now */
> >>> +    if ( !p2m_is_ram(t) )
> >>> +        goto err_put_rx_pg;
> >>> +
> >>> +    tx = __map_domain_page_global(tx_pg);
> >>> +    if ( !tx )
> >>> +        goto err_put_rx_pg;
> >>> +
> >>> +    rx = __map_domain_page_global(rx_pg);
> >>> +    if ( !rx )
> >>> +        goto err_unmap_tx;
> >>> +
> >>> +    ctx->rx = rx;
> >>> +    ctx->tx = tx;
> >>> +    ctx->rx_pg = rx_pg;
> >>> +    ctx->tx_pg = tx_pg;
> >>> +    ctx->page_count = 1;
> >>
> >> please use page_count here instead of 1 so that this is not forgotten once
> >> we add support for more pages.
> >
> > OK
>
> Cheers
> Bertrand
>
> >
> > Cheers,
> > Jens
> >
> >>
> >>
> >> Cheers
> >> Bertrand
> >>
> >>> +    ctx->tx_is_mine = true;
> >>> +    return FFA_RET_OK;
> >>> +
> >>> +err_unmap_tx:
> >>> +    unmap_domain_page_global(tx);
> >>> +err_put_rx_pg:
> >>> +    put_page(rx_pg);
> >>> +err_put_tx_pg:
> >>> +    put_page(tx_pg);
> >>> +
> >>> +    return ret;
> >>> +}
> >>> +
> >>> +static void rxtx_unmap(struct ffa_ctx *ctx)
> >>> +{
> >>> +    unmap_domain_page_global(ctx->rx);
> >>> +    unmap_domain_page_global(ctx->tx);
> >>> +    put_page(ctx->rx_pg);
> >>> +    put_page(ctx->tx_pg);
> >>> +    ctx->rx = NULL;
> >>> +    ctx->tx = NULL;
> >>> +    ctx->rx_pg = NULL;
> >>> +    ctx->tx_pg = NULL;
> >>> +    ctx->page_count = 0;
> >>> +    ctx->tx_is_mine = false;
> >>> +}
> >>> +
> >>> +static uint32_t handle_rxtx_unmap(void)
> >>> +{
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +
> >>> +    if ( !ctx->rx )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    rxtx_unmap(ctx);
> >>> +
> >>> +    return FFA_RET_OK;
> >>> +}
> >>> +
> >>> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> >>> {
> >>>    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> >>> @@ -423,6 +528,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>    uint32_t fid = get_user_reg(regs, 0);
> >>>    struct domain *d = current->domain;
> >>>    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    int e;
> >>>
> >>>    if ( !ctx )
> >>>        return false;
> >>> @@ -435,6 +541,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>    case FFA_ID_GET:
> >>>        set_regs_success(regs, get_vm_id(d), 0);
> >>>        return true;
> >>> +    case FFA_RXTX_MAP_32:
> >>> +#ifdef CONFIG_ARM_64
> >>> +    case FFA_RXTX_MAP_64:
> >>> +#endif
> >>> +        e = handle_rxtx_map(fid, get_user_reg(regs, 1), get_user_reg(regs, 2),
> >>> +                            get_user_reg(regs, 3));
> >>> +        if ( e )
> >>> +            set_regs_error(regs, e);
> >>> +        else
> >>> +            set_regs_success(regs, 0, 0);
> >>> +        return true;
> >>> +    case FFA_RXTX_UNMAP:
> >>> +        e = handle_rxtx_unmap();
> >>> +        if ( e )
> >>> +            set_regs_error(regs, e);
> >>> +        else
> >>> +            set_regs_success(regs, 0, 0);
> >>> +        return true;
> >>>    case FFA_MSG_SEND_DIRECT_REQ_32:
> >>> #ifdef CONFIG_ARM_64
> >>>    case FFA_MSG_SEND_DIRECT_REQ_64:
> >>> @@ -515,6 +639,9 @@ static int ffa_relinquish_resources(struct domain *d)
> >>>                   get_vm_id(d), subscr_vm_destroyed[n], res);
> >>>    }
> >>>
> >>> +    if ( ctx->rx )
> >>> +        rxtx_unmap(ctx);
> >>> +
> >>>    XFREE(d->arch.tee);
> >>>
> >>>    return 0;
> >>> --
> >>> 2.34.1
>
>

Re: [XEN PATCH v7 15/20] xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Moves the two helper functions regpair_to_uint64() and
> uint64_to_regpair() from xen/arch/arm/tee/optee.c to the common arm
> specific regs.h. This enables reuse of these functions in the FF-A
> mediator in a subsequent patch.
> 
> Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

Please move Michal R-B after your signed off.

With that fixed:
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> xen/arch/arm/include/asm/regs.h | 12 ++++++++++++
> xen/arch/arm/tee/optee.c        | 11 -----------
> 2 files changed, 12 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/arch/arm/include/asm/regs.h b/xen/arch/arm/include/asm/regs.h
> index 0693a681315f..aa39e83ee5f4 100644
> --- a/xen/arch/arm/include/asm/regs.h
> +++ b/xen/arch/arm/include/asm/regs.h
> @@ -60,6 +60,18 @@ static inline bool guest_mode(const struct cpu_user_regs *r)
> register_t get_user_reg(struct cpu_user_regs *regs, int reg);
> void set_user_reg(struct cpu_user_regs *regs, int reg, register_t val);
> 
> +static inline uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
> +{
> +    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
> +}
> +
> +static inline void uint64_to_regpair(register_t *reg0, register_t *reg1,
> +                                     uint64_t val)
> +{
> +    *reg0 = val >> 32;
> +    *reg1 = (uint32_t)val;
> +}
> +
> #endif
> 
> #endif /* __ARM_REGS_H__ */
> diff --git a/xen/arch/arm/tee/optee.c b/xen/arch/arm/tee/optee.c
> index 9cb9f16d43cb..47027ecef47c 100644
> --- a/xen/arch/arm/tee/optee.c
> +++ b/xen/arch/arm/tee/optee.c
> @@ -268,17 +268,6 @@ static int optee_domain_init(struct domain *d)
>     return 0;
> }
> 
> -static uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
> -{
> -    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
> -}
> -
> -static void uint64_to_regpair(register_t *reg0, register_t *reg1, uint64_t val)
> -{
> -    *reg0 = val >> 32;
> -    *reg1 = (uint32_t)val;
> -}
> -
> static struct page_info *get_domain_ram_page(gfn_t gfn)
> {
>     struct page_info *page;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 14/20] xen/arm: ffa: support guest FFA_PARTITION_INFO_GET

[Jens Wiklander]

Hi Bertrand,

On Fri, Mar 3, 2023 at 10:51 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds support in the mediator to handle FFA_PARTITION_INFO_GET requests
> > from a guest. The requests are forwarded to the SPMC and the response is
> > translated according to the FF-A version in use by the guest.
> >
> > Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> > caller (the guest in this case), so once it is done with the buffer it
> > must be released using FFA_RX_RELEASE before another call can be made.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 126 ++++++++++++++++++++++++++++++++++++++++-
> > 1 file changed, 124 insertions(+), 2 deletions(-)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 953b6dfd5eca..3571817c0bcd 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -141,6 +141,12 @@
> > #define FFA_MSG_POLL                    0x8400006AU
> >
> > /* Partition information descriptor */
> > +struct ffa_partition_info_1_0 {
> > +    uint16_t id;
> > +    uint16_t execution_context;
> > +    uint32_t partition_properties;
> > +};
> > +
> > struct ffa_partition_info_1_1 {
> >     uint16_t id;
> >     uint16_t execution_context;
> > @@ -157,9 +163,8 @@ struct ffa_ctx {
> >     uint32_t guest_vers;
> >     bool tx_is_mine;
> >     bool interrupted;
> > +    spinlock_t lock;
> > };
> > -
> > -
>
> This is removing 2 empty lines (previous patch was wrongly adding one)
> but one empty line is required here.

I'll fix it.

>
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > @@ -173,10 +178,16 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
> >  * Our rx/tx buffers shared with the SPMC.
> >  *
> >  * ffa_page_count is the number of pages used in each of these buffers.
> > + *
> > + * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
> > + * Note that the SPMC is also tracking the ownership of our RX buffer so
> > + * for calls which uses our RX buffer to deliver a result we must call
> > + * ffa_rx_release() to let the SPMC know that we're done with the buffer.
> >  */
> > static void *ffa_rx __read_mostly;
> > static void *ffa_tx __read_mostly;
> > static unsigned int ffa_page_count __read_mostly;
> > +static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> >
> > static bool ffa_get_version(uint32_t *vers)
> > {
> > @@ -463,6 +474,98 @@ static uint32_t handle_rxtx_unmap(void)
> >     return FFA_RET_OK;
> > }
> >
> > +static uint32_t handle_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> > +                                          uint32_t w4, uint32_t w5,
> > +                                          uint32_t *count)
> > +{
> > +    bool query_count_only = w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG;
> > +    uint32_t w5_mask = 0;
> > +    uint32_t ret = FFA_RET_DENIED;
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +
> > +    /*
> > +     * FF-A v1.0 has w5 MBZ while v1.1 allows
> > +     * FFA_PARTITION_INFO_GET_COUNT_FLAG to be non-zero.
> > +     */
> > +    if ( ctx->guest_vers == FFA_VERSION_1_1 )
> > +        w5_mask = FFA_PARTITION_INFO_GET_COUNT_FLAG;
> > +    if ( w5 & ~w5_mask )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    if ( query_count_only )
> > +        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
>
> This code seems a bit to complex.
> I would suggest the following:
>
> if (w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG)
> {
>      if ( ctx->guest_vers == FFA_VERSION_1_1 )
>         return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
>      else
>         return FFA_RET_INVALID_PARAMETERS;
> }

OK, I can use that. I'll have to add a
if (w5)
    return FFA_RET_INVALID_PARAMETERS;

since the rest of the bits must be zero.

>
> > +
> > +    if ( !ffa_page_count )
> > +        return FFA_RET_DENIED;
> > +
> > +    spin_lock(&ctx->lock);
> > +    spin_lock(&ffa_rx_buffer_lock);
> > +    if ( !ctx->page_count || !ctx->tx_is_mine )
>
> If i understand correctly tx_is_mine is protecting the guest rx
> buffer until rx_release is called by the guest so that we do not
> write in it before the guest has retrieved the data from it.
>
> The name is very misleading, maybe rx_is_writeable or free would be better ?

The FF-A specification talks about ownership of the TX buffer (from
the VMs point of view), hence the name.
I'll change it to rx_is_free to be more intuitive without the spec.

>
> Also it would be more optimal to test it before taking ffa_rx_buffer_lock.

Yes, I'll change that.

>
>
> > +        goto out;
> > +    ret = ffa_partition_info_get(w1, w2, w3, w4, w5, count);
> > +    if ( ret )
> > +        goto out;
> > +
> > +    if ( ctx->guest_vers == FFA_VERSION_1_0 )
> > +    {
> > +        size_t n;
> > +        struct ffa_partition_info_1_1 *src = ffa_rx;
> > +        struct ffa_partition_info_1_0 *dst = ctx->rx;
> > +
> > +        if ( ctx->page_count * FFA_PAGE_SIZE < *count * sizeof(*dst) )
> > +        {
> > +            ret = FFA_RET_NO_MEMORY;
> > +            goto out_rx_release;
> > +        }
> > +
> > +        for ( n = 0; n < *count; n++ )
> > +        {
> > +            dst[n].id = src[n].id;
> > +            dst[n].execution_context = src[n].execution_context;
> > +            dst[n].partition_properties = src[n].partition_properties;
> > +        }
> > +    }
> > +    else
> > +    {
> > +        size_t sz = *count * sizeof(struct ffa_partition_info_1_1);
> > +
> > +        if ( ctx->page_count * FFA_PAGE_SIZE < sz )
> > +        {
> > +            ret = FFA_RET_NO_MEMORY;
> > +            goto out_rx_release;
> > +        }
> > +
> > +
> > +        memcpy(ctx->rx, ffa_rx, sz);
> > +    }
> > +    ctx->tx_is_mine = false;
>
> at this point we have no reason to hold ctx->lock

ctx->lock is special, we're never supposed to have contention on that
lock. I believe that we in principle could use spin_trylock() instead
and return FFA_RET_BUSY if it fails, but that might be a bit too much.
The VM is supposed to synchronize calls that use the RXTX buffers. So
unlocking ctx->lock early should give nothing for well-behaving
guests, I'd prefer to keep things simple and unlock in reverse order
if you don't mind. I'll add a comment.

>
> > +out_rx_release:
> > +    ffa_rx_release();
>
> There should be no case where do release without unlocking.
>
> It might be cleaner to have 2 functions ffa_rx_get and ffa_rx_release
>  handling both the lock and the rx_release message.

I'd like to keep ffa_rx_release() as a dumb wrapper. ffa_rx_release()
is also used in init_subscribers()  where we don't use any locks.
ffa_rx_release() is called after a successful call to
ffa_partition_info_get() where we also gained ownership of our RX
buffer. Things might be a bit too intertwined for further abstraction.
I'll add a comment explaining the relationship between
ffa_partition_info_get() and ffa_rx_release().

>
> > +out:
> > +    spin_unlock(&ffa_rx_buffer_lock);
>
> this should stay with ffa_rx_release

Depending on if you accept my explanation above.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > +    spin_unlock(&ctx->lock);
> > +
> > +    return ret;
> > +}
> > +
> > +static uint32_t handle_rx_release(void)
> > +{
> > +    uint32_t ret = FFA_RET_DENIED;
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +
> > +    spin_lock(&ctx->lock);
> > +    if ( !ctx->page_count || ctx->tx_is_mine )
> > +        goto out;
> > +    ret = FFA_RET_OK;
> > +    ctx->tx_is_mine = true;
> > +out:
> > +    spin_unlock(&ctx->lock);
> > +
> > +    return ret;
> > +}
> > +
> > static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> > {
> >     struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> > @@ -528,6 +631,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >     uint32_t fid = get_user_reg(regs, 0);
> >     struct domain *d = current->domain;
> >     struct ffa_ctx *ctx = d->arch.tee;
> > +    uint32_t count;
> >     int e;
> >
> >     if ( !ctx )
> > @@ -559,6 +663,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >         else
> >             set_regs_success(regs, 0, 0);
> >         return true;
> > +    case FFA_PARTITION_INFO_GET:
> > +        e = handle_partition_info_get(get_user_reg(regs, 1),
> > +                                      get_user_reg(regs, 2),
> > +                                      get_user_reg(regs, 3),
> > +                                      get_user_reg(regs, 4),
> > +                                      get_user_reg(regs, 5), &count);
> > +        if ( e )
> > +            set_regs_error(regs, e);
> > +        else
> > +            set_regs_success(regs, count, 0);
> > +        return true;
> > +    case FFA_RX_RELEASE:
> > +        e = handle_rx_release();
> > +        if ( e )
> > +            set_regs_error(regs, e);
> > +        else
> > +            set_regs_success(regs, 0, 0);
> > +        return true;
> >     case FFA_MSG_SEND_DIRECT_REQ_32:
> > #ifdef CONFIG_ARM_64
> >     case FFA_MSG_SEND_DIRECT_REQ_64:
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 15/20] xen/arm: move regpair_to_uint64() and uint64_to_regpair() to regs.h

[Jens Wiklander]

On Fri, Mar 3, 2023 at 11:51 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Moves the two helper functions regpair_to_uint64() and
> > uint64_to_regpair() from xen/arch/arm/tee/optee.c to the common arm
> > specific regs.h. This enables reuse of these functions in the FF-A
> > mediator in a subsequent patch.
> >
> > Reviewed-by: Michal Orzel <michal.orzel@amd.com>
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> Please move Michal R-B after your signed off.

OK

>
> With that fixed:
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > ---
> > xen/arch/arm/include/asm/regs.h | 12 ++++++++++++
> > xen/arch/arm/tee/optee.c        | 11 -----------
> > 2 files changed, 12 insertions(+), 11 deletions(-)
> >
> > diff --git a/xen/arch/arm/include/asm/regs.h b/xen/arch/arm/include/asm/regs.h
> > index 0693a681315f..aa39e83ee5f4 100644
> > --- a/xen/arch/arm/include/asm/regs.h
> > +++ b/xen/arch/arm/include/asm/regs.h
> > @@ -60,6 +60,18 @@ static inline bool guest_mode(const struct cpu_user_regs *r)
> > register_t get_user_reg(struct cpu_user_regs *regs, int reg);
> > void set_user_reg(struct cpu_user_regs *regs, int reg, register_t val);
> >
> > +static inline uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
> > +{
> > +    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
> > +}
> > +
> > +static inline void uint64_to_regpair(register_t *reg0, register_t *reg1,
> > +                                     uint64_t val)
> > +{
> > +    *reg0 = val >> 32;
> > +    *reg1 = (uint32_t)val;
> > +}
> > +
> > #endif
> >
> > #endif /* __ARM_REGS_H__ */
> > diff --git a/xen/arch/arm/tee/optee.c b/xen/arch/arm/tee/optee.c
> > index 9cb9f16d43cb..47027ecef47c 100644
> > --- a/xen/arch/arm/tee/optee.c
> > +++ b/xen/arch/arm/tee/optee.c
> > @@ -268,17 +268,6 @@ static int optee_domain_init(struct domain *d)
> >     return 0;
> > }
> >
> > -static uint64_t regpair_to_uint64(register_t reg0, register_t reg1)
> > -{
> > -    return ((uint64_t)reg0 << 32) | (uint32_t)reg1;
> > -}
> > -
> > -static void uint64_to_regpair(register_t *reg0, register_t *reg1, uint64_t val)
> > -{
> > -    *reg0 = val >> 32;
> > -    *reg1 = (uint32_t)val;
> > -}
> > -
> > static struct page_info *get_domain_ram_page(gfn_t gfn)
> > {
> >     struct page_info *page;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 16/20] xen/arm: ffa: add defines for sharing memory

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds defines needed for sharing using the function FFA_MEM_SHARE and
> friends.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 57 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 57 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 3571817c0bcd..bfd378f7fcd7 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -22,6 +22,14 @@
> #include <asm/tee/ffa.h>
> #include <asm/regs.h>
> 
> +/*
> + * References:
> + * FF-A-1.0-REL: FF-A specification version 1.0 available at
> + *               https://developer.arm.com/documentation/den0077/a
> + * FF-A-1.1-REL0: FF-A specification version 1.1 available at
> + *                https://developer.arm.com/documentation/den0077/e
> + */

This could actually be directly in the file header.

> +
> /* Error codes */
> #define FFA_RET_OK                      0
> #define FFA_RET_NOT_SUPPORTED           -1
> @@ -66,6 +74,55 @@
>  */
> #define FFA_PAGE_SIZE                   SZ_4K
> 
> +/*
> + * Limit for shared buffer size. Please note that this define limits
> + * number of pages. But user buffer can be not aligned to a page
> + * boundary. So it is possible that user would not be able to share
> + * exactly FFA_MAX_SHM_BUFFER_PG * FFA_PAGE_SIZE bytes.
> + *
> + * FF-A doesn't have any direct requirments on GlobalPlatform or vice
> + * versa, but an implementation can very well use FF-A in order to provide
> + * a GlobalPlatform interface on top.
> + *
> + * Global Platform specification for TEE requires that any TEE
> + * implementation should allow to share buffers with size of at least
> + * 512KB. Due to align issue mentioned above, we need to increase this
> + * value with one.

Could you give a bit more details here: what spec version, where in the spec ?

I did download one version of it and at a first glance could not find this value.

> + */
> +#define FFA_MAX_SHM_PAGE_COUNT          (SZ_512K / FFA_PAGE_SIZE + 1)
> +
> +/*
> + * Limits the number of shared buffers that guest can have at once. This
> + * is to prevent case, when guests tricks XEN into exhausting its own
> + * memory by allocating many small buffers. This value has been chosen
> + * arbitrary.
> + */
> +#define FFA_MAX_SHM_COUNT               32

For those 2 limits, I am a bit wondering which should be defined in the code or maybe
configurable in kconfig or through xen command line or per guest.

I do not think we can answer that now but if one of these limits is reached we should
have a clear error message so that we could easily say what to modify and decide
if/how to solve it.


> +
> +/* FF-A-1.1-REL0 section 10.9.2 Memory region handle, page 167 */
> +#define FFA_HANDLE_HYP_FLAG             BIT(63, ULL)
> +#define FFA_HANDLE_INVALID              0xffffffffffffffffULL
> +
> +/*
> + * The bits for FFA_NORMAL_MEM_REG_ATTR FFA_MEM_ACC_RW below are
> + * defined in FF-A-1.1-REL0 Table 10.18 at page 175.
> + */
> + /* Memory attributes: Normal memory, Write-Back cacheable, Inner shareable */
> +#define FFA_NORMAL_MEM_REG_ATTR         0x2fU
> +/* Memory access permissions: Read-write */
> +#define FFA_MEM_ACC_RW                  0x2U

RW bits are defined in table 10.15 at page 168 if i am not mistaking

> +
> +/* FF-A-1.1-REL0 section 10.11.4 Flags usage, page 184-187 */
> +/* Clear memory before mapping in receiver */
> +#define FFA_MEMORY_REGION_FLAG_CLEAR            BIT(0, U)
> +/* Relayer may time slice this operation */
> +#define FFA_MEMORY_REGION_FLAG_TIME_SLICE       BIT(1, U)
> +/* Clear memory after receiver relinquishes it */
> +#define FFA_MEMORY_REGION_FLAG_CLEAR_RELINQUISH BIT(2, U)
> +/* Share memory transaction */
> +#define FFA_MEMORY_REGION_TRANSACTION_TYPE_SHARE (1U << 3)
> +
> +
Please only add one empty line here.

I checked the definitions and they are all coherent with the spec.

Cheers
Bertrand

> /* Framework direct request/response */
> #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> #define FFA_MSG_TYPE_MASK               0xFFU;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 14/20] xen/arm: ffa: support guest FFA_PARTITION_INFO_GET

[Bertrand Marquis]

> On 3 Mar 2023, at 14:17, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Fri, Mar 3, 2023 at 10:51 AM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support in the mediator to handle FFA_PARTITION_INFO_GET requests
>>> from a guest. The requests are forwarded to the SPMC and the response is
>>> translated according to the FF-A version in use by the guest.
>>> 
>>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
>>> caller (the guest in this case), so once it is done with the buffer it
>>> must be released using FFA_RX_RELEASE before another call can be made.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 126 ++++++++++++++++++++++++++++++++++++++++-
>>> 1 file changed, 124 insertions(+), 2 deletions(-)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 953b6dfd5eca..3571817c0bcd 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -141,6 +141,12 @@
>>> #define FFA_MSG_POLL                    0x8400006AU
>>> 
>>> /* Partition information descriptor */
>>> +struct ffa_partition_info_1_0 {
>>> +    uint16_t id;
>>> +    uint16_t execution_context;
>>> +    uint32_t partition_properties;
>>> +};
>>> +
>>> struct ffa_partition_info_1_1 {
>>>    uint16_t id;
>>>    uint16_t execution_context;
>>> @@ -157,9 +163,8 @@ struct ffa_ctx {
>>>    uint32_t guest_vers;
>>>    bool tx_is_mine;
>>>    bool interrupted;
>>> +    spinlock_t lock;
>>> };
>>> -
>>> -
>> 
>> This is removing 2 empty lines (previous patch was wrongly adding one)
>> but one empty line is required here.
> 
> I'll fix it.
> 
>> 
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> @@ -173,10 +178,16 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
>>> * Our rx/tx buffers shared with the SPMC.
>>> *
>>> * ffa_page_count is the number of pages used in each of these buffers.
>>> + *
>>> + * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
>>> + * Note that the SPMC is also tracking the ownership of our RX buffer so
>>> + * for calls which uses our RX buffer to deliver a result we must call
>>> + * ffa_rx_release() to let the SPMC know that we're done with the buffer.
>>> */
>>> static void *ffa_rx __read_mostly;
>>> static void *ffa_tx __read_mostly;
>>> static unsigned int ffa_page_count __read_mostly;
>>> +static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
>>> 
>>> static bool ffa_get_version(uint32_t *vers)
>>> {
>>> @@ -463,6 +474,98 @@ static uint32_t handle_rxtx_unmap(void)
>>>    return FFA_RET_OK;
>>> }
>>> 
>>> +static uint32_t handle_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
>>> +                                          uint32_t w4, uint32_t w5,
>>> +                                          uint32_t *count)
>>> +{
>>> +    bool query_count_only = w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG;
>>> +    uint32_t w5_mask = 0;
>>> +    uint32_t ret = FFA_RET_DENIED;
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +
>>> +    /*
>>> +     * FF-A v1.0 has w5 MBZ while v1.1 allows
>>> +     * FFA_PARTITION_INFO_GET_COUNT_FLAG to be non-zero.
>>> +     */
>>> +    if ( ctx->guest_vers == FFA_VERSION_1_1 )
>>> +        w5_mask = FFA_PARTITION_INFO_GET_COUNT_FLAG;
>>> +    if ( w5 & ~w5_mask )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    if ( query_count_only )
>>> +        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
>> 
>> This code seems a bit to complex.
>> I would suggest the following:
>> 
>> if (w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG)
>> {
>>     if ( ctx->guest_vers == FFA_VERSION_1_1 )
>>        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
>>     else
>>        return FFA_RET_INVALID_PARAMETERS;
>> }
> 
> OK, I can use that. I'll have to add a
> if (w5)
>    return FFA_RET_INVALID_PARAMETERS;
> 
> since the rest of the bits must be zero.

ok

> 
>> 
>>> +
>>> +    if ( !ffa_page_count )
>>> +        return FFA_RET_DENIED;
>>> +
>>> +    spin_lock(&ctx->lock);
>>> +    spin_lock(&ffa_rx_buffer_lock);
>>> +    if ( !ctx->page_count || !ctx->tx_is_mine )
>> 
>> If i understand correctly tx_is_mine is protecting the guest rx
>> buffer until rx_release is called by the guest so that we do not
>> write in it before the guest has retrieved the data from it.
>> 
>> The name is very misleading, maybe rx_is_writeable or free would be better ?
> 
> The FF-A specification talks about ownership of the TX buffer (from
> the VMs point of view), hence the name.
> I'll change it to rx_is_free to be more intuitive without the spec.

Yes but in the code it is quite unclear so better to rename it here.

> 
>> 
>> Also it would be more optimal to test it before taking ffa_rx_buffer_lock.
> 
> Yes, I'll change that.
> 
>> 
>> 
>>> +        goto out;
>>> +    ret = ffa_partition_info_get(w1, w2, w3, w4, w5, count);
>>> +    if ( ret )
>>> +        goto out;
>>> +
>>> +    if ( ctx->guest_vers == FFA_VERSION_1_0 )
>>> +    {
>>> +        size_t n;
>>> +        struct ffa_partition_info_1_1 *src = ffa_rx;
>>> +        struct ffa_partition_info_1_0 *dst = ctx->rx;
>>> +
>>> +        if ( ctx->page_count * FFA_PAGE_SIZE < *count * sizeof(*dst) )
>>> +        {
>>> +            ret = FFA_RET_NO_MEMORY;
>>> +            goto out_rx_release;
>>> +        }
>>> +
>>> +        for ( n = 0; n < *count; n++ )
>>> +        {
>>> +            dst[n].id = src[n].id;
>>> +            dst[n].execution_context = src[n].execution_context;
>>> +            dst[n].partition_properties = src[n].partition_properties;
>>> +        }
>>> +    }
>>> +    else
>>> +    {
>>> +        size_t sz = *count * sizeof(struct ffa_partition_info_1_1);
>>> +
>>> +        if ( ctx->page_count * FFA_PAGE_SIZE < sz )
>>> +        {
>>> +            ret = FFA_RET_NO_MEMORY;
>>> +            goto out_rx_release;
>>> +        }
>>> +
>>> +
>>> +        memcpy(ctx->rx, ffa_rx, sz);
>>> +    }
>>> +    ctx->tx_is_mine = false;
>> 
>> at this point we have no reason to hold ctx->lock
> 
> ctx->lock is special, we're never supposed to have contention on that
> lock. I believe that we in principle could use spin_trylock() instead
> and return FFA_RET_BUSY if it fails, but that might be a bit too much.
> The VM is supposed to synchronize calls that use the RXTX buffers. So
> unlocking ctx->lock early should give nothing for well-behaving
> guests, I'd prefer to keep things simple and unlock in reverse order
> if you don't mind. I'll add a comment.

Please add a comment and a TODO as we have very big locked sections
here and xen is not preemptible so having someone blocked here by an
other core doing something is a concern.

If it is expected that a VM should synchronize calls then we might want to
switch the ctx lock to use trylock and return busy.

> 
>> 
>>> +out_rx_release:
>>> +    ffa_rx_release();
>> 
>> There should be no case where do release without unlocking.
>> 
>> It might be cleaner to have 2 functions ffa_rx_get and ffa_rx_release
>> handling both the lock and the rx_release message.
> 
> I'd like to keep ffa_rx_release() as a dumb wrapper. ffa_rx_release()
> is also used in init_subscribers()  where we don't use any locks.
> ffa_rx_release() is called after a successful call to
> ffa_partition_info_get() where we also gained ownership of our RX
> buffer. Things might be a bit too intertwined for further abstraction.
> I'll add a comment explaining the relationship between
> ffa_partition_info_get() and ffa_rx_release().

That would not create any problem to take the lock where it is not
done already and would make the implemenation a bit more robust.

If at some point some FFA services are used in different cores during
the boot phase, it might make sense to take the lock even if we are in 
situations where there should be no concurrency just to make the code
safer.

Please tell me what you think here.

> 
>> 
>>> +out:
>>> +    spin_unlock(&ffa_rx_buffer_lock);
>> 
>> this should stay with ffa_rx_release
> 
> Depending on if you accept my explanation above.


Cheers
Bertrand

> 
> Thanks,
> Jens
> 
>> 
>> Cheers
>> Bertrand
>> 
>>> +    spin_unlock(&ctx->lock);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> +static uint32_t handle_rx_release(void)
>>> +{
>>> +    uint32_t ret = FFA_RET_DENIED;
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +
>>> +    spin_lock(&ctx->lock);
>>> +    if ( !ctx->page_count || ctx->tx_is_mine )
>>> +        goto out;
>>> +    ret = FFA_RET_OK;
>>> +    ctx->tx_is_mine = true;
>>> +out:
>>> +    spin_unlock(&ctx->lock);
>>> +
>>> +    return ret;
>>> +}
>>> +
>>> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
>>> {
>>>    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
>>> @@ -528,6 +631,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>    uint32_t fid = get_user_reg(regs, 0);
>>>    struct domain *d = current->domain;
>>>    struct ffa_ctx *ctx = d->arch.tee;
>>> +    uint32_t count;
>>>    int e;
>>> 
>>>    if ( !ctx )
>>> @@ -559,6 +663,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>        else
>>>            set_regs_success(regs, 0, 0);
>>>        return true;
>>> +    case FFA_PARTITION_INFO_GET:
>>> +        e = handle_partition_info_get(get_user_reg(regs, 1),
>>> +                                      get_user_reg(regs, 2),
>>> +                                      get_user_reg(regs, 3),
>>> +                                      get_user_reg(regs, 4),
>>> +                                      get_user_reg(regs, 5), &count);
>>> +        if ( e )
>>> +            set_regs_error(regs, e);
>>> +        else
>>> +            set_regs_success(regs, count, 0);
>>> +        return true;
>>> +    case FFA_RX_RELEASE:
>>> +        e = handle_rx_release();
>>> +        if ( e )
>>> +            set_regs_error(regs, e);
>>> +        else
>>> +            set_regs_success(regs, 0, 0);
>>> +        return true;
>>>    case FFA_MSG_SEND_DIRECT_REQ_32:
>>> #ifdef CONFIG_ARM_64
>>>    case FFA_MSG_SEND_DIRECT_REQ_64:
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 17/20] xen/arm: ffa: add ABI structs for sharing memory

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds the ABI structs used by function FFA_MEM_SHARE and friends for
> sharing memory.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>

All the structures are coherent with the spec.

Just one small question after but independent if you choose or not to change the names:
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> xen/arch/arm/tee/ffa.c | 74 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 74 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index bfd378f7fcd7..94c90b252454 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -197,6 +197,11 @@
> #define FFA_MSG_SEND                    0x8400006EU
> #define FFA_MSG_POLL                    0x8400006AU
> 
> +/*
> + * Structs below ending with _1_0 are defined in FF-A-1.0-REL and
> + * struct ending with _1_1 are defined in FF-A-1.1-REL0.
> + */
> +
> /* Partition information descriptor */
> struct ffa_partition_info_1_0 {
>     uint16_t id;
> @@ -211,6 +216,75 @@ struct ffa_partition_info_1_1 {
>     uint8_t uuid[16];
> };
> 
> +/* Constituent memory region descriptor */
> +struct ffa_address_range {
> +    uint64_t address;
> +    uint32_t page_count;
> +    uint32_t reserved;
> +};
> +
> +/* Composite memory region descriptor */
> +struct ffa_mem_region {
> +    uint32_t total_page_count;
> +    uint32_t address_range_count;
> +    uint64_t reserved;
> +    struct ffa_address_range address_range_array[];
> +};
> +
> +/* Memory access permissions descriptor */
> +struct ffa_mem_access_perm {
> +    uint16_t endpoint_id;
> +    uint8_t perm;
> +    uint8_t flags;
> +};
> +
> +/* Endpoint memory access descriptor */
> +struct ffa_mem_access {
> +    struct ffa_mem_access_perm access_perm;
> +    uint32_t region_offs;
> +    uint64_t reserved;
> +};
> +
> +/* Lend, donate or share memory transaction descriptor */
> +struct ffa_mem_transaction_1_0 {
> +    uint16_t sender_id;
> +    uint8_t mem_reg_attr;
> +    uint8_t reserved0;
> +    uint32_t flags;
> +    uint64_t global_handle;

Why global ? spec is just saying handle.

> +    uint64_t tag;
> +    uint32_t reserved1;
> +    uint32_t mem_access_count;
> +    struct ffa_mem_access mem_access_array[];
> +};
> +
> +struct ffa_mem_transaction_1_1 {
> +    uint16_t sender_id;
> +    uint16_t mem_reg_attr;
> +    uint32_t flags;
> +    uint64_t global_handle;

Same here

> +    uint64_t tag;
> +    uint32_t mem_access_size;
> +    uint32_t mem_access_count;
> +    uint32_t mem_access_offs;
> +    uint8_t reserved[12];
> +};
> +
> +/* Endpoint RX/TX descriptor */
> +struct ffa_endpoint_rxtx_descriptor_1_0 {
> +    uint16_t sender_id;
> +    uint16_t reserved;
> +    uint32_t rx_range_count;
> +    uint32_t tx_range_count;
> +};
> +
> +struct ffa_endpoint_rxtx_descriptor_1_1 {
> +    uint16_t sender_id;
> +    uint16_t reserved;
> +    uint32_t rx_region_offs;
> +    uint32_t tx_region_offs;
> +};
> +
> struct ffa_ctx {
>     void *rx;
>     const void *tx;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 14/20] xen/arm: ffa: support guest FFA_PARTITION_INFO_GET

[Jens Wiklander]

Hi,

On Fri, Mar 3, 2023 at 2:50 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
>
>
> > On 3 Mar 2023, at 14:17, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Fri, Mar 3, 2023 at 10:51 AM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support in the mediator to handle FFA_PARTITION_INFO_GET requests
> >>> from a guest. The requests are forwarded to the SPMC and the response is
> >>> translated according to the FF-A version in use by the guest.
> >>>
> >>> Using FFA_PARTITION_INFO_GET changes the owner of the RX buffer to the
> >>> caller (the guest in this case), so once it is done with the buffer it
> >>> must be released using FFA_RX_RELEASE before another call can be made.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 126 ++++++++++++++++++++++++++++++++++++++++-
> >>> 1 file changed, 124 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 953b6dfd5eca..3571817c0bcd 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -141,6 +141,12 @@
> >>> #define FFA_MSG_POLL                    0x8400006AU
> >>>
> >>> /* Partition information descriptor */
> >>> +struct ffa_partition_info_1_0 {
> >>> +    uint16_t id;
> >>> +    uint16_t execution_context;
> >>> +    uint32_t partition_properties;
> >>> +};
> >>> +
> >>> struct ffa_partition_info_1_1 {
> >>>    uint16_t id;
> >>>    uint16_t execution_context;
> >>> @@ -157,9 +163,8 @@ struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>>    bool tx_is_mine;
> >>>    bool interrupted;
> >>> +    spinlock_t lock;
> >>> };
> >>> -
> >>> -
> >>
> >> This is removing 2 empty lines (previous patch was wrongly adding one)
> >> but one empty line is required here.
> >
> > I'll fix it.
> >
> >>
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> @@ -173,10 +178,16 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
> >>> * Our rx/tx buffers shared with the SPMC.
> >>> *
> >>> * ffa_page_count is the number of pages used in each of these buffers.
> >>> + *
> >>> + * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
> >>> + * Note that the SPMC is also tracking the ownership of our RX buffer so
> >>> + * for calls which uses our RX buffer to deliver a result we must call
> >>> + * ffa_rx_release() to let the SPMC know that we're done with the buffer.
> >>> */
> >>> static void *ffa_rx __read_mostly;
> >>> static void *ffa_tx __read_mostly;
> >>> static unsigned int ffa_page_count __read_mostly;
> >>> +static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> >>>
> >>> static bool ffa_get_version(uint32_t *vers)
> >>> {
> >>> @@ -463,6 +474,98 @@ static uint32_t handle_rxtx_unmap(void)
> >>>    return FFA_RET_OK;
> >>> }
> >>>
> >>> +static uint32_t handle_partition_info_get(uint32_t w1, uint32_t w2, uint32_t w3,
> >>> +                                          uint32_t w4, uint32_t w5,
> >>> +                                          uint32_t *count)
> >>> +{
> >>> +    bool query_count_only = w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG;
> >>> +    uint32_t w5_mask = 0;
> >>> +    uint32_t ret = FFA_RET_DENIED;
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +
> >>> +    /*
> >>> +     * FF-A v1.0 has w5 MBZ while v1.1 allows
> >>> +     * FFA_PARTITION_INFO_GET_COUNT_FLAG to be non-zero.
> >>> +     */
> >>> +    if ( ctx->guest_vers == FFA_VERSION_1_1 )
> >>> +        w5_mask = FFA_PARTITION_INFO_GET_COUNT_FLAG;
> >>> +    if ( w5 & ~w5_mask )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    if ( query_count_only )
> >>> +        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
> >>
> >> This code seems a bit to complex.
> >> I would suggest the following:
> >>
> >> if (w5 & FFA_PARTITION_INFO_GET_COUNT_FLAG)
> >> {
> >>     if ( ctx->guest_vers == FFA_VERSION_1_1 )
> >>        return ffa_partition_info_get(w1, w2, w3, w4, w5, count);
> >>     else
> >>        return FFA_RET_INVALID_PARAMETERS;
> >> }
> >
> > OK, I can use that. I'll have to add a
> > if (w5)
> >    return FFA_RET_INVALID_PARAMETERS;
> >
> > since the rest of the bits must be zero.
>
> ok
>
> >
> >>
> >>> +
> >>> +    if ( !ffa_page_count )
> >>> +        return FFA_RET_DENIED;
> >>> +
> >>> +    spin_lock(&ctx->lock);
> >>> +    spin_lock(&ffa_rx_buffer_lock);
> >>> +    if ( !ctx->page_count || !ctx->tx_is_mine )
> >>
> >> If i understand correctly tx_is_mine is protecting the guest rx
> >> buffer until rx_release is called by the guest so that we do not
> >> write in it before the guest has retrieved the data from it.
> >>
> >> The name is very misleading, maybe rx_is_writeable or free would be better ?
> >
> > The FF-A specification talks about ownership of the TX buffer (from
> > the VMs point of view), hence the name.
> > I'll change it to rx_is_free to be more intuitive without the spec.
>
> Yes but in the code it is quite unclear so better to rename it here.
>
> >
> >>
> >> Also it would be more optimal to test it before taking ffa_rx_buffer_lock.
> >
> > Yes, I'll change that.
> >
> >>
> >>
> >>> +        goto out;
> >>> +    ret = ffa_partition_info_get(w1, w2, w3, w4, w5, count);
> >>> +    if ( ret )
> >>> +        goto out;
> >>> +
> >>> +    if ( ctx->guest_vers == FFA_VERSION_1_0 )
> >>> +    {
> >>> +        size_t n;
> >>> +        struct ffa_partition_info_1_1 *src = ffa_rx;
> >>> +        struct ffa_partition_info_1_0 *dst = ctx->rx;
> >>> +
> >>> +        if ( ctx->page_count * FFA_PAGE_SIZE < *count * sizeof(*dst) )
> >>> +        {
> >>> +            ret = FFA_RET_NO_MEMORY;
> >>> +            goto out_rx_release;
> >>> +        }
> >>> +
> >>> +        for ( n = 0; n < *count; n++ )
> >>> +        {
> >>> +            dst[n].id = src[n].id;
> >>> +            dst[n].execution_context = src[n].execution_context;
> >>> +            dst[n].partition_properties = src[n].partition_properties;
> >>> +        }
> >>> +    }
> >>> +    else
> >>> +    {
> >>> +        size_t sz = *count * sizeof(struct ffa_partition_info_1_1);
> >>> +
> >>> +        if ( ctx->page_count * FFA_PAGE_SIZE < sz )
> >>> +        {
> >>> +            ret = FFA_RET_NO_MEMORY;
> >>> +            goto out_rx_release;
> >>> +        }
> >>> +
> >>> +
> >>> +        memcpy(ctx->rx, ffa_rx, sz);
> >>> +    }
> >>> +    ctx->tx_is_mine = false;
> >>
> >> at this point we have no reason to hold ctx->lock
> >
> > ctx->lock is special, we're never supposed to have contention on that
> > lock. I believe that we in principle could use spin_trylock() instead
> > and return FFA_RET_BUSY if it fails, but that might be a bit too much.
> > The VM is supposed to synchronize calls that use the RXTX buffers. So
> > unlocking ctx->lock early should give nothing for well-behaving
> > guests, I'd prefer to keep things simple and unlock in reverse order
> > if you don't mind. I'll add a comment.
>
> Please add a comment and a TODO as we have very big locked sections
> here and xen is not preemptible so having someone blocked here by an
> other core doing something is a concern.
>
> If it is expected that a VM should synchronize calls then we might want to
> switch the ctx lock to use trylock and return busy.

I'll try that. In fact, it should be possible to do the same with the
ffa_rx_buffer_lock at least here in handle_partition_info_get(). In a
later patch where share_shm() is introduced, it might only be possible
with trylock if the guest uses the FFA_MEMORY_REGION_FLAG_TIME_SLICE
flag.

>
> >
> >>
> >>> +out_rx_release:
> >>> +    ffa_rx_release();
> >>
> >> There should be no case where do release without unlocking.
> >>
> >> It might be cleaner to have 2 functions ffa_rx_get and ffa_rx_release
> >> handling both the lock and the rx_release message.
> >
> > I'd like to keep ffa_rx_release() as a dumb wrapper. ffa_rx_release()
> > is also used in init_subscribers()  where we don't use any locks.
> > ffa_rx_release() is called after a successful call to
> > ffa_partition_info_get() where we also gained ownership of our RX
> > buffer. Things might be a bit too intertwined for further abstraction.
> > I'll add a comment explaining the relationship between
> > ffa_partition_info_get() and ffa_rx_release().
>
> That would not create any problem to take the lock where it is not
> done already and would make the implemenation a bit more robust.
>
> If at some point some FFA services are used in different cores during
> the boot phase, it might make sense to take the lock even if we are in
> situations where there should be no concurrency just to make the code
> safer.
>
> Please tell me what you think here.

I agree that some extra locking would be harmless, but there's a
problem that ffa_rx_release() should only be called if we in fact have
ownership of the RX buffer. We must if for instance
ffa_partition_info_get() returns an error not call ffa_rx_release().

I think there is some value in only managing the locks in the
handle_*() function since that makes it easier to see the locking
order etc. That said, I wouldn't mind doing some locking in a helper
function if it came naturally, but so far it hasn't in my opinion.

Cheers,
Jens

>
> >
> >>
> >>> +out:
> >>> +    spin_unlock(&ffa_rx_buffer_lock);
> >>
> >> this should stay with ffa_rx_release
> >
> > Depending on if you accept my explanation above.
>
>
> Cheers
> Bertrand
>
> >
> > Thanks,
> > Jens
> >
> >>
> >> Cheers
> >> Bertrand
> >>
> >>> +    spin_unlock(&ctx->lock);
> >>> +
> >>> +    return ret;
> >>> +}
> >>> +
> >>> +static uint32_t handle_rx_release(void)
> >>> +{
> >>> +    uint32_t ret = FFA_RET_DENIED;
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +
> >>> +    spin_lock(&ctx->lock);
> >>> +    if ( !ctx->page_count || ctx->tx_is_mine )
> >>> +        goto out;
> >>> +    ret = FFA_RET_OK;
> >>> +    ctx->tx_is_mine = true;
> >>> +out:
> >>> +    spin_unlock(&ctx->lock);
> >>> +
> >>> +    return ret;
> >>> +}
> >>> +
> >>> static void handle_msg_send_direct_req(struct cpu_user_regs *regs, uint32_t fid)
> >>> {
> >>>    struct arm_smccc_1_2_regs arg = { .a0 = fid, };
> >>> @@ -528,6 +631,7 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>    uint32_t fid = get_user_reg(regs, 0);
> >>>    struct domain *d = current->domain;
> >>>    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    uint32_t count;
> >>>    int e;
> >>>
> >>>    if ( !ctx )
> >>> @@ -559,6 +663,24 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>        else
> >>>            set_regs_success(regs, 0, 0);
> >>>        return true;
> >>> +    case FFA_PARTITION_INFO_GET:
> >>> +        e = handle_partition_info_get(get_user_reg(regs, 1),
> >>> +                                      get_user_reg(regs, 2),
> >>> +                                      get_user_reg(regs, 3),
> >>> +                                      get_user_reg(regs, 4),
> >>> +                                      get_user_reg(regs, 5), &count);
> >>> +        if ( e )
> >>> +            set_regs_error(regs, e);
> >>> +        else
> >>> +            set_regs_success(regs, count, 0);
> >>> +        return true;
> >>> +    case FFA_RX_RELEASE:
> >>> +        e = handle_rx_release();
> >>> +        if ( e )
> >>> +            set_regs_error(regs, e);
> >>> +        else
> >>> +            set_regs_success(regs, 0, 0);
> >>> +        return true;
> >>>    case FFA_MSG_SEND_DIRECT_REQ_32:
> >>> #ifdef CONFIG_ARM_64
> >>>    case FFA_MSG_SEND_DIRECT_REQ_64:
> >>> --
> >>> 2.34.1
>
>

Re: [XEN PATCH v7 16/20] xen/arm: ffa: add defines for sharing memory

[Jens Wiklander]

Hi Bertrand,

On Fri, Mar 3, 2023 at 2:38 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds defines needed for sharing using the function FFA_MEM_SHARE and
> > friends.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 57 ++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 57 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 3571817c0bcd..bfd378f7fcd7 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -22,6 +22,14 @@
> > #include <asm/tee/ffa.h>
> > #include <asm/regs.h>
> >
> > +/*
> > + * References:
> > + * FF-A-1.0-REL: FF-A specification version 1.0 available at
> > + *               https://developer.arm.com/documentation/den0077/a
> > + * FF-A-1.1-REL0: FF-A specification version 1.1 available at
> > + *                https://developer.arm.com/documentation/den0077/e
> > + */
>
> This could actually be directly in the file header.

OK, I'll move it.

>
> > +
> > /* Error codes */
> > #define FFA_RET_OK                      0
> > #define FFA_RET_NOT_SUPPORTED           -1
> > @@ -66,6 +74,55 @@
> >  */
> > #define FFA_PAGE_SIZE                   SZ_4K
> >
> > +/*
> > + * Limit for shared buffer size. Please note that this define limits
> > + * number of pages. But user buffer can be not aligned to a page
> > + * boundary. So it is possible that user would not be able to share
> > + * exactly FFA_MAX_SHM_BUFFER_PG * FFA_PAGE_SIZE bytes.
> > + *
> > + * FF-A doesn't have any direct requirments on GlobalPlatform or vice
> > + * versa, but an implementation can very well use FF-A in order to provide
> > + * a GlobalPlatform interface on top.
> > + *
> > + * Global Platform specification for TEE requires that any TEE
> > + * implementation should allow to share buffers with size of at least
> > + * 512KB. Due to align issue mentioned above, we need to increase this
> > + * value with one.
>
> Could you give a bit more details here: what spec version, where in the spec ?
>
> I did download one version of it and at a first glance could not find this value.

https://globalplatform.org/specs-library/tee-client-api-specification/
TEE Client API Specification version 1.0c, page 24, Table 4-1: API
Configuration Constants, look for TEEC_CONFIG_SHAREDMEM_MAX_SIZE. I'll
add a reference.

>
> > + */
> > +#define FFA_MAX_SHM_PAGE_COUNT          (SZ_512K / FFA_PAGE_SIZE + 1)
> > +
> > +/*
> > + * Limits the number of shared buffers that guest can have at once. This
> > + * is to prevent case, when guests tricks XEN into exhausting its own
> > + * memory by allocating many small buffers. This value has been chosen
> > + * arbitrary.
> > + */
> > +#define FFA_MAX_SHM_COUNT               32
>
> For those 2 limits, I am a bit wondering which should be defined in the code or maybe
> configurable in kconfig or through xen command line or per guest.

Per guest sounds like an interesting option when these values aren't
good enough.

>
> I do not think we can answer that now but if one of these limits is reached we should
> have a clear error message so that we could easily say what to modify and decide
> if/how to solve it.

I'll add prints in the later patches.

>
>
> > +
> > +/* FF-A-1.1-REL0 section 10.9.2 Memory region handle, page 167 */
> > +#define FFA_HANDLE_HYP_FLAG             BIT(63, ULL)
> > +#define FFA_HANDLE_INVALID              0xffffffffffffffffULL
> > +
> > +/*
> > + * The bits for FFA_NORMAL_MEM_REG_ATTR FFA_MEM_ACC_RW below are
> > + * defined in FF-A-1.1-REL0 Table 10.18 at page 175.
> > + */
> > + /* Memory attributes: Normal memory, Write-Back cacheable, Inner shareable */
> > +#define FFA_NORMAL_MEM_REG_ATTR         0x2fU
> > +/* Memory access permissions: Read-write */
> > +#define FFA_MEM_ACC_RW                  0x2U
>
> RW bits are defined in table 10.15 at page 168 if i am not mistaking

Thanks, I'll update it.

>
> > +
> > +/* FF-A-1.1-REL0 section 10.11.4 Flags usage, page 184-187 */
> > +/* Clear memory before mapping in receiver */
> > +#define FFA_MEMORY_REGION_FLAG_CLEAR            BIT(0, U)
> > +/* Relayer may time slice this operation */
> > +#define FFA_MEMORY_REGION_FLAG_TIME_SLICE       BIT(1, U)
> > +/* Clear memory after receiver relinquishes it */
> > +#define FFA_MEMORY_REGION_FLAG_CLEAR_RELINQUISH BIT(2, U)
> > +/* Share memory transaction */
> > +#define FFA_MEMORY_REGION_TRANSACTION_TYPE_SHARE (1U << 3)
> > +
> > +
> Please only add one empty line here.

OK, I'll fix.

>
> I checked the definitions and they are all coherent with the spec.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > /* Framework direct request/response */
> > #define FFA_MSG_FLAG_FRAMEWORK          BIT(31, U)
> > #define FFA_MSG_TYPE_MASK               0xFFU;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 17/20] xen/arm: ffa: add ABI structs for sharing memory

[Jens Wiklander]

Hi Bertrand,

On Fri, Mar 3, 2023 at 3:20 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds the ABI structs used by function FFA_MEM_SHARE and friends for
> > sharing memory.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>
> All the structures are coherent with the spec.

Thanks for double-checking.

>
> Just one small question after but independent if you choose or not to change the names:
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
>
> Cheers
> Bertrand
>
> > ---
> > xen/arch/arm/tee/ffa.c | 74 ++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 74 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index bfd378f7fcd7..94c90b252454 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -197,6 +197,11 @@
> > #define FFA_MSG_SEND                    0x8400006EU
> > #define FFA_MSG_POLL                    0x8400006AU
> >
> > +/*
> > + * Structs below ending with _1_0 are defined in FF-A-1.0-REL and
> > + * struct ending with _1_1 are defined in FF-A-1.1-REL0.
> > + */
> > +
> > /* Partition information descriptor */
> > struct ffa_partition_info_1_0 {
> >     uint16_t id;
> > @@ -211,6 +216,75 @@ struct ffa_partition_info_1_1 {
> >     uint8_t uuid[16];
> > };
> >
> > +/* Constituent memory region descriptor */
> > +struct ffa_address_range {
> > +    uint64_t address;
> > +    uint32_t page_count;
> > +    uint32_t reserved;
> > +};
> > +
> > +/* Composite memory region descriptor */
> > +struct ffa_mem_region {
> > +    uint32_t total_page_count;
> > +    uint32_t address_range_count;
> > +    uint64_t reserved;
> > +    struct ffa_address_range address_range_array[];
> > +};
> > +
> > +/* Memory access permissions descriptor */
> > +struct ffa_mem_access_perm {
> > +    uint16_t endpoint_id;
> > +    uint8_t perm;
> > +    uint8_t flags;
> > +};
> > +
> > +/* Endpoint memory access descriptor */
> > +struct ffa_mem_access {
> > +    struct ffa_mem_access_perm access_perm;
> > +    uint32_t region_offs;
> > +    uint64_t reserved;
> > +};
> > +
> > +/* Lend, donate or share memory transaction descriptor */
> > +struct ffa_mem_transaction_1_0 {
> > +    uint16_t sender_id;
> > +    uint8_t mem_reg_attr;
> > +    uint8_t reserved0;
> > +    uint32_t flags;
> > +    uint64_t global_handle;
>
> Why global ? spec is just saying handle.
>
> > +    uint64_t tag;
> > +    uint32_t reserved1;
> > +    uint32_t mem_access_count;
> > +    struct ffa_mem_access mem_access_array[];
> > +};
> > +
> > +struct ffa_mem_transaction_1_1 {
> > +    uint16_t sender_id;
> > +    uint16_t mem_reg_attr;
> > +    uint32_t flags;
> > +    uint64_t global_handle;
>
> Same here

I'll change it.

Cheers,
Jens

>
> > +    uint64_t tag;
> > +    uint32_t mem_access_size;
> > +    uint32_t mem_access_count;
> > +    uint32_t mem_access_offs;
> > +    uint8_t reserved[12];
> > +};
> > +
> > +/* Endpoint RX/TX descriptor */
> > +struct ffa_endpoint_rxtx_descriptor_1_0 {
> > +    uint16_t sender_id;
> > +    uint16_t reserved;
> > +    uint32_t rx_range_count;
> > +    uint32_t tx_range_count;
> > +};
> > +
> > +struct ffa_endpoint_rxtx_descriptor_1_1 {
> > +    uint16_t sender_id;
> > +    uint16_t reserved;
> > +    uint32_t rx_region_offs;
> > +    uint32_t tx_region_offs;
> > +};
> > +
> > struct ffa_ctx {
> >     void *rx;
> >     const void *tx;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 18/20] xen/arm: ffa: support sharing memory

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support for a guest to share memory with an SP using FFA_MEM_SHARE
> and FFA_MEM_RECLAIM. Only small memory regions can be shared using a
> single call to FFA_MEM_SHARE are supported.

This sentence needs a bit of rephrasing and to add more details: what is "small".

> 
> A memory region that doesn't need to be shared any longer can be
> reclaimed with FFA_MEM_RECLAIM once the SP doesn't use it any longer.
> This is checked by the SPMC and not in control of the mediator.

This explanation would make more sense in the following patch adding support
 for Reclaim.

> 
> With this commit we have a FF-A version 1.1 [1] mediator able to
> communicate with a Secure Partition in secure world using shared memory.
> The secure world must use FF-A version 1.1, but the guest is free to use
> version 1.0 or version 1.1.

I do not see anything limiting that in the code.
During init we accept 1.0 or 1.1 versions of the secure world.

> 
> Adds a check that the SP supports the needed FF-A features
> FFA_MEM_SHARE_64 or FFA_MEM_SHARE_32.
> 
> [1] https://developer.arm.com/documentation/den0077/latest
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 491 +++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 491 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 94c90b252454..cdc286caf62c 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -270,6 +270,38 @@ struct ffa_mem_transaction_1_1 {
>     uint8_t reserved[12];
> };
> 
> +/* Calculate offset of struct ffa_mem_access from start of buffer */
> +#define MEM_ACCESS_OFFSET(access_idx) \
> +    ( sizeof(struct ffa_mem_transaction_1_1) + \
> +      ( access_idx ) * sizeof(struct ffa_mem_access) )
> +
> +/* Calculate offset of struct ffa_mem_region from start of buffer */
> +#define REGION_OFFSET(access_count, region_idx) \
> +    ( MEM_ACCESS_OFFSET(access_count) + \
> +      ( region_idx ) * sizeof(struct ffa_mem_region) )
> +
> +/* Calculate offset of struct ffa_address_range from start of buffer */
> +#define ADDR_RANGE_OFFSET(access_count, region_count, range_idx) \
> +    ( REGION_OFFSET(access_count, region_count) + \
> +      ( range_idx ) * sizeof(struct ffa_address_range) )
> +
> +/*
> + * The parts needed from struct ffa_mem_transaction_1_0 or struct
> + * ffa_mem_transaction_1_1, used to provide an abstraction of difference in
> + * data structures between version 1.0 and 1.1. This is just an internal
> + * interface and can be changed without changing any ABI.
> + */
> +struct ffa_mem_transaction_x {

I would suggest to s/_x/_int/ in the name here

> +    uint16_t sender_id;
> +    uint8_t mem_reg_attr;
> +    uint8_t flags;
> +    uint8_t mem_access_size;
> +    uint8_t mem_access_count;
> +    uint16_t mem_access_offs;
> +    uint64_t global_handle;
> +    uint64_t tag;
> +};
> +
> /* Endpoint RX/TX descriptor */
> struct ffa_endpoint_rxtx_descriptor_1_0 {
>     uint16_t sender_id;
> @@ -294,8 +326,20 @@ struct ffa_ctx {
>     uint32_t guest_vers;
>     bool tx_is_mine;
>     bool interrupted;
> +    struct list_head shm_list;
> +    unsigned int shm_count;
>     spinlock_t lock;
> };
> +
> +struct ffa_shm_mem {
> +    struct list_head list;
> +    uint16_t sender_id;
> +    uint16_t ep_id;     /* endpoint, the one lending */
> +    uint64_t handle;    /* FFA_HANDLE_INVALID if not set yet */
> +    unsigned int page_count;
> +    struct page_info *pages[];
> +};
> +
> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> @@ -310,6 +354,8 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
>  *
>  * ffa_page_count is the number of pages used in each of these buffers.
>  *
> + * The TX buffer is protected from concurrent usage with ffa_tx_buffer_lock.
> + *
>  * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
>  * Note that the SPMC is also tracking the ownership of our RX buffer so
>  * for calls which uses our RX buffer to deliver a result we must call
> @@ -319,6 +365,7 @@ static void *ffa_rx __read_mostly;
> static void *ffa_tx __read_mostly;
> static unsigned int ffa_page_count __read_mostly;
> static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> +static DEFINE_SPINLOCK(ffa_tx_buffer_lock);
> 
> static bool ffa_get_version(uint32_t *vers)
> {
> @@ -429,6 +476,42 @@ static int32_t ffa_rx_release(void)
>     return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> }
> 
> +static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
> +                             register_t addr, uint32_t pg_count,
> +                             uint64_t *handle)
> +{
> +    struct arm_smccc_1_2_regs arg = {
> +        .a0 = FFA_MEM_SHARE_32,
> +        .a1 = tot_len,
> +        .a2 = frag_len,
> +        .a3 = addr,
> +        .a4 = pg_count,
> +    };
> +    struct arm_smccc_1_2_regs resp;
> +
> +    if ( IS_ENABLED(CONFIG_ARM_64) )
> +        arg.a0 = FFA_MEM_SHARE_64;
> +
> +    arm_smccc_1_2_smc(&arg, &resp);
> +
> +    switch ( resp.a0 )
> +    {
> +    case FFA_ERROR:
> +        if ( resp.a2 )
> +            return resp.a2;
> +        else
> +            return FFA_RET_NOT_SUPPORTED;
> +    case FFA_SUCCESS_32:
> +        *handle = regpair_to_uint64(resp.a3, resp.a2);
> +        return FFA_RET_OK;
> +    case FFA_MEM_FRAG_RX:
> +        *handle = regpair_to_uint64(resp.a2, resp.a1);
> +        return resp.a3;

You are using an int32_t type to cast something that is uint32_t from the spec
and here could in fact be a uint64_t.


> +    default:
> +        return FFA_RET_NOT_SUPPORTED;
> +    }
> +}
> +
> static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>                                       uint8_t msg)
> {
> @@ -757,6 +840,404 @@ out:
>              resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> }
> 
> +/*
> + * Gets all page and assigns them to the supplied shared memory object. If
> + * this function fails then the caller is still expected to call
> + * put_shm_pages() as a cleanup.
> + */
> +static int get_shm_pages(struct domain *d, struct ffa_shm_mem *shm,
> +                         const struct ffa_address_range *range,
> +                         uint32_t range_count, unsigned int start_page_idx,
> +                         unsigned int *last_page_idx)
> +{
> +    unsigned int pg_idx = start_page_idx;
> +    gfn_t gfn;
> +    unsigned int n;
> +    unsigned int m;
> +    p2m_type_t t;
> +    uint64_t addr;
> +
> +    for ( n = 0; n < range_count; n++ )
> +    {
> +        for ( m = 0; m < range[n].page_count; m++ )
> +        {
> +            if ( pg_idx >= shm->page_count )
> +                return FFA_RET_INVALID_PARAMETERS;
> +
> +            addr = read_atomic(&range[n].address);
> +            gfn = gaddr_to_gfn(addr + m * FFA_PAGE_SIZE);
> +            shm->pages[pg_idx] = get_page_from_gfn(d, gfn_x(gfn), &t,
> +   P2M_ALLOC);
> +            if ( !shm->pages[pg_idx] )
> +                return FFA_RET_DENIED;
> +            pg_idx++;

This increment could be done at the end, why here ?

> +            /* Only normal RAM for now */
> +            if ( !p2m_is_ram(t) )
> +                return FFA_RET_DENIED;
> +        }
> +    }
> +
> +    *last_page_idx = pg_idx;
> +
> +    return FFA_RET_OK;
> +}
> +
> +static void put_shm_pages(struct ffa_shm_mem *shm)
> +{
> +    unsigned int n;
> +
> +    for ( n = 0; n < shm->page_count && shm->pages[n]; n++ )
> +    {
> +        put_page(shm->pages[n]);
> +        shm->pages[n] = NULL;

If an error occured during the generation you might have part
of the pages which are NULL already.

So you should do a if (pages[n] != NULL) here

> +    }
> +}
> +
> +static struct ffa_shm_mem *alloc_ffa_shm_mem(struct ffa_ctx *ctx,
> +                                             unsigned int page_count)
> +{
> +    struct ffa_shm_mem *shm;
> +
> +    if ( page_count >= FFA_MAX_SHM_PAGE_COUNT ||
> +         ctx->shm_count >= FFA_MAX_SHM_COUNT )
> +        return NULL;

Shouldn't you also filter out for page_count = 0 ?

> +
> +    shm = xzalloc_flex_struct(struct ffa_shm_mem, pages, page_count);
> +    if ( shm )
> +    {
> +        ctx->shm_count++;
> +        shm->page_count = page_count;
> +    }
> +
> +    return shm;
> +}
> +
> +static void free_ffa_shm_mem(struct ffa_ctx *ctx, struct ffa_shm_mem *shm)
> +{
> +    if ( shm ) {
> +        ASSERT(ctx->shm_count > 0);
> +        ctx->shm_count--;
> +        put_shm_pages(shm);
> +        xfree(shm);
> +    }
> +}
> +
> +static void init_range(struct ffa_address_range *addr_range,
> +                       paddr_t pa)
> +{
> +    memset(addr_range, 0, sizeof(*addr_range));
> +    addr_range->address = pa;
> +    addr_range->page_count = 1;
> +}
> +
> +/*
> + * This function uses the ffa_tx buffer to transmit the memory transaction
> + * descriptor. The function depends ffa_tx_buffer_lock to be used to guard
> + * the buffer from concurrent use.
> + */
> +static int share_shm(struct ffa_shm_mem *shm)
> +{
> +    const uint32_t max_frag_len = ffa_page_count * FFA_PAGE_SIZE;
> +    struct ffa_mem_access *mem_access_array;
> +    struct ffa_mem_transaction_1_1 *descr;
> +    struct ffa_address_range *addr_range;
> +    struct ffa_mem_region *region_descr;
> +    const unsigned int region_count = 1;
> +    void *buf = ffa_tx;
> +    uint32_t frag_len;
> +    uint32_t tot_len;
> +    paddr_t last_pa;
> +    unsigned int n;
> +    paddr_t pa;
> +
> +    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
> +    if ( !shm->page_count )
> +    {
> +        ASSERT_UNREACHABLE();
> +        return FFA_RET_INVALID_PARAMETERS;

page_count = 0 should be filtered out before reaching this and this should
only be an assert if you want but no unreachable with a return.

> +    }
> +
> +    descr = buf;
> +    memset(descr, 0, sizeof(*descr));
> +    descr->sender_id = shm->sender_id;
> +    descr->global_handle = shm->handle;
> +    descr->mem_reg_attr = FFA_NORMAL_MEM_REG_ATTR;
> +    descr->mem_access_count = 1;
> +    descr->mem_access_size = sizeof(*mem_access_array);
> +    descr->mem_access_offs = MEM_ACCESS_OFFSET(0);
> +
> +    mem_access_array = buf + descr->mem_access_offs;
> +    memset(mem_access_array, 0, sizeof(*mem_access_array));
> +    mem_access_array[0].access_perm.endpoint_id = shm->ep_id;
> +    mem_access_array[0].access_perm.perm = FFA_MEM_ACC_RW;
> +    mem_access_array[0].region_offs = REGION_OFFSET(descr->mem_access_count, 0);
> +
> +    region_descr = buf + mem_access_array[0].region_offs;
> +    memset(region_descr, 0, sizeof(*region_descr));
> +    region_descr->total_page_count = shm->page_count;
> +
> +    region_descr->address_range_count = 1;
> +    last_pa = page_to_maddr(shm->pages[0]);
> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> +    {
> +        pa = page_to_maddr(shm->pages[n]);
> +        if ( last_pa + FFA_PAGE_SIZE == pa )
> +            continue;
> +        region_descr->address_range_count++;
> +    }
> +
> +    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
> +                                region_descr->address_range_count);
> +    if ( tot_len > max_frag_len )
> +        return FFA_RET_NOT_SUPPORTED;
> +
> +    addr_range = region_descr->address_range_array;
> +    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
> +    last_pa = page_to_maddr(shm->pages[0]);
> +    init_range(addr_range, last_pa);
> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> +    {
> +        pa = page_to_maddr(shm->pages[n]);
> +        if ( last_pa + FFA_PAGE_SIZE == pa )
> +        {
> +            addr_range->page_count++;
> +            continue;
> +        }
> +
> +        frag_len += sizeof(*addr_range);
> +        addr_range++;
> +        init_range(addr_range, pa);
> +    }
> +
> +    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> +}
> +
> +static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> +                                struct ffa_mem_transaction_x *trans)
> +{
> +    uint16_t mem_reg_attr;
> +    uint32_t flags;
> +    uint32_t count;
> +    uint32_t offs;
> +    uint32_t size;
> +
> +    if ( ffa_vers >= FFA_VERSION_1_1 )
> +    {
> +        const struct ffa_mem_transaction_1_1 *descr;
> +
> +        if ( blen < sizeof(*descr) )
> +            return FFA_RET_INVALID_PARAMETERS;
> +
> +        descr = buf;
> +        trans->sender_id = descr->sender_id;
> +        mem_reg_attr = descr->mem_reg_attr;
> +        flags = descr->flags;
> +        trans->global_handle = descr->global_handle;
> +        trans->tag = descr->tag;
> +
> +        count = descr->mem_access_count;
> +        size = descr->mem_access_size;
> +        offs = descr->mem_access_offs;
> +    }
> +    else
> +    {
> +        const struct ffa_mem_transaction_1_0 *descr;
> +
> +        if ( blen < sizeof(*descr) )
> +            return FFA_RET_INVALID_PARAMETERS;
> +
> +        descr = buf;
> +        trans->sender_id = descr->sender_id;
> +        mem_reg_attr = descr->mem_reg_attr;
> +        flags = descr->flags;
> +        trans->global_handle = descr->global_handle;
> +        trans->tag = descr->tag;
> +
> +        count = descr->mem_access_count;
> +        size = sizeof(struct ffa_mem_access);
> +        offs = offsetof(struct ffa_mem_transaction_1_0, mem_access_array);
> +    }
> +    /*
> +     * Make sure that "descr" which is shared with the guest isn't accessed
> +     * again after this point.
> +     */
> +    barrier();

I am not really following the comment here. You accessed the content of descr
before and it is in the rxtx buffer so why is this needed ?

> +
> +    /*
> +     * We're doing a rough check to see that no information is lost when
> +     * tranfering the values into a struct ffa_mem_transaction_x below. The
> +     * fields in struct ffa_mem_transaction_x are wide enough to hold any
> +     * valid value so being out of range means that something is wrong.
> +     */
> +    if ( mem_reg_attr > UINT8_MAX || flags > UINT8_MAX || size > UINT8_MAX ||
> +        count > UINT8_MAX || offs > UINT16_MAX )
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    /* Check that the endpoint memory access descriptor array fits */
> +    if ( size * count + offs > blen )
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    trans->mem_reg_attr = mem_reg_attr;
> +    trans->flags = flags;
> +    trans->mem_access_size = size;
> +    trans->mem_access_count = count;
> +    trans->mem_access_offs = offs;
> +
> +    return 0;
> +}
> +
> +static void handle_mem_share(struct cpu_user_regs *regs)
> +{
> +    uint32_t tot_len = get_user_reg(regs, 1);
> +    uint32_t frag_len = get_user_reg(regs, 2);
> +    uint64_t addr = get_user_reg(regs, 3);
> +    uint32_t page_count = get_user_reg(regs, 4);
> +    const struct ffa_mem_region *region_descr;
> +    const struct ffa_mem_access *mem_access;
> +    struct ffa_mem_transaction_x trans;
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    struct ffa_shm_mem *shm = NULL;
> +    unsigned int last_page_idx = 0;
> +    register_t handle_hi = 0;
> +    register_t handle_lo = 0;
> +    int ret = FFA_RET_DENIED;
> +    uint32_t range_count;
> +    uint32_t region_offs;
> +
> +    /*
> +     * We're only accepting memory transaction descriptors via the rx/tx
> +     * buffer.

Is this a limitation coming fomr the spec or from the implementation ?

> +     */
> +    if ( addr )
> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_set_ret;
> +    }
> +
> +    /* Check that fragment length doesn't exceed total length */
> +    if ( frag_len > tot_len )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out_set_ret;
> +    }
> +
> +    /* We currently only support a single fragment */

It would make sense to add some text at the beginning of the files listing
the current limitations of the implementation.

> +    if ( frag_len != tot_len )
> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_set_ret;
> +    }
> +
> +    spin_lock(&ctx->lock);
> +
> +    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> +        goto out_unlock;
> +
> +    if ( !ffa_page_count )
> +    {
> +        ret = FFA_RET_NO_MEMORY;
> +        goto out_unlock;
> +    }
> +
> +    ret = read_mem_transaction(ctx->guest_vers, ctx->tx, frag_len, &trans);
> +    if ( ret )
> +        goto out_unlock;
> +
> +    if ( trans.mem_reg_attr != FFA_NORMAL_MEM_REG_ATTR )
> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_unlock;
> +    }
> +
> +    /* Only supports sharing it with one SP for now */

Also a limitation to list.

> +    if ( trans.mem_access_count != 1 )
> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_unlock;
> +    }
> +
> +    if ( trans.sender_id != get_vm_id(d) )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out_unlock;
> +    }
> +
> +    /* Check that it fits in the supplied data */
> +    if ( trans.mem_access_offs + trans.mem_access_size > frag_len )
> +        goto out_unlock;
> +

Why are you using atomic operations to access rxtx buffer after here ?

> +    mem_access = ctx->tx + trans.mem_access_offs;
> +    if ( read_atomic(&mem_access->access_perm.perm) != FFA_MEM_ACC_RW )

Also a limitation to list.

> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_unlock;
> +    }
> +
> +    region_offs = read_atomic(&mem_access->region_offs);
> +    if ( sizeof(*region_descr) + region_offs > frag_len )
> +    {
> +        ret = FFA_RET_NOT_SUPPORTED;
> +        goto out_unlock;
> +    }
> +
> +    region_descr = ctx->tx + region_offs;
> +    range_count = read_atomic(&region_descr->address_range_count);
> +    page_count = read_atomic(&region_descr->total_page_count);
> +
> +    shm = alloc_ffa_shm_mem(ctx, page_count);
> +    if ( !shm )
> +    {
> +        ret = FFA_RET_NO_MEMORY;
> +        goto out_unlock;
> +    }
> +    shm->sender_id = trans.sender_id;
> +    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> +
> +    /*
> +     * Check that the Composite memory region descriptor fits.
> +     */
> +    if ( sizeof(*region_descr) + region_offs +
> +         range_count * sizeof(struct ffa_address_range) > frag_len )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out;
> +    }
> +
> +    ret = get_shm_pages(d, shm, region_descr->address_range_array, range_count,
> +                        0, &last_page_idx);
> +    if ( ret )
> +        goto out;
> +    if ( last_page_idx != shm->page_count )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out;
> +    }
> +
> +    /* Note that share_shm() uses our tx buffer */
> +    spin_lock(&ffa_tx_buffer_lock);
> +    ret = share_shm(shm);
> +    spin_unlock(&ffa_tx_buffer_lock);
> +    if ( ret )
> +        goto out;
> +
> +    list_add_tail(&shm->list, &ctx->shm_list);
> +
> +    uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> +
> +out:
> +    if ( ret )
> +        free_ffa_shm_mem(ctx, shm);
> +out_unlock:
> +    spin_unlock(&ctx->lock);
> +
> +out_set_ret:
> +    if ( ret == 0)
> +            set_regs_success(regs, handle_lo, handle_hi);
> +    else
> +            set_regs_error(regs, ret);
> +}
> +
> static bool ffa_handle_call(struct cpu_user_regs *regs)
> {
>     uint32_t fid = get_user_reg(regs, 0);
> @@ -818,6 +1299,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> #endif
>         handle_msg_send_direct_req(regs, fid);
>         return true;
> +    case FFA_MEM_SHARE_32:
> +#ifdef CONFIG_ARM_64
> +    case FFA_MEM_SHARE_64:
> +#endif
> +        handle_mem_share(regs);
> +        return true;
> 
>     default:
>         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> @@ -857,6 +1344,8 @@ static int ffa_domain_init(struct domain *d)
>         }
>     }
> 
> +    INIT_LIST_HEAD(&ctx->shm_list);
> +
>     d->arch.tee = ctx;
> 
>     return 0;
> @@ -1012,11 +1501,13 @@ static bool ffa_probe(void)
>          !check_mandatory_feature(FFA_RX_RELEASE) ||
> #ifdef CONFIG_ARM_64
>          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> +         !check_mandatory_feature(FFA_MEM_SHARE_64) ||
> #endif
> #ifdef CONFIG_ARM_32
>          !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> #endif
>          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> +         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
>          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>         return false;
> 
> -- 
> 2.34.1


Cheers
Bertrand

Re: [XEN PATCH v7 19/20] xen/arm: ffa: add support to reclaim shared memory

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support to reclaim memory previously shared with FFA_MEM_SHARE.
> 
> Adds a check that the SP supports the needed FF-A feature
> FFA_MEM_RECLAIM.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Just one thing: we need to discuss what you do with the feature check because
i would expect to return unsupported if a feature is not supported but instead here
we make it mandatory for all features we have to be supported to have FFA working.
I am not convince that this was the aim of this and we might need to distinguish between
features we do need to do something and features which would just make some
functionalities not available.

I will try to check how other implementations are handling this.

Cheers
Bertrand

> ---
> xen/arch/arm/tee/ffa.c | 53 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 53 insertions(+)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index cdc286caf62c..3557edc455d0 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -512,6 +512,12 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
>     }
> }
> 
> +static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
> +                               uint32_t flags)
> +{
> +    return ffa_simple_call(FFA_MEM_RECLAIM, handle_lo, handle_hi, flags, 0);
> +}
> +
> static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>                                       uint8_t msg)
> {
> @@ -1238,6 +1244,43 @@ out_set_ret:
>             set_regs_error(regs, ret);
> }
> 
> +static int handle_mem_reclaim(uint64_t handle, uint32_t flags)
> +{
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    struct ffa_shm_mem *shm;
> +    register_t handle_hi;
> +    register_t handle_lo;
> +    int ret;
> +
> +    spin_lock(&ctx->lock);
> +    list_for_each_entry(shm, &ctx->shm_list, list)
> +    {
> +        if ( shm->handle == handle )
> +            goto found_it;
> +    }
> +    shm = NULL;
> +    ret = FFA_RET_INVALID_PARAMETERS;
> +    goto out;
> +found_it:
> +
> +    uint64_to_regpair(&handle_hi, &handle_lo, handle);
> +    ret = ffa_mem_reclaim(handle_lo, handle_hi, flags);
> +    if ( ret )
> +    {
> +        shm = NULL;
> +        goto out;
> +    }
> +
> +    list_del(&shm->list);
> +
> +out:
> +    free_ffa_shm_mem(ctx, shm);
> +    spin_unlock(&ctx->lock);
> +
> +    return ret;
> +}
> +
> static bool ffa_handle_call(struct cpu_user_regs *regs)
> {
>     uint32_t fid = get_user_reg(regs, 0);
> @@ -1305,6 +1348,15 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> #endif
>         handle_mem_share(regs);
>         return true;
> +    case FFA_MEM_RECLAIM:
> +        e = handle_mem_reclaim(regpair_to_uint64(get_user_reg(regs, 2),
> +                                                 get_user_reg(regs, 1)),
> +                               get_user_reg(regs, 3));
> +        if ( e )
> +            set_regs_error(regs, e);
> +        else
> +            set_regs_success(regs, 0, 0);
> +        return true;
> 
>     default:
>         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> @@ -1508,6 +1560,7 @@ static bool ffa_probe(void)
> #endif
>          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
>          !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> +         !check_mandatory_feature(FFA_MEM_RECLAIM) ||
>          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>         return false;
> 
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 18/20] xen/arm: ffa: support sharing memory

[Jens Wiklander]

Hi Bertrand,

On Mon, Mar 13, 2023 at 9:49 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds support for a guest to share memory with an SP using FFA_MEM_SHARE
> > and FFA_MEM_RECLAIM. Only small memory regions can be shared using a
> > single call to FFA_MEM_SHARE are supported.
>
> This sentence needs a bit of rephrasing and to add more details: what is "small".

OK, how about "Only memory regions small enough to be shared with a
single call..."

>
> >
> > A memory region that doesn't need to be shared any longer can be
> > reclaimed with FFA_MEM_RECLAIM once the SP doesn't use it any longer.
> > This is checked by the SPMC and not in control of the mediator.
>
> This explanation would make more sense in the following patch adding support
>  for Reclaim.

Quite right, I'll move it to the next patch.

>
> >
> > With this commit we have a FF-A version 1.1 [1] mediator able to
> > communicate with a Secure Partition in secure world using shared memory.
> > The secure world must use FF-A version 1.1, but the guest is free to use
> > version 1.0 or version 1.1.
>
> I do not see anything limiting that in the code.
> During init we accept 1.0 or 1.1 versions of the secure world.

Good catch, I'll update to only accept version 1.1 in the secure world.

>
> >
> > Adds a check that the SP supports the needed FF-A features
> > FFA_MEM_SHARE_64 or FFA_MEM_SHARE_32.
> >
> > [1] https://developer.arm.com/documentation/den0077/latest
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 491 +++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 491 insertions(+)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 94c90b252454..cdc286caf62c 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -270,6 +270,38 @@ struct ffa_mem_transaction_1_1 {
> >     uint8_t reserved[12];
> > };
> >
> > +/* Calculate offset of struct ffa_mem_access from start of buffer */
> > +#define MEM_ACCESS_OFFSET(access_idx) \
> > +    ( sizeof(struct ffa_mem_transaction_1_1) + \
> > +      ( access_idx ) * sizeof(struct ffa_mem_access) )
> > +
> > +/* Calculate offset of struct ffa_mem_region from start of buffer */
> > +#define REGION_OFFSET(access_count, region_idx) \
> > +    ( MEM_ACCESS_OFFSET(access_count) + \
> > +      ( region_idx ) * sizeof(struct ffa_mem_region) )
> > +
> > +/* Calculate offset of struct ffa_address_range from start of buffer */
> > +#define ADDR_RANGE_OFFSET(access_count, region_count, range_idx) \
> > +    ( REGION_OFFSET(access_count, region_count) + \
> > +      ( range_idx ) * sizeof(struct ffa_address_range) )
> > +
> > +/*
> > + * The parts needed from struct ffa_mem_transaction_1_0 or struct
> > + * ffa_mem_transaction_1_1, used to provide an abstraction of difference in
> > + * data structures between version 1.0 and 1.1. This is just an internal
> > + * interface and can be changed without changing any ABI.
> > + */
> > +struct ffa_mem_transaction_x {
>
> I would suggest to s/_x/_int/ in the name here

OK, I'll update

>
> > +    uint16_t sender_id;
> > +    uint8_t mem_reg_attr;
> > +    uint8_t flags;
> > +    uint8_t mem_access_size;
> > +    uint8_t mem_access_count;
> > +    uint16_t mem_access_offs;
> > +    uint64_t global_handle;
> > +    uint64_t tag;
> > +};
> > +
> > /* Endpoint RX/TX descriptor */
> > struct ffa_endpoint_rxtx_descriptor_1_0 {
> >     uint16_t sender_id;
> > @@ -294,8 +326,20 @@ struct ffa_ctx {
> >     uint32_t guest_vers;
> >     bool tx_is_mine;
> >     bool interrupted;
> > +    struct list_head shm_list;
> > +    unsigned int shm_count;
> >     spinlock_t lock;
> > };
> > +
> > +struct ffa_shm_mem {
> > +    struct list_head list;
> > +    uint16_t sender_id;
> > +    uint16_t ep_id;     /* endpoint, the one lending */
> > +    uint64_t handle;    /* FFA_HANDLE_INVALID if not set yet */
> > +    unsigned int page_count;
> > +    struct page_info *pages[];
> > +};
> > +
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > @@ -310,6 +354,8 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
> >  *
> >  * ffa_page_count is the number of pages used in each of these buffers.
> >  *
> > + * The TX buffer is protected from concurrent usage with ffa_tx_buffer_lock.
> > + *
> >  * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
> >  * Note that the SPMC is also tracking the ownership of our RX buffer so
> >  * for calls which uses our RX buffer to deliver a result we must call
> > @@ -319,6 +365,7 @@ static void *ffa_rx __read_mostly;
> > static void *ffa_tx __read_mostly;
> > static unsigned int ffa_page_count __read_mostly;
> > static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> > +static DEFINE_SPINLOCK(ffa_tx_buffer_lock);
> >
> > static bool ffa_get_version(uint32_t *vers)
> > {
> > @@ -429,6 +476,42 @@ static int32_t ffa_rx_release(void)
> >     return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> > }
> >
> > +static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
> > +                             register_t addr, uint32_t pg_count,
> > +                             uint64_t *handle)
> > +{
> > +    struct arm_smccc_1_2_regs arg = {
> > +        .a0 = FFA_MEM_SHARE_32,
> > +        .a1 = tot_len,
> > +        .a2 = frag_len,
> > +        .a3 = addr,
> > +        .a4 = pg_count,
> > +    };
> > +    struct arm_smccc_1_2_regs resp;
> > +
> > +    if ( IS_ENABLED(CONFIG_ARM_64) )
> > +        arg.a0 = FFA_MEM_SHARE_64;
> > +
> > +    arm_smccc_1_2_smc(&arg, &resp);
> > +
> > +    switch ( resp.a0 )
> > +    {
> > +    case FFA_ERROR:
> > +        if ( resp.a2 )
> > +            return resp.a2;
> > +        else
> > +            return FFA_RET_NOT_SUPPORTED;
> > +    case FFA_SUCCESS_32:
> > +        *handle = regpair_to_uint64(resp.a3, resp.a2);
> > +        return FFA_RET_OK;
> > +    case FFA_MEM_FRAG_RX:
> > +        *handle = regpair_to_uint64(resp.a2, resp.a1);
> > +        return resp.a3;
>
> You are using an int32_t type to cast something that is uint32_t from the spec
> and here could in fact be a uint64_t.

In practice only much smaller values can be valid, however, I
understand that that's not your point. What's the best recovery if the
SPMC gives an invalid value for FFA_MEM_FRAG_RX? The SPMC has
allocated a memory-sharing state when it returns FFA_MEM_FRAG_RX. The
specification doesn't say how to remove that state apart from either
completing it successfully or if it's terminated earlier by the SPMC.
One option is to do a FFA_MEM_FRAG_TX call with invalid arguments so
that the SPMC can free up the memory-sharing state. Thoughts?

>
>
> > +    default:
> > +        return FFA_RET_NOT_SUPPORTED;
> > +    }
> > +}
> > +
> > static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
> >                                       uint8_t msg)
> > {
> > @@ -757,6 +840,404 @@ out:
> >              resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> > }
> >
> > +/*
> > + * Gets all page and assigns them to the supplied shared memory object. If
> > + * this function fails then the caller is still expected to call
> > + * put_shm_pages() as a cleanup.
> > + */
> > +static int get_shm_pages(struct domain *d, struct ffa_shm_mem *shm,
> > +                         const struct ffa_address_range *range,
> > +                         uint32_t range_count, unsigned int start_page_idx,
> > +                         unsigned int *last_page_idx)
> > +{
> > +    unsigned int pg_idx = start_page_idx;
> > +    gfn_t gfn;
> > +    unsigned int n;
> > +    unsigned int m;
> > +    p2m_type_t t;
> > +    uint64_t addr;
> > +
> > +    for ( n = 0; n < range_count; n++ )
> > +    {
> > +        for ( m = 0; m < range[n].page_count; m++ )
> > +        {
> > +            if ( pg_idx >= shm->page_count )
> > +                return FFA_RET_INVALID_PARAMETERS;
> > +
> > +            addr = read_atomic(&range[n].address);
> > +            gfn = gaddr_to_gfn(addr + m * FFA_PAGE_SIZE);
> > +            shm->pages[pg_idx] = get_page_from_gfn(d, gfn_x(gfn), &t,
> > +   P2M_ALLOC);
> > +            if ( !shm->pages[pg_idx] )
> > +                return FFA_RET_DENIED;
> > +            pg_idx++;
>
> This increment could be done at the end, why here ?

Do you mean after the p2m_is_ram() check? I'll move it there.

>
> > +            /* Only normal RAM for now */
> > +            if ( !p2m_is_ram(t) )
> > +                return FFA_RET_DENIED;
> > +        }
> > +    }
> > +
> > +    *last_page_idx = pg_idx;
> > +
> > +    return FFA_RET_OK;
> > +}
> > +
> > +static void put_shm_pages(struct ffa_shm_mem *shm)
> > +{
> > +    unsigned int n;
> > +
> > +    for ( n = 0; n < shm->page_count && shm->pages[n]; n++ )
> > +    {
> > +        put_page(shm->pages[n]);
> > +        shm->pages[n] = NULL;
>
> If an error occured during the generation you might have part
> of the pages which are NULL already.
>
> So you should do a if (pages[n] != NULL) here

I'm doing that above in the head of the loop, the loop is terminated
at the first pages[n] == NULL.

>
> > +    }
> > +}
> > +
> > +static struct ffa_shm_mem *alloc_ffa_shm_mem(struct ffa_ctx *ctx,
> > +                                             unsigned int page_count)
> > +{
> > +    struct ffa_shm_mem *shm;
> > +
> > +    if ( page_count >= FFA_MAX_SHM_PAGE_COUNT ||
> > +         ctx->shm_count >= FFA_MAX_SHM_COUNT )
> > +        return NULL;
>
> Shouldn't you also filter out for page_count = 0 ?

Sure, 0 doesn't make sense. But I should probably do it before this
function is called since I suppose we'd like to return something
different from FFA_RET_NO_MEMORY.

>
> > +
> > +    shm = xzalloc_flex_struct(struct ffa_shm_mem, pages, page_count);
> > +    if ( shm )
> > +    {
> > +        ctx->shm_count++;
> > +        shm->page_count = page_count;
> > +    }
> > +
> > +    return shm;
> > +}
> > +
> > +static void free_ffa_shm_mem(struct ffa_ctx *ctx, struct ffa_shm_mem *shm)
> > +{
> > +    if ( shm ) {
> > +        ASSERT(ctx->shm_count > 0);
> > +        ctx->shm_count--;
> > +        put_shm_pages(shm);
> > +        xfree(shm);
> > +    }
> > +}
> > +
> > +static void init_range(struct ffa_address_range *addr_range,
> > +                       paddr_t pa)
> > +{
> > +    memset(addr_range, 0, sizeof(*addr_range));
> > +    addr_range->address = pa;
> > +    addr_range->page_count = 1;
> > +}
> > +
> > +/*
> > + * This function uses the ffa_tx buffer to transmit the memory transaction
> > + * descriptor. The function depends ffa_tx_buffer_lock to be used to guard
> > + * the buffer from concurrent use.
> > + */
> > +static int share_shm(struct ffa_shm_mem *shm)
> > +{
> > +    const uint32_t max_frag_len = ffa_page_count * FFA_PAGE_SIZE;
> > +    struct ffa_mem_access *mem_access_array;
> > +    struct ffa_mem_transaction_1_1 *descr;
> > +    struct ffa_address_range *addr_range;
> > +    struct ffa_mem_region *region_descr;
> > +    const unsigned int region_count = 1;
> > +    void *buf = ffa_tx;
> > +    uint32_t frag_len;
> > +    uint32_t tot_len;
> > +    paddr_t last_pa;
> > +    unsigned int n;
> > +    paddr_t pa;
> > +
> > +    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
> > +    if ( !shm->page_count )
> > +    {
> > +        ASSERT_UNREACHABLE();
> > +        return FFA_RET_INVALID_PARAMETERS;
>
> page_count = 0 should be filtered out before reaching this and this should
> only be an assert if you want but no unreachable with a return.

I'm adding code to filter out page_count = 0. I'm not sure what you
expect here, should I remove the entire check here or what do you
want?

>
> > +    }
> > +
> > +    descr = buf;
> > +    memset(descr, 0, sizeof(*descr));
> > +    descr->sender_id = shm->sender_id;
> > +    descr->global_handle = shm->handle;
> > +    descr->mem_reg_attr = FFA_NORMAL_MEM_REG_ATTR;
> > +    descr->mem_access_count = 1;
> > +    descr->mem_access_size = sizeof(*mem_access_array);
> > +    descr->mem_access_offs = MEM_ACCESS_OFFSET(0);
> > +
> > +    mem_access_array = buf + descr->mem_access_offs;
> > +    memset(mem_access_array, 0, sizeof(*mem_access_array));
> > +    mem_access_array[0].access_perm.endpoint_id = shm->ep_id;
> > +    mem_access_array[0].access_perm.perm = FFA_MEM_ACC_RW;
> > +    mem_access_array[0].region_offs = REGION_OFFSET(descr->mem_access_count, 0);
> > +
> > +    region_descr = buf + mem_access_array[0].region_offs;
> > +    memset(region_descr, 0, sizeof(*region_descr));
> > +    region_descr->total_page_count = shm->page_count;
> > +
> > +    region_descr->address_range_count = 1;
> > +    last_pa = page_to_maddr(shm->pages[0]);
> > +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> > +    {
> > +        pa = page_to_maddr(shm->pages[n]);
> > +        if ( last_pa + FFA_PAGE_SIZE == pa )
> > +            continue;
> > +        region_descr->address_range_count++;
> > +    }
> > +
> > +    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
> > +                                region_descr->address_range_count);
> > +    if ( tot_len > max_frag_len )
> > +        return FFA_RET_NOT_SUPPORTED;
> > +
> > +    addr_range = region_descr->address_range_array;
> > +    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
> > +    last_pa = page_to_maddr(shm->pages[0]);
> > +    init_range(addr_range, last_pa);
> > +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> > +    {
> > +        pa = page_to_maddr(shm->pages[n]);
> > +        if ( last_pa + FFA_PAGE_SIZE == pa )
> > +        {
> > +            addr_range->page_count++;
> > +            continue;
> > +        }
> > +
> > +        frag_len += sizeof(*addr_range);
> > +        addr_range++;
> > +        init_range(addr_range, pa);
> > +    }
> > +
> > +    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> > +}
> > +
> > +static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> > +                                struct ffa_mem_transaction_x *trans)
> > +{
> > +    uint16_t mem_reg_attr;
> > +    uint32_t flags;
> > +    uint32_t count;
> > +    uint32_t offs;
> > +    uint32_t size;
> > +
> > +    if ( ffa_vers >= FFA_VERSION_1_1 )
> > +    {
> > +        const struct ffa_mem_transaction_1_1 *descr;
> > +
> > +        if ( blen < sizeof(*descr) )
> > +            return FFA_RET_INVALID_PARAMETERS;
> > +
> > +        descr = buf;
> > +        trans->sender_id = descr->sender_id;
> > +        mem_reg_attr = descr->mem_reg_attr;
> > +        flags = descr->flags;
> > +        trans->global_handle = descr->global_handle;
> > +        trans->tag = descr->tag;
> > +
> > +        count = descr->mem_access_count;
> > +        size = descr->mem_access_size;
> > +        offs = descr->mem_access_offs;
> > +    }
> > +    else
> > +    {
> > +        const struct ffa_mem_transaction_1_0 *descr;
> > +
> > +        if ( blen < sizeof(*descr) )
> > +            return FFA_RET_INVALID_PARAMETERS;
> > +
> > +        descr = buf;
> > +        trans->sender_id = descr->sender_id;
> > +        mem_reg_attr = descr->mem_reg_attr;
> > +        flags = descr->flags;
> > +        trans->global_handle = descr->global_handle;
> > +        trans->tag = descr->tag;
> > +
> > +        count = descr->mem_access_count;
> > +        size = sizeof(struct ffa_mem_access);
> > +        offs = offsetof(struct ffa_mem_transaction_1_0, mem_access_array);
> > +    }
> > +    /*
> > +     * Make sure that "descr" which is shared with the guest isn't accessed
> > +     * again after this point.
> > +     */
> > +    barrier();
>
> I am not really following the comment here. You accessed the content of descr
> before and it is in the rxtx buffer so why is this needed ?

I'm making sure that the compiler doesn't optimize and reorders the
reads from memory in funny ways, for instance, reading again after the
ifs just below. The RXTX buffer is shared with the guest so it can
potentially be updated concurrently by another CPU.

>
> > +
> > +    /*
> > +     * We're doing a rough check to see that no information is lost when
> > +     * tranfering the values into a struct ffa_mem_transaction_x below. The
> > +     * fields in struct ffa_mem_transaction_x are wide enough to hold any
> > +     * valid value so being out of range means that something is wrong.
> > +     */
> > +    if ( mem_reg_attr > UINT8_MAX || flags > UINT8_MAX || size > UINT8_MAX ||
> > +        count > UINT8_MAX || offs > UINT16_MAX )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    /* Check that the endpoint memory access descriptor array fits */
> > +    if ( size * count + offs > blen )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    trans->mem_reg_attr = mem_reg_attr;
> > +    trans->flags = flags;
> > +    trans->mem_access_size = size;
> > +    trans->mem_access_count = count;
> > +    trans->mem_access_offs = offs;
> > +
> > +    return 0;
> > +}
> > +
> > +static void handle_mem_share(struct cpu_user_regs *regs)
> > +{
> > +    uint32_t tot_len = get_user_reg(regs, 1);
> > +    uint32_t frag_len = get_user_reg(regs, 2);
> > +    uint64_t addr = get_user_reg(regs, 3);
> > +    uint32_t page_count = get_user_reg(regs, 4);
> > +    const struct ffa_mem_region *region_descr;
> > +    const struct ffa_mem_access *mem_access;
> > +    struct ffa_mem_transaction_x trans;
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +    struct ffa_shm_mem *shm = NULL;
> > +    unsigned int last_page_idx = 0;
> > +    register_t handle_hi = 0;
> > +    register_t handle_lo = 0;
> > +    int ret = FFA_RET_DENIED;
> > +    uint32_t range_count;
> > +    uint32_t region_offs;
> > +
> > +    /*
> > +     * We're only accepting memory transaction descriptors via the rx/tx
> > +     * buffer.
>
> Is this a limitation coming fomr the spec or from the implementation ?

This is just a limitation in the implementation.

>
> > +     */
> > +    if ( addr )
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_set_ret;
> > +    }
> > +
> > +    /* Check that fragment length doesn't exceed total length */
> > +    if ( frag_len > tot_len )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out_set_ret;
> > +    }
> > +
> > +    /* We currently only support a single fragment */
>
> It would make sense to add some text at the beginning of the files listing
> the current limitations of the implementation.

That's quite a bit to keep track of, especially since it will change
with each patch. If it's important perhaps we can summarize that in a
final commit instead. By the way, this particular limitation is
removed in a later patch.

>
> > +    if ( frag_len != tot_len )
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_set_ret;
> > +    }
> > +
> > +    spin_lock(&ctx->lock);
> > +
> > +    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> > +        goto out_unlock;
> > +
> > +    if ( !ffa_page_count )
> > +    {
> > +        ret = FFA_RET_NO_MEMORY;
> > +        goto out_unlock;
> > +    }
> > +
> > +    ret = read_mem_transaction(ctx->guest_vers, ctx->tx, frag_len, &trans);
> > +    if ( ret )
> > +        goto out_unlock;
> > +
> > +    if ( trans.mem_reg_attr != FFA_NORMAL_MEM_REG_ATTR )
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_unlock;
> > +    }
> > +
> > +    /* Only supports sharing it with one SP for now */
>
> Also a limitation to list.

OK

>
> > +    if ( trans.mem_access_count != 1 )
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_unlock;
> > +    }
> > +
> > +    if ( trans.sender_id != get_vm_id(d) )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out_unlock;
> > +    }
> > +
> > +    /* Check that it fits in the supplied data */
> > +    if ( trans.mem_access_offs + trans.mem_access_size > frag_len )
> > +        goto out_unlock;
> > +
>
> Why are you using atomic operations to access rxtx buffer after here ?

To limit how the compiler can reorder the reads from memory.

>
> > +    mem_access = ctx->tx + trans.mem_access_offs;
> > +    if ( read_atomic(&mem_access->access_perm.perm) != FFA_MEM_ACC_RW )
>
> Also a limitation to list.

OK

Thanks,
Jens

>
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_unlock;
> > +    }
> > +
> > +    region_offs = read_atomic(&mem_access->region_offs);
> > +    if ( sizeof(*region_descr) + region_offs > frag_len )
> > +    {
> > +        ret = FFA_RET_NOT_SUPPORTED;
> > +        goto out_unlock;
> > +    }
> > +
> > +    region_descr = ctx->tx + region_offs;
> > +    range_count = read_atomic(&region_descr->address_range_count);
> > +    page_count = read_atomic(&region_descr->total_page_count);
> > +
> > +    shm = alloc_ffa_shm_mem(ctx, page_count);
> > +    if ( !shm )
> > +    {
> > +        ret = FFA_RET_NO_MEMORY;
> > +        goto out_unlock;
> > +    }
> > +    shm->sender_id = trans.sender_id;
> > +    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> > +
> > +    /*
> > +     * Check that the Composite memory region descriptor fits.
> > +     */
> > +    if ( sizeof(*region_descr) + region_offs +
> > +         range_count * sizeof(struct ffa_address_range) > frag_len )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out;
> > +    }
> > +
> > +    ret = get_shm_pages(d, shm, region_descr->address_range_array, range_count,
> > +                        0, &last_page_idx);
> > +    if ( ret )
> > +        goto out;
> > +    if ( last_page_idx != shm->page_count )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out;
> > +    }
> > +
> > +    /* Note that share_shm() uses our tx buffer */
> > +    spin_lock(&ffa_tx_buffer_lock);
> > +    ret = share_shm(shm);
> > +    spin_unlock(&ffa_tx_buffer_lock);
> > +    if ( ret )
> > +        goto out;
> > +
> > +    list_add_tail(&shm->list, &ctx->shm_list);
> > +
> > +    uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> > +
> > +out:
> > +    if ( ret )
> > +        free_ffa_shm_mem(ctx, shm);
> > +out_unlock:
> > +    spin_unlock(&ctx->lock);
> > +
> > +out_set_ret:
> > +    if ( ret == 0)
> > +            set_regs_success(regs, handle_lo, handle_hi);
> > +    else
> > +            set_regs_error(regs, ret);
> > +}
> > +
> > static bool ffa_handle_call(struct cpu_user_regs *regs)
> > {
> >     uint32_t fid = get_user_reg(regs, 0);
> > @@ -818,6 +1299,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> > #endif
> >         handle_msg_send_direct_req(regs, fid);
> >         return true;
> > +    case FFA_MEM_SHARE_32:
> > +#ifdef CONFIG_ARM_64
> > +    case FFA_MEM_SHARE_64:
> > +#endif
> > +        handle_mem_share(regs);
> > +        return true;
> >
> >     default:
> >         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> > @@ -857,6 +1344,8 @@ static int ffa_domain_init(struct domain *d)
> >         }
> >     }
> >
> > +    INIT_LIST_HEAD(&ctx->shm_list);
> > +
> >     d->arch.tee = ctx;
> >
> >     return 0;
> > @@ -1012,11 +1501,13 @@ static bool ffa_probe(void)
> >          !check_mandatory_feature(FFA_RX_RELEASE) ||
> > #ifdef CONFIG_ARM_64
> >          !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> > +         !check_mandatory_feature(FFA_MEM_SHARE_64) ||
> > #endif
> > #ifdef CONFIG_ARM_32
> >          !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> > #endif
> >          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> > +         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> >          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >         return false;
> >
> > --
> > 2.34.1
>
>
> Cheers
> Bertrand
>

Re: [XEN PATCH v7 20/20] xen/arm: ffa: support sharing large memory ranges

[Bertrand Marquis]

Hi Jens,

> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Adds support for sharing large memory ranges transmitted in fragments
> using FFA_MEM_FRAG_TX.
> 
> The implementation is the bare minimum to be able to communicate with
> OP-TEE running as an SPMC at S-EL1.
> 
> Adds a check that the SP supports the needed FF-A feature
> FFA_MEM_FRAG_TX.
> 
> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> ---
> xen/arch/arm/tee/ffa.c | 254 ++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 240 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> index 3557edc455d0..72c0249d8cad 100644
> --- a/xen/arch/arm/tee/ffa.c
> +++ b/xen/arch/arm/tee/ffa.c
> @@ -326,6 +326,7 @@ struct ffa_ctx {
>     uint32_t guest_vers;
>     bool tx_is_mine;
>     bool interrupted;
> +    struct list_head frag_list;
>     struct list_head shm_list;
>     unsigned int shm_count;
>     spinlock_t lock;
> @@ -340,6 +341,18 @@ struct ffa_shm_mem {
>     struct page_info *pages[];
> };

We start to have a lot of fields here.
It might be useful to have some quick documentation
in comment here as some names are a bit generic.
For example "frag_list" does not say much.

> 
> +struct mem_frag_state {
> +    struct list_head list;
> +    struct ffa_shm_mem *shm;
> +    uint32_t range_count;
> +    unsigned int current_page_idx;
> +    unsigned int frag_offset;
> +    unsigned int range_offset;
> +    const uint8_t *buf;
> +    unsigned int buf_size;
> +    struct ffa_address_range range;
> +};

same here, at a first glance it is not quite clear why
a fragment needs that much info. Some seem to only
be needed during the transaction but do not need to be saved.

> +
> /* Negotiated FF-A version to use with the SPMC */
> static uint32_t ffa_version __ro_after_init;
> 
> @@ -512,6 +525,36 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
>     }
> }
> 
> +static int32_t ffa_mem_frag_tx(uint64_t handle, uint32_t frag_len,
> +                               uint16_t sender_id)
> +{
> +    struct arm_smccc_1_2_regs arg = {
> +        .a0 = FFA_MEM_FRAG_TX,
> +        .a1 = handle & UINT32_MAX,
> +        .a2 = handle >> 32,
> +        .a3 = frag_len,
> +        .a4 = (uint32_t)sender_id << 16,
> +    };
> +    struct arm_smccc_1_2_regs resp;
> +
> +    arm_smccc_1_2_smc(&arg, &resp);
> +
> +    switch ( resp.a0 )
> +    {
> +    case FFA_ERROR:
> +        if ( resp.a2 )
> +            return resp.a2;
> +        else
> +            return FFA_RET_NOT_SUPPORTED;
> +    case FFA_SUCCESS_32:
> +        return FFA_RET_OK;
> +    case FFA_MEM_FRAG_RX:
> +        return resp.a3;
> +    default:
> +            return FFA_RET_NOT_SUPPORTED;
> +    }
> +}
> +
> static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
>                                uint32_t flags)
> {
> @@ -586,6 +629,14 @@ static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
>     set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
> }
> 
> +static void set_regs_frag_rx(struct cpu_user_regs *regs, uint32_t handle_lo,
> +                             uint32_t handle_hi, uint32_t frag_offset,
> +                             uint16_t sender_id)
> +{
> +    set_regs(regs, FFA_MEM_FRAG_RX, handle_lo, handle_hi, frag_offset,
> +             (uint32_t)sender_id << 16, 0, 0, 0);
> +}
> +
> static void handle_version(struct cpu_user_regs *regs)
> {
>     struct domain *d = current->domain;
> @@ -955,6 +1006,8 @@ static int share_shm(struct ffa_shm_mem *shm)
>     paddr_t last_pa;
>     unsigned int n;
>     paddr_t pa;
> +    bool first;
> +    int ret;
> 
>     ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
>     if ( !shm->page_count )
> @@ -994,13 +1047,23 @@ static int share_shm(struct ffa_shm_mem *shm)
> 
>     tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
>                                 region_descr->address_range_count);
> -    if ( tot_len > max_frag_len )
> -        return FFA_RET_NOT_SUPPORTED;
> 
> +    /*
> +     * Sharing memory with secure world may have to be done with multiple
> +     * calls depending on how many address ranges will be needed. If we're
> +     * sharing physically contiguous memory we will only need one range but
> +     * we will also need to deal with the worst case where all physical
> +     * pages are non-contiguous. For the first batch of address ranges we
> +     * call ffa_mem_share() and for all that follows ffa_mem_frag_tx().
> +     *
> +     * We use frag_len to keep track of how far into the transmit buffer we
> +     * have gone.
> +     */
>     addr_range = region_descr->address_range_array;
>     frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
>     last_pa = page_to_maddr(shm->pages[0]);
>     init_range(addr_range, last_pa);
> +    first = true;
>     for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
>     {
>         pa = page_to_maddr(shm->pages[n]);
> @@ -1010,12 +1073,34 @@ static int share_shm(struct ffa_shm_mem *shm)
>             continue;
>         }
> 
> -        frag_len += sizeof(*addr_range);
> -        addr_range++;
> +        if ( frag_len == max_frag_len )
> +        {
> +            if ( first )
> +            {
> +                ret = ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> +                first = false;
> +            }
> +            else
> +            {
> +                ret = ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> +            }
> +            if ( ret <= 0 )
> +                return ret;
> +            frag_len = sizeof(*addr_range);
> +            addr_range = buf;
> +        }
> +        else
> +        {
> +            frag_len += sizeof(*addr_range);
> +            addr_range++;
> +        }
>         init_range(addr_range, pa);
>     }
> 
> -    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> +    if ( first )
> +        return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> +    else
> +        return ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> }
> 
> static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> @@ -1092,8 +1177,53 @@ static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
>     return 0;
> }
> 
> +static int add_mem_share_frag(struct mem_frag_state *s, unsigned int offs,
> +                              unsigned int frag_len)
> +{
> +    struct domain *d = current->domain;
> +    unsigned int o = offs;
> +    unsigned int l;
> +    int ret;
> +
> +    if ( frag_len < o )
> +        return FFA_RET_INVALID_PARAMETERS;
> +
> +    /* Fill up the first struct ffa_address_range */
> +    l = min_t(unsigned int, frag_len - o, sizeof(s->range) - s->range_offset);
> +    memcpy((uint8_t *)&s->range + s->range_offset, s->buf + o, l);
> +    s->range_offset += l;
> +    o += l;
> +    if ( s->range_offset != sizeof(s->range) )
> +        goto out;
> +    s->range_offset = 0;
> +
> +    while ( true )
> +    {
> +        ret = get_shm_pages(d, s->shm, &s->range, 1, s->current_page_idx,
> +                            &s->current_page_idx);
> +        if ( ret )
> +            return ret;
> +        if ( s->range_count == 1 )
> +            return 0;
> +        s->range_count--;
> +        if ( frag_len - o < sizeof(s->range) )
> +            break;
> +        memcpy(&s->range, s->buf + o, sizeof(s->range));
> +        o += sizeof(s->range);
> +    }
> +
> +    /* Collect any remaining bytes for the next struct ffa_address_range */
> +    s->range_offset = frag_len - o;
> +    memcpy(&s->range, s->buf + o, frag_len - o);
> +out:
> +    s->frag_offset += frag_len;
> +
> +    return s->frag_offset;
> +}
> +
> static void handle_mem_share(struct cpu_user_regs *regs)
> {
> +    static uint64_t next_handle = FFA_HANDLE_HYP_FLAG;
>     uint32_t tot_len = get_user_reg(regs, 1);
>     uint32_t frag_len = get_user_reg(regs, 2);
>     uint64_t addr = get_user_reg(regs, 3);
> @@ -1128,13 +1258,6 @@ static void handle_mem_share(struct cpu_user_regs *regs)
>         goto out_set_ret;
>     }
> 
> -    /* We currently only support a single fragment */
> -    if ( frag_len != tot_len )
> -    {
> -        ret = FFA_RET_NOT_SUPPORTED;
> -        goto out_set_ret;
> -    }
> -
>     spin_lock(&ctx->lock);
> 
>     if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> @@ -1195,11 +1318,41 @@ static void handle_mem_share(struct cpu_user_regs *regs)
>     if ( !shm )
>     {
>         ret = FFA_RET_NO_MEMORY;
> -        goto out_unlock;
> +        goto out;

Why is this changed ?
You still have no shm here so call free_sha_shm does not make sense

>     }
>     shm->sender_id = trans.sender_id;
>     shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> 
> +    if ( frag_len != tot_len )
> +    {
> +        struct mem_frag_state *s = xzalloc(struct mem_frag_state);
> +
> +        if ( !s )
> +        {
> +            ret = FFA_RET_NO_MEMORY;
> +            goto out;
> +        }
> +        s->shm = shm;
> +        s->range_count = range_count;
> +        s->buf = ctx->tx;
> +        s->buf_size = ffa_page_count * FFA_PAGE_SIZE;
> +        ret = add_mem_share_frag(s, sizeof(*region_descr)  + region_offs,
> +                                 frag_len);
> +        if ( ret <= 0 )
> +        {
> +            xfree(s);
> +            if ( ret < 0 )
> +                goto out;
> +        }
> +        else
> +        {
> +            shm->handle = next_handle++;
> +            uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> +            list_add_tail(&s->list, &ctx->frag_list);
> +        }
> +        goto out_unlock;
> +    }
> +
>     /*
>      * Check that the Composite memory region descriptor fits.
>      */
> @@ -1238,7 +1391,75 @@ out_unlock:
>     spin_unlock(&ctx->lock);
> 
> out_set_ret:
> -    if ( ret == 0)
> +    if ( ret > 0 )
> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, trans.sender_id);
> +    else if ( ret == 0)
> +            set_regs_success(regs, handle_lo, handle_hi);
> +    else
> +            set_regs_error(regs, ret);
> +}
> +
> +static struct mem_frag_state *find_frag_state(struct ffa_ctx *ctx,
> +                                              uint64_t handle)
> +{
> +    struct mem_frag_state *s;
> +
> +    list_for_each_entry(s, &ctx->frag_list, list)
> +        if ( s->shm->handle == handle )
> +            return s;
> +
> +    return NULL;
> +}
> +
> +static void handle_mem_frag_tx(struct cpu_user_regs *regs)
> +{
> +    struct domain *d = current->domain;
> +    struct ffa_ctx *ctx = d->arch.tee;
> +    uint32_t frag_len = get_user_reg(regs, 3);
> +    uint32_t handle_lo = get_user_reg(regs, 1);
> +    uint32_t handle_hi = get_user_reg(regs, 2);
> +    uint64_t handle = regpair_to_uint64(handle_hi, handle_lo);
> +    struct mem_frag_state *s;
> +    uint16_t sender_id = 0;
> +    int ret;
> +
> +    spin_lock(&ctx->lock);
> +    s = find_frag_state(ctx, handle);
> +    if ( !s )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out;
> +    }
> +    sender_id = s->shm->sender_id;
> +
> +    if ( frag_len > s->buf_size )
> +    {
> +        ret = FFA_RET_INVALID_PARAMETERS;
> +        goto out;
> +    }
> +
> +    ret = add_mem_share_frag(s, 0, frag_len);
> +    if ( ret == 0 )
> +    {
> +        /* Note that share_shm() uses our tx buffer */
> +        spin_lock(&ffa_tx_buffer_lock);
> +        ret = share_shm(s->shm);
> +        spin_unlock(&ffa_tx_buffer_lock);
> +        if ( ret == 0 )
> +            list_add_tail(&s->shm->list, &ctx->shm_list);
> +        else
> +            free_ffa_shm_mem(ctx, s->shm);
> +    }
> +    else if ( ret < 0 )
> +        free_ffa_shm_mem(ctx, s->shm);


If there is not error the stuff allocated are kept but i do not see
where/when they would be freed or used.
Could you explain why we need to save all those ?

Cheers
Bertrand

> +    list_del(&s->list);
> +    xfree(s);
> +out:
> +    spin_unlock(&ctx->lock);
> +
> +    if ( ret > 0 )
> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, sender_id);
> +    else if ( ret == 0)
>             set_regs_success(regs, handle_lo, handle_hi);
>     else
>             set_regs_error(regs, ret);
> @@ -1357,6 +1578,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>         else
>             set_regs_success(regs, 0, 0);
>         return true;
> +    case FFA_MEM_FRAG_TX:
> +        handle_mem_frag_tx(regs);
> +        return true;
> 
>     default:
>         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> @@ -1396,6 +1620,7 @@ static int ffa_domain_init(struct domain *d)
>         }
>     }
> 
> +    INIT_LIST_HEAD(&ctx->frag_list);
>     INIT_LIST_HEAD(&ctx->shm_list);
> 
>     d->arch.tee = ctx;
> @@ -1560,6 +1785,7 @@ static bool ffa_probe(void)
> #endif
>          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
>          !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> +         !check_mandatory_feature(FFA_MEM_FRAG_TX) ||
>          !check_mandatory_feature(FFA_MEM_RECLAIM) ||
>          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>         return false;
> -- 
> 2.34.1
> 

Re: [XEN PATCH v7 20/20] xen/arm: ffa: support sharing large memory ranges

[Jens Wiklander]

Hi Bertrand,

On Wed, Mar 15, 2023 at 11:13 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Adds support for sharing large memory ranges transmitted in fragments
> > using FFA_MEM_FRAG_TX.
> >
> > The implementation is the bare minimum to be able to communicate with
> > OP-TEE running as an SPMC at S-EL1.
> >
> > Adds a check that the SP supports the needed FF-A feature
> > FFA_MEM_FRAG_TX.
> >
> > Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> > ---
> > xen/arch/arm/tee/ffa.c | 254 ++++++++++++++++++++++++++++++++++++++---
> > 1 file changed, 240 insertions(+), 14 deletions(-)
> >
> > diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> > index 3557edc455d0..72c0249d8cad 100644
> > --- a/xen/arch/arm/tee/ffa.c
> > +++ b/xen/arch/arm/tee/ffa.c
> > @@ -326,6 +326,7 @@ struct ffa_ctx {
> >     uint32_t guest_vers;
> >     bool tx_is_mine;
> >     bool interrupted;
> > +    struct list_head frag_list;
> >     struct list_head shm_list;
> >     unsigned int shm_count;
> >     spinlock_t lock;
> > @@ -340,6 +341,18 @@ struct ffa_shm_mem {
> >     struct page_info *pages[];
> > };
>
> We start to have a lot of fields here.
> It might be useful to have some quick documentation
> in comment here as some names are a bit generic.
> For example "frag_list" does not say much.

I'll add some comments.

>
> >
> > +struct mem_frag_state {
> > +    struct list_head list;
> > +    struct ffa_shm_mem *shm;
> > +    uint32_t range_count;
> > +    unsigned int current_page_idx;
> > +    unsigned int frag_offset;
> > +    unsigned int range_offset;
> > +    const uint8_t *buf;
> > +    unsigned int buf_size;
> > +    struct ffa_address_range range;
> > +};
>
> same here, at a first glance it is not quite clear why
> a fragment needs that much info. Some seem to only
> be needed during the transaction but do not need to be saved.

This struct is only used during the transaction, so this is freed once
the entire memory transaction descriptor has been processed.

>
> > +
> > /* Negotiated FF-A version to use with the SPMC */
> > static uint32_t ffa_version __ro_after_init;
> >
> > @@ -512,6 +525,36 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
> >     }
> > }
> >
> > +static int32_t ffa_mem_frag_tx(uint64_t handle, uint32_t frag_len,
> > +                               uint16_t sender_id)
> > +{
> > +    struct arm_smccc_1_2_regs arg = {
> > +        .a0 = FFA_MEM_FRAG_TX,
> > +        .a1 = handle & UINT32_MAX,
> > +        .a2 = handle >> 32,
> > +        .a3 = frag_len,
> > +        .a4 = (uint32_t)sender_id << 16,
> > +    };
> > +    struct arm_smccc_1_2_regs resp;
> > +
> > +    arm_smccc_1_2_smc(&arg, &resp);
> > +
> > +    switch ( resp.a0 )
> > +    {
> > +    case FFA_ERROR:
> > +        if ( resp.a2 )
> > +            return resp.a2;
> > +        else
> > +            return FFA_RET_NOT_SUPPORTED;
> > +    case FFA_SUCCESS_32:
> > +        return FFA_RET_OK;
> > +    case FFA_MEM_FRAG_RX:
> > +        return resp.a3;
> > +    default:
> > +            return FFA_RET_NOT_SUPPORTED;
> > +    }
> > +}
> > +
> > static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
> >                                uint32_t flags)
> > {
> > @@ -586,6 +629,14 @@ static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> >     set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
> > }
> >
> > +static void set_regs_frag_rx(struct cpu_user_regs *regs, uint32_t handle_lo,
> > +                             uint32_t handle_hi, uint32_t frag_offset,
> > +                             uint16_t sender_id)
> > +{
> > +    set_regs(regs, FFA_MEM_FRAG_RX, handle_lo, handle_hi, frag_offset,
> > +             (uint32_t)sender_id << 16, 0, 0, 0);
> > +}
> > +
> > static void handle_version(struct cpu_user_regs *regs)
> > {
> >     struct domain *d = current->domain;
> > @@ -955,6 +1006,8 @@ static int share_shm(struct ffa_shm_mem *shm)
> >     paddr_t last_pa;
> >     unsigned int n;
> >     paddr_t pa;
> > +    bool first;
> > +    int ret;
> >
> >     ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
> >     if ( !shm->page_count )
> > @@ -994,13 +1047,23 @@ static int share_shm(struct ffa_shm_mem *shm)
> >
> >     tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
> >                                 region_descr->address_range_count);
> > -    if ( tot_len > max_frag_len )
> > -        return FFA_RET_NOT_SUPPORTED;
> >
> > +    /*
> > +     * Sharing memory with secure world may have to be done with multiple
> > +     * calls depending on how many address ranges will be needed. If we're
> > +     * sharing physically contiguous memory we will only need one range but
> > +     * we will also need to deal with the worst case where all physical
> > +     * pages are non-contiguous. For the first batch of address ranges we
> > +     * call ffa_mem_share() and for all that follows ffa_mem_frag_tx().
> > +     *
> > +     * We use frag_len to keep track of how far into the transmit buffer we
> > +     * have gone.
> > +     */
> >     addr_range = region_descr->address_range_array;
> >     frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
> >     last_pa = page_to_maddr(shm->pages[0]);
> >     init_range(addr_range, last_pa);
> > +    first = true;
> >     for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> >     {
> >         pa = page_to_maddr(shm->pages[n]);
> > @@ -1010,12 +1073,34 @@ static int share_shm(struct ffa_shm_mem *shm)
> >             continue;
> >         }
> >
> > -        frag_len += sizeof(*addr_range);
> > -        addr_range++;
> > +        if ( frag_len == max_frag_len )
> > +        {
> > +            if ( first )
> > +            {
> > +                ret = ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> > +                first = false;
> > +            }
> > +            else
> > +            {
> > +                ret = ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> > +            }
> > +            if ( ret <= 0 )
> > +                return ret;
> > +            frag_len = sizeof(*addr_range);
> > +            addr_range = buf;
> > +        }
> > +        else
> > +        {
> > +            frag_len += sizeof(*addr_range);
> > +            addr_range++;
> > +        }
> >         init_range(addr_range, pa);
> >     }
> >
> > -    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> > +    if ( first )
> > +        return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> > +    else
> > +        return ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> > }
> >
> > static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> > @@ -1092,8 +1177,53 @@ static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> >     return 0;
> > }
> >
> > +static int add_mem_share_frag(struct mem_frag_state *s, unsigned int offs,
> > +                              unsigned int frag_len)
> > +{
> > +    struct domain *d = current->domain;
> > +    unsigned int o = offs;
> > +    unsigned int l;
> > +    int ret;
> > +
> > +    if ( frag_len < o )
> > +        return FFA_RET_INVALID_PARAMETERS;
> > +
> > +    /* Fill up the first struct ffa_address_range */
> > +    l = min_t(unsigned int, frag_len - o, sizeof(s->range) - s->range_offset);
> > +    memcpy((uint8_t *)&s->range + s->range_offset, s->buf + o, l);
> > +    s->range_offset += l;
> > +    o += l;
> > +    if ( s->range_offset != sizeof(s->range) )
> > +        goto out;
> > +    s->range_offset = 0;
> > +
> > +    while ( true )
> > +    {
> > +        ret = get_shm_pages(d, s->shm, &s->range, 1, s->current_page_idx,
> > +                            &s->current_page_idx);
> > +        if ( ret )
> > +            return ret;
> > +        if ( s->range_count == 1 )
> > +            return 0;
> > +        s->range_count--;
> > +        if ( frag_len - o < sizeof(s->range) )
> > +            break;
> > +        memcpy(&s->range, s->buf + o, sizeof(s->range));
> > +        o += sizeof(s->range);
> > +    }
> > +
> > +    /* Collect any remaining bytes for the next struct ffa_address_range */
> > +    s->range_offset = frag_len - o;
> > +    memcpy(&s->range, s->buf + o, frag_len - o);
> > +out:
> > +    s->frag_offset += frag_len;
> > +
> > +    return s->frag_offset;
> > +}
> > +
> > static void handle_mem_share(struct cpu_user_regs *regs)
> > {
> > +    static uint64_t next_handle = FFA_HANDLE_HYP_FLAG;
> >     uint32_t tot_len = get_user_reg(regs, 1);
> >     uint32_t frag_len = get_user_reg(regs, 2);
> >     uint64_t addr = get_user_reg(regs, 3);
> > @@ -1128,13 +1258,6 @@ static void handle_mem_share(struct cpu_user_regs *regs)
> >         goto out_set_ret;
> >     }
> >
> > -    /* We currently only support a single fragment */
> > -    if ( frag_len != tot_len )
> > -    {
> > -        ret = FFA_RET_NOT_SUPPORTED;
> > -        goto out_set_ret;
> > -    }
> > -
> >     spin_lock(&ctx->lock);
> >
> >     if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> > @@ -1195,11 +1318,41 @@ static void handle_mem_share(struct cpu_user_regs *regs)
> >     if ( !shm )
> >     {
> >         ret = FFA_RET_NO_MEMORY;
> > -        goto out_unlock;
> > +        goto out;
>
> Why is this changed ?
> You still have no shm here so call free_sha_shm does not make sense

Good catch, I'll fix it.

>
> >     }
> >     shm->sender_id = trans.sender_id;
> >     shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> >
> > +    if ( frag_len != tot_len )
> > +    {
> > +        struct mem_frag_state *s = xzalloc(struct mem_frag_state);
> > +
> > +        if ( !s )
> > +        {
> > +            ret = FFA_RET_NO_MEMORY;
> > +            goto out;
> > +        }
> > +        s->shm = shm;
> > +        s->range_count = range_count;
> > +        s->buf = ctx->tx;
> > +        s->buf_size = ffa_page_count * FFA_PAGE_SIZE;
> > +        ret = add_mem_share_frag(s, sizeof(*region_descr)  + region_offs,
> > +                                 frag_len);
> > +        if ( ret <= 0 )
> > +        {
> > +            xfree(s);
> > +            if ( ret < 0 )
> > +                goto out;
> > +        }
> > +        else
> > +        {
> > +            shm->handle = next_handle++;
> > +            uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> > +            list_add_tail(&s->list, &ctx->frag_list);
> > +        }
> > +        goto out_unlock;
> > +    }
> > +
> >     /*
> >      * Check that the Composite memory region descriptor fits.
> >      */
> > @@ -1238,7 +1391,75 @@ out_unlock:
> >     spin_unlock(&ctx->lock);
> >
> > out_set_ret:
> > -    if ( ret == 0)
> > +    if ( ret > 0 )
> > +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, trans.sender_id);
> > +    else if ( ret == 0)
> > +            set_regs_success(regs, handle_lo, handle_hi);
> > +    else
> > +            set_regs_error(regs, ret);
> > +}
> > +
> > +static struct mem_frag_state *find_frag_state(struct ffa_ctx *ctx,
> > +                                              uint64_t handle)
> > +{
> > +    struct mem_frag_state *s;
> > +
> > +    list_for_each_entry(s, &ctx->frag_list, list)
> > +        if ( s->shm->handle == handle )
> > +            return s;
> > +
> > +    return NULL;
> > +}
> > +
> > +static void handle_mem_frag_tx(struct cpu_user_regs *regs)
> > +{
> > +    struct domain *d = current->domain;
> > +    struct ffa_ctx *ctx = d->arch.tee;
> > +    uint32_t frag_len = get_user_reg(regs, 3);
> > +    uint32_t handle_lo = get_user_reg(regs, 1);
> > +    uint32_t handle_hi = get_user_reg(regs, 2);
> > +    uint64_t handle = regpair_to_uint64(handle_hi, handle_lo);
> > +    struct mem_frag_state *s;
> > +    uint16_t sender_id = 0;
> > +    int ret;
> > +
> > +    spin_lock(&ctx->lock);
> > +    s = find_frag_state(ctx, handle);
> > +    if ( !s )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out;
> > +    }
> > +    sender_id = s->shm->sender_id;
> > +
> > +    if ( frag_len > s->buf_size )
> > +    {
> > +        ret = FFA_RET_INVALID_PARAMETERS;
> > +        goto out;
> > +    }
> > +
> > +    ret = add_mem_share_frag(s, 0, frag_len);
> > +    if ( ret == 0 )
> > +    {
> > +        /* Note that share_shm() uses our tx buffer */
> > +        spin_lock(&ffa_tx_buffer_lock);
> > +        ret = share_shm(s->shm);
> > +        spin_unlock(&ffa_tx_buffer_lock);
> > +        if ( ret == 0 )
> > +            list_add_tail(&s->shm->list, &ctx->shm_list);
> > +        else
> > +            free_ffa_shm_mem(ctx, s->shm);
> > +    }
> > +    else if ( ret < 0 )
> > +        free_ffa_shm_mem(ctx, s->shm);
>
>
> If there is not error the stuff allocated are kept but i do not see
> where/when they would be freed or used.
> Could you explain why we need to save all those ?

s->shm is the final shared memory object which is added to the list of
shared memory objects when the transaction is completed successfully.
The fragment state, s, is kept as long as the transaction is ongoing.
Once the transaction is completed successfully or due to a failure
it's freed.

The specification doesn't say what to do if an invalid frag_len is
detected, except that we should return FFA_RET_INVALID_PARAMETERS.
Currently, we just return an error, but keep the state. Another option
is to free the state instead since the caller might have lost track of
the state.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> > +    list_del(&s->list);
> > +    xfree(s);
> > +out:
> > +    spin_unlock(&ctx->lock);
> > +
> > +    if ( ret > 0 )
> > +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, sender_id);
> > +    else if ( ret == 0)
> >             set_regs_success(regs, handle_lo, handle_hi);
> >     else
> >             set_regs_error(regs, ret);
> > @@ -1357,6 +1578,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >         else
> >             set_regs_success(regs, 0, 0);
> >         return true;
> > +    case FFA_MEM_FRAG_TX:
> > +        handle_mem_frag_tx(regs);
> > +        return true;
> >
> >     default:
> >         gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> > @@ -1396,6 +1620,7 @@ static int ffa_domain_init(struct domain *d)
> >         }
> >     }
> >
> > +    INIT_LIST_HEAD(&ctx->frag_list);
> >     INIT_LIST_HEAD(&ctx->shm_list);
> >
> >     d->arch.tee = ctx;
> > @@ -1560,6 +1785,7 @@ static bool ffa_probe(void)
> > #endif
> >          !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> >          !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> > +         !check_mandatory_feature(FFA_MEM_FRAG_TX) ||
> >          !check_mandatory_feature(FFA_MEM_RECLAIM) ||
> >          !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >         return false;
> > --
> > 2.34.1
> >
>

Re: [XEN PATCH v7 18/20] xen/arm: ffa: support sharing memory

[Bertrand Marquis]

Hi Jens,

> On 14 Mar 2023, at 18:56, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Mon, Mar 13, 2023 at 9:49 AM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support for a guest to share memory with an SP using FFA_MEM_SHARE
>>> and FFA_MEM_RECLAIM. Only small memory regions can be shared using a
>>> single call to FFA_MEM_SHARE are supported.
>> 
>> This sentence needs a bit of rephrasing and to add more details: what is "small".
> 
> OK, how about "Only memory regions small enough to be shared with a
> single call..."

Ok

> 
>> 
>>> 
>>> A memory region that doesn't need to be shared any longer can be
>>> reclaimed with FFA_MEM_RECLAIM once the SP doesn't use it any longer.
>>> This is checked by the SPMC and not in control of the mediator.
>> 
>> This explanation would make more sense in the following patch adding support
>> for Reclaim.
> 
> Quite right, I'll move it to the next patch.
> 
>> 
>>> 
>>> With this commit we have a FF-A version 1.1 [1] mediator able to
>>> communicate with a Secure Partition in secure world using shared memory.
>>> The secure world must use FF-A version 1.1, but the guest is free to use
>>> version 1.0 or version 1.1.
>> 
>> I do not see anything limiting that in the code.
>> During init we accept 1.0 or 1.1 versions of the secure world.
> 
> Good catch, I'll update to only accept version 1.1 in the secure world.
> 
>> 
>>> 
>>> Adds a check that the SP supports the needed FF-A features
>>> FFA_MEM_SHARE_64 or FFA_MEM_SHARE_32.
>>> 
>>> [1] https://developer.arm.com/documentation/den0077/latest
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 491 +++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 491 insertions(+)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 94c90b252454..cdc286caf62c 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -270,6 +270,38 @@ struct ffa_mem_transaction_1_1 {
>>>    uint8_t reserved[12];
>>> };
>>> 
>>> +/* Calculate offset of struct ffa_mem_access from start of buffer */
>>> +#define MEM_ACCESS_OFFSET(access_idx) \
>>> +    ( sizeof(struct ffa_mem_transaction_1_1) + \
>>> +      ( access_idx ) * sizeof(struct ffa_mem_access) )
>>> +
>>> +/* Calculate offset of struct ffa_mem_region from start of buffer */
>>> +#define REGION_OFFSET(access_count, region_idx) \
>>> +    ( MEM_ACCESS_OFFSET(access_count) + \
>>> +      ( region_idx ) * sizeof(struct ffa_mem_region) )
>>> +
>>> +/* Calculate offset of struct ffa_address_range from start of buffer */
>>> +#define ADDR_RANGE_OFFSET(access_count, region_count, range_idx) \
>>> +    ( REGION_OFFSET(access_count, region_count) + \
>>> +      ( range_idx ) * sizeof(struct ffa_address_range) )
>>> +
>>> +/*
>>> + * The parts needed from struct ffa_mem_transaction_1_0 or struct
>>> + * ffa_mem_transaction_1_1, used to provide an abstraction of difference in
>>> + * data structures between version 1.0 and 1.1. This is just an internal
>>> + * interface and can be changed without changing any ABI.
>>> + */
>>> +struct ffa_mem_transaction_x {
>> 
>> I would suggest to s/_x/_int/ in the name here
> 
> OK, I'll update
> 
>> 
>>> +    uint16_t sender_id;
>>> +    uint8_t mem_reg_attr;
>>> +    uint8_t flags;
>>> +    uint8_t mem_access_size;
>>> +    uint8_t mem_access_count;
>>> +    uint16_t mem_access_offs;
>>> +    uint64_t global_handle;
>>> +    uint64_t tag;
>>> +};
>>> +
>>> /* Endpoint RX/TX descriptor */
>>> struct ffa_endpoint_rxtx_descriptor_1_0 {
>>>    uint16_t sender_id;
>>> @@ -294,8 +326,20 @@ struct ffa_ctx {
>>>    uint32_t guest_vers;
>>>    bool tx_is_mine;
>>>    bool interrupted;
>>> +    struct list_head shm_list;
>>> +    unsigned int shm_count;
>>>    spinlock_t lock;
>>> };
>>> +
>>> +struct ffa_shm_mem {
>>> +    struct list_head list;
>>> +    uint16_t sender_id;
>>> +    uint16_t ep_id;     /* endpoint, the one lending */
>>> +    uint64_t handle;    /* FFA_HANDLE_INVALID if not set yet */
>>> +    unsigned int page_count;
>>> +    struct page_info *pages[];
>>> +};
>>> +
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> @@ -310,6 +354,8 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
>>> *
>>> * ffa_page_count is the number of pages used in each of these buffers.
>>> *
>>> + * The TX buffer is protected from concurrent usage with ffa_tx_buffer_lock.
>>> + *
>>> * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
>>> * Note that the SPMC is also tracking the ownership of our RX buffer so
>>> * for calls which uses our RX buffer to deliver a result we must call
>>> @@ -319,6 +365,7 @@ static void *ffa_rx __read_mostly;
>>> static void *ffa_tx __read_mostly;
>>> static unsigned int ffa_page_count __read_mostly;
>>> static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
>>> +static DEFINE_SPINLOCK(ffa_tx_buffer_lock);
>>> 
>>> static bool ffa_get_version(uint32_t *vers)
>>> {
>>> @@ -429,6 +476,42 @@ static int32_t ffa_rx_release(void)
>>>    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
>>> }
>>> 
>>> +static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
>>> +                             register_t addr, uint32_t pg_count,
>>> +                             uint64_t *handle)
>>> +{
>>> +    struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = FFA_MEM_SHARE_32,
>>> +        .a1 = tot_len,
>>> +        .a2 = frag_len,
>>> +        .a3 = addr,
>>> +        .a4 = pg_count,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +
>>> +    if ( IS_ENABLED(CONFIG_ARM_64) )
>>> +        arg.a0 = FFA_MEM_SHARE_64;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    switch ( resp.a0 )
>>> +    {
>>> +    case FFA_ERROR:
>>> +        if ( resp.a2 )
>>> +            return resp.a2;
>>> +        else
>>> +            return FFA_RET_NOT_SUPPORTED;
>>> +    case FFA_SUCCESS_32:
>>> +        *handle = regpair_to_uint64(resp.a3, resp.a2);
>>> +        return FFA_RET_OK;
>>> +    case FFA_MEM_FRAG_RX:
>>> +        *handle = regpair_to_uint64(resp.a2, resp.a1);
>>> +        return resp.a3;
>> 
>> You are using an int32_t type to cast something that is uint32_t from the spec
>> and here could in fact be a uint64_t.
> 
> In practice only much smaller values can be valid, however, I
> understand that that's not your point. What's the best recovery if the
> SPMC gives an invalid value for FFA_MEM_FRAG_RX? The SPMC has
> allocated a memory-sharing state when it returns FFA_MEM_FRAG_RX. The
> specification doesn't say how to remove that state apart from either
> completing it successfully or if it's terminated earlier by the SPMC.
> One option is to do a FFA_MEM_FRAG_TX call with invalid arguments so
> that the SPMC can free up the memory-sharing state. Thoughts?

I do not think it is Xen responsability to fix SPMC problems here.
If we detect something of an error we should just return back an error to the client.

If the SPMC is returning an invalid value, we cannot really make much assumptions.

An other solution here would just be to mask out to prevent the implicit cast.

> 
>> 
>> 
>>> +    default:
>>> +        return FFA_RET_NOT_SUPPORTED;
>>> +    }
>>> +}
>>> +
>>> static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
>>>                                      uint8_t msg)
>>> {
>>> @@ -757,6 +840,404 @@ out:
>>>             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
>>> }
>>> 
>>> +/*
>>> + * Gets all page and assigns them to the supplied shared memory object. If
>>> + * this function fails then the caller is still expected to call
>>> + * put_shm_pages() as a cleanup.
>>> + */
>>> +static int get_shm_pages(struct domain *d, struct ffa_shm_mem *shm,
>>> +                         const struct ffa_address_range *range,
>>> +                         uint32_t range_count, unsigned int start_page_idx,
>>> +                         unsigned int *last_page_idx)
>>> +{
>>> +    unsigned int pg_idx = start_page_idx;
>>> +    gfn_t gfn;
>>> +    unsigned int n;
>>> +    unsigned int m;
>>> +    p2m_type_t t;
>>> +    uint64_t addr;
>>> +
>>> +    for ( n = 0; n < range_count; n++ )
>>> +    {
>>> +        for ( m = 0; m < range[n].page_count; m++ )
>>> +        {
>>> +            if ( pg_idx >= shm->page_count )
>>> +                return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +            addr = read_atomic(&range[n].address);
>>> +            gfn = gaddr_to_gfn(addr + m * FFA_PAGE_SIZE);
>>> +            shm->pages[pg_idx] = get_page_from_gfn(d, gfn_x(gfn), &t,
>>> +   P2M_ALLOC);
>>> +            if ( !shm->pages[pg_idx] )
>>> +                return FFA_RET_DENIED;
>>> +            pg_idx++;
>> 
>> This increment could be done at the end, why here ?
> 
> Do you mean after the p2m_is_ram() check? I'll move it there.

yes that would be more natural i think.

> 
>> 
>>> +            /* Only normal RAM for now */
>>> +            if ( !p2m_is_ram(t) )
>>> +                return FFA_RET_DENIED;
>>> +        }
>>> +    }
>>> +
>>> +    *last_page_idx = pg_idx;
>>> +
>>> +    return FFA_RET_OK;
>>> +}
>>> +
>>> +static void put_shm_pages(struct ffa_shm_mem *shm)
>>> +{
>>> +    unsigned int n;
>>> +
>>> +    for ( n = 0; n < shm->page_count && shm->pages[n]; n++ )
>>> +    {
>>> +        put_page(shm->pages[n]);
>>> +        shm->pages[n] = NULL;
>> 
>> If an error occured during the generation you might have part
>> of the pages which are NULL already.
>> 
>> So you should do a if (pages[n] != NULL) here
> 
> I'm doing that above in the head of the loop, the loop is terminated
> at the first pages[n] == NULL.

Right, sorry i missed that.

> 
>> 
>>> +    }
>>> +}
>>> +
>>> +static struct ffa_shm_mem *alloc_ffa_shm_mem(struct ffa_ctx *ctx,
>>> +                                             unsigned int page_count)
>>> +{
>>> +    struct ffa_shm_mem *shm;
>>> +
>>> +    if ( page_count >= FFA_MAX_SHM_PAGE_COUNT ||
>>> +         ctx->shm_count >= FFA_MAX_SHM_COUNT )
>>> +        return NULL;
>> 
>> Shouldn't you also filter out for page_count = 0 ?
> 
> Sure, 0 doesn't make sense. But I should probably do it before this
> function is called since I suppose we'd like to return something
> different from FFA_RET_NO_MEMORY.

Very true.

> 
>> 
>>> +
>>> +    shm = xzalloc_flex_struct(struct ffa_shm_mem, pages, page_count);
>>> +    if ( shm )
>>> +    {
>>> +        ctx->shm_count++;
>>> +        shm->page_count = page_count;
>>> +    }
>>> +
>>> +    return shm;
>>> +}
>>> +
>>> +static void free_ffa_shm_mem(struct ffa_ctx *ctx, struct ffa_shm_mem *shm)
>>> +{
>>> +    if ( shm ) {
>>> +        ASSERT(ctx->shm_count > 0);
>>> +        ctx->shm_count--;
>>> +        put_shm_pages(shm);
>>> +        xfree(shm);
>>> +    }
>>> +}
>>> +
>>> +static void init_range(struct ffa_address_range *addr_range,
>>> +                       paddr_t pa)
>>> +{
>>> +    memset(addr_range, 0, sizeof(*addr_range));
>>> +    addr_range->address = pa;
>>> +    addr_range->page_count = 1;
>>> +}
>>> +
>>> +/*
>>> + * This function uses the ffa_tx buffer to transmit the memory transaction
>>> + * descriptor. The function depends ffa_tx_buffer_lock to be used to guard
>>> + * the buffer from concurrent use.
>>> + */
>>> +static int share_shm(struct ffa_shm_mem *shm)
>>> +{
>>> +    const uint32_t max_frag_len = ffa_page_count * FFA_PAGE_SIZE;
>>> +    struct ffa_mem_access *mem_access_array;
>>> +    struct ffa_mem_transaction_1_1 *descr;
>>> +    struct ffa_address_range *addr_range;
>>> +    struct ffa_mem_region *region_descr;
>>> +    const unsigned int region_count = 1;
>>> +    void *buf = ffa_tx;
>>> +    uint32_t frag_len;
>>> +    uint32_t tot_len;
>>> +    paddr_t last_pa;
>>> +    unsigned int n;
>>> +    paddr_t pa;
>>> +
>>> +    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
>>> +    if ( !shm->page_count )
>>> +    {
>>> +        ASSERT_UNREACHABLE();
>>> +        return FFA_RET_INVALID_PARAMETERS;
>> 
>> page_count = 0 should be filtered out before reaching this and this should
>> only be an assert if you want but no unreachable with a return.
> 
> I'm adding code to filter out page_count = 0. I'm not sure what you
> expect here, should I remove the entire check here or what do you
> want?

As this is checked before, this could just be an assert and not something with a return.

> 
>> 
>>> +    }
>>> +
>>> +    descr = buf;
>>> +    memset(descr, 0, sizeof(*descr));
>>> +    descr->sender_id = shm->sender_id;
>>> +    descr->global_handle = shm->handle;
>>> +    descr->mem_reg_attr = FFA_NORMAL_MEM_REG_ATTR;
>>> +    descr->mem_access_count = 1;
>>> +    descr->mem_access_size = sizeof(*mem_access_array);
>>> +    descr->mem_access_offs = MEM_ACCESS_OFFSET(0);
>>> +
>>> +    mem_access_array = buf + descr->mem_access_offs;
>>> +    memset(mem_access_array, 0, sizeof(*mem_access_array));
>>> +    mem_access_array[0].access_perm.endpoint_id = shm->ep_id;
>>> +    mem_access_array[0].access_perm.perm = FFA_MEM_ACC_RW;
>>> +    mem_access_array[0].region_offs = REGION_OFFSET(descr->mem_access_count, 0);
>>> +
>>> +    region_descr = buf + mem_access_array[0].region_offs;
>>> +    memset(region_descr, 0, sizeof(*region_descr));
>>> +    region_descr->total_page_count = shm->page_count;
>>> +
>>> +    region_descr->address_range_count = 1;
>>> +    last_pa = page_to_maddr(shm->pages[0]);
>>> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
>>> +    {
>>> +        pa = page_to_maddr(shm->pages[n]);
>>> +        if ( last_pa + FFA_PAGE_SIZE == pa )
>>> +            continue;
>>> +        region_descr->address_range_count++;
>>> +    }
>>> +
>>> +    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
>>> +                                region_descr->address_range_count);
>>> +    if ( tot_len > max_frag_len )
>>> +        return FFA_RET_NOT_SUPPORTED;
>>> +
>>> +    addr_range = region_descr->address_range_array;
>>> +    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
>>> +    last_pa = page_to_maddr(shm->pages[0]);
>>> +    init_range(addr_range, last_pa);
>>> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
>>> +    {
>>> +        pa = page_to_maddr(shm->pages[n]);
>>> +        if ( last_pa + FFA_PAGE_SIZE == pa )
>>> +        {
>>> +            addr_range->page_count++;
>>> +            continue;
>>> +        }
>>> +
>>> +        frag_len += sizeof(*addr_range);
>>> +        addr_range++;
>>> +        init_range(addr_range, pa);
>>> +    }
>>> +
>>> +    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
>>> +}
>>> +
>>> +static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
>>> +                                struct ffa_mem_transaction_x *trans)
>>> +{
>>> +    uint16_t mem_reg_attr;
>>> +    uint32_t flags;
>>> +    uint32_t count;
>>> +    uint32_t offs;
>>> +    uint32_t size;
>>> +
>>> +    if ( ffa_vers >= FFA_VERSION_1_1 )
>>> +    {
>>> +        const struct ffa_mem_transaction_1_1 *descr;
>>> +
>>> +        if ( blen < sizeof(*descr) )
>>> +            return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +        descr = buf;
>>> +        trans->sender_id = descr->sender_id;
>>> +        mem_reg_attr = descr->mem_reg_attr;
>>> +        flags = descr->flags;
>>> +        trans->global_handle = descr->global_handle;
>>> +        trans->tag = descr->tag;
>>> +
>>> +        count = descr->mem_access_count;
>>> +        size = descr->mem_access_size;
>>> +        offs = descr->mem_access_offs;
>>> +    }
>>> +    else
>>> +    {
>>> +        const struct ffa_mem_transaction_1_0 *descr;
>>> +
>>> +        if ( blen < sizeof(*descr) )
>>> +            return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +        descr = buf;
>>> +        trans->sender_id = descr->sender_id;
>>> +        mem_reg_attr = descr->mem_reg_attr;
>>> +        flags = descr->flags;
>>> +        trans->global_handle = descr->global_handle;
>>> +        trans->tag = descr->tag;
>>> +
>>> +        count = descr->mem_access_count;
>>> +        size = sizeof(struct ffa_mem_access);
>>> +        offs = offsetof(struct ffa_mem_transaction_1_0, mem_access_array);
>>> +    }
>>> +    /*
>>> +     * Make sure that "descr" which is shared with the guest isn't accessed
>>> +     * again after this point.
>>> +     */
>>> +    barrier();
>> 
>> I am not really following the comment here. You accessed the content of descr
>> before and it is in the rxtx buffer so why is this needed ?
> 
> I'm making sure that the compiler doesn't optimize and reorders the
> reads from memory in funny ways, for instance, reading again after the
> ifs just below. The RXTX buffer is shared with the guest so it can
> potentially be updated concurrently by another CPU.

The client is not suppose to modify the buffer during the call, using atomic
operations here is not really solving any issue if this is modified while we use it right ?

> 
>> 
>>> +
>>> +    /*
>>> +     * We're doing a rough check to see that no information is lost when
>>> +     * tranfering the values into a struct ffa_mem_transaction_x below. The
>>> +     * fields in struct ffa_mem_transaction_x are wide enough to hold any
>>> +     * valid value so being out of range means that something is wrong.
>>> +     */
>>> +    if ( mem_reg_attr > UINT8_MAX || flags > UINT8_MAX || size > UINT8_MAX ||
>>> +        count > UINT8_MAX || offs > UINT16_MAX )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    /* Check that the endpoint memory access descriptor array fits */
>>> +    if ( size * count + offs > blen )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    trans->mem_reg_attr = mem_reg_attr;
>>> +    trans->flags = flags;
>>> +    trans->mem_access_size = size;
>>> +    trans->mem_access_count = count;
>>> +    trans->mem_access_offs = offs;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void handle_mem_share(struct cpu_user_regs *regs)
>>> +{
>>> +    uint32_t tot_len = get_user_reg(regs, 1);
>>> +    uint32_t frag_len = get_user_reg(regs, 2);
>>> +    uint64_t addr = get_user_reg(regs, 3);
>>> +    uint32_t page_count = get_user_reg(regs, 4);
>>> +    const struct ffa_mem_region *region_descr;
>>> +    const struct ffa_mem_access *mem_access;
>>> +    struct ffa_mem_transaction_x trans;
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +    struct ffa_shm_mem *shm = NULL;
>>> +    unsigned int last_page_idx = 0;
>>> +    register_t handle_hi = 0;
>>> +    register_t handle_lo = 0;
>>> +    int ret = FFA_RET_DENIED;
>>> +    uint32_t range_count;
>>> +    uint32_t region_offs;
>>> +
>>> +    /*
>>> +     * We're only accepting memory transaction descriptors via the rx/tx
>>> +     * buffer.
>> 
>> Is this a limitation coming fomr the spec or from the implementation ?
> 
> This is just a limitation in the implementation.
> 
>> 
>>> +     */
>>> +    if ( addr )
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_set_ret;
>>> +    }
>>> +
>>> +    /* Check that fragment length doesn't exceed total length */
>>> +    if ( frag_len > tot_len )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out_set_ret;
>>> +    }
>>> +
>>> +    /* We currently only support a single fragment */
>> 
>> It would make sense to add some text at the beginning of the files listing
>> the current limitations of the implementation.
> 
> That's quite a bit to keep track of, especially since it will change
> with each patch. If it's important perhaps we can summarize that in a
> final commit instead. By the way, this particular limitation is
> removed in a later patch.

I am more thinking at the end of the serie to have one place with the current
state and limitations of the implementation.

We cannot really expect someone to browse all comments to get what is
 supported or not.

> 
>> 
>>> +    if ( frag_len != tot_len )
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_set_ret;
>>> +    }
>>> +
>>> +    spin_lock(&ctx->lock);
>>> +
>>> +    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
>>> +        goto out_unlock;
>>> +
>>> +    if ( !ffa_page_count )
>>> +    {
>>> +        ret = FFA_RET_NO_MEMORY;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    ret = read_mem_transaction(ctx->guest_vers, ctx->tx, frag_len, &trans);
>>> +    if ( ret )
>>> +        goto out_unlock;
>>> +
>>> +    if ( trans.mem_reg_attr != FFA_NORMAL_MEM_REG_ATTR )
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    /* Only supports sharing it with one SP for now */
>> 
>> Also a limitation to list.
> 
> OK
> 
>> 
>>> +    if ( trans.mem_access_count != 1 )
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    if ( trans.sender_id != get_vm_id(d) )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    /* Check that it fits in the supplied data */
>>> +    if ( trans.mem_access_offs + trans.mem_access_size > frag_len )
>>> +        goto out_unlock;
>>> +
>> 
>> Why are you using atomic operations to access rxtx buffer after here ?
> 
> To limit how the compiler can reorder the reads from memory.

As stated before, don't we assume that the buffer is not modified by the client
 while we use it ?

> 
>> 
>>> +    mem_access = ctx->tx + trans.mem_access_offs;
>>> +    if ( read_atomic(&mem_access->access_perm.perm) != FFA_MEM_ACC_RW )
>> 
>> Also a limitation to list.
> 
> OK

Cheers
Bertrand

> 
> Thanks,
> Jens
> 
>> 
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    region_offs = read_atomic(&mem_access->region_offs);
>>> +    if ( sizeof(*region_descr) + region_offs > frag_len )
>>> +    {
>>> +        ret = FFA_RET_NOT_SUPPORTED;
>>> +        goto out_unlock;
>>> +    }
>>> +
>>> +    region_descr = ctx->tx + region_offs;
>>> +    range_count = read_atomic(&region_descr->address_range_count);
>>> +    page_count = read_atomic(&region_descr->total_page_count);
>>> +
>>> +    shm = alloc_ffa_shm_mem(ctx, page_count);
>>> +    if ( !shm )
>>> +    {
>>> +        ret = FFA_RET_NO_MEMORY;
>>> +        goto out_unlock;
>>> +    }
>>> +    shm->sender_id = trans.sender_id;
>>> +    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
>>> +
>>> +    /*
>>> +     * Check that the Composite memory region descriptor fits.
>>> +     */
>>> +    if ( sizeof(*region_descr) + region_offs +
>>> +         range_count * sizeof(struct ffa_address_range) > frag_len )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +
>>> +    ret = get_shm_pages(d, shm, region_descr->address_range_array, range_count,
>>> +                        0, &last_page_idx);
>>> +    if ( ret )
>>> +        goto out;
>>> +    if ( last_page_idx != shm->page_count )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +
>>> +    /* Note that share_shm() uses our tx buffer */
>>> +    spin_lock(&ffa_tx_buffer_lock);
>>> +    ret = share_shm(shm);
>>> +    spin_unlock(&ffa_tx_buffer_lock);
>>> +    if ( ret )
>>> +        goto out;
>>> +
>>> +    list_add_tail(&shm->list, &ctx->shm_list);
>>> +
>>> +    uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
>>> +
>>> +out:
>>> +    if ( ret )
>>> +        free_ffa_shm_mem(ctx, shm);
>>> +out_unlock:
>>> +    spin_unlock(&ctx->lock);
>>> +
>>> +out_set_ret:
>>> +    if ( ret == 0)
>>> +            set_regs_success(regs, handle_lo, handle_hi);
>>> +    else
>>> +            set_regs_error(regs, ret);
>>> +}
>>> +
>>> static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> {
>>>    uint32_t fid = get_user_reg(regs, 0);
>>> @@ -818,6 +1299,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>> #endif
>>>        handle_msg_send_direct_req(regs, fid);
>>>        return true;
>>> +    case FFA_MEM_SHARE_32:
>>> +#ifdef CONFIG_ARM_64
>>> +    case FFA_MEM_SHARE_64:
>>> +#endif
>>> +        handle_mem_share(regs);
>>> +        return true;
>>> 
>>>    default:
>>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
>>> @@ -857,6 +1344,8 @@ static int ffa_domain_init(struct domain *d)
>>>        }
>>>    }
>>> 
>>> +    INIT_LIST_HEAD(&ctx->shm_list);
>>> +
>>>    d->arch.tee = ctx;
>>> 
>>>    return 0;
>>> @@ -1012,11 +1501,13 @@ static bool ffa_probe(void)
>>>         !check_mandatory_feature(FFA_RX_RELEASE) ||
>>> #ifdef CONFIG_ARM_64
>>>         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
>>> +         !check_mandatory_feature(FFA_MEM_SHARE_64) ||
>>> #endif
>>> #ifdef CONFIG_ARM_32
>>>         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
>>> #endif
>>>         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
>>> +         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
>>>         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>>        return false;
>>> 
>>> --
>>> 2.34.1
>> 
>> 
>> Cheers
>> Bertrand

Re: [XEN PATCH v7 20/20] xen/arm: ffa: support sharing large memory ranges

[Bertrand Marquis]

Hi Jens,

> On 15 Mar 2023, at 12:47, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> 
> Hi Bertrand,
> 
> On Wed, Mar 15, 2023 at 11:13 AM Bertrand Marquis
> <Bertrand.Marquis@arm.com> wrote:
>> 
>> Hi Jens,
>> 
>>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>> 
>>> Adds support for sharing large memory ranges transmitted in fragments
>>> using FFA_MEM_FRAG_TX.
>>> 
>>> The implementation is the bare minimum to be able to communicate with
>>> OP-TEE running as an SPMC at S-EL1.
>>> 
>>> Adds a check that the SP supports the needed FF-A feature
>>> FFA_MEM_FRAG_TX.
>>> 
>>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
>>> ---
>>> xen/arch/arm/tee/ffa.c | 254 ++++++++++++++++++++++++++++++++++++++---
>>> 1 file changed, 240 insertions(+), 14 deletions(-)
>>> 
>>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
>>> index 3557edc455d0..72c0249d8cad 100644
>>> --- a/xen/arch/arm/tee/ffa.c
>>> +++ b/xen/arch/arm/tee/ffa.c
>>> @@ -326,6 +326,7 @@ struct ffa_ctx {
>>>    uint32_t guest_vers;
>>>    bool tx_is_mine;
>>>    bool interrupted;
>>> +    struct list_head frag_list;
>>>    struct list_head shm_list;
>>>    unsigned int shm_count;
>>>    spinlock_t lock;
>>> @@ -340,6 +341,18 @@ struct ffa_shm_mem {
>>>    struct page_info *pages[];
>>> };
>> 
>> We start to have a lot of fields here.
>> It might be useful to have some quick documentation
>> in comment here as some names are a bit generic.
>> For example "frag_list" does not say much.
> 
> I'll add some comments.
> 
>> 
>>> 
>>> +struct mem_frag_state {
>>> +    struct list_head list;
>>> +    struct ffa_shm_mem *shm;
>>> +    uint32_t range_count;
>>> +    unsigned int current_page_idx;
>>> +    unsigned int frag_offset;
>>> +    unsigned int range_offset;
>>> +    const uint8_t *buf;
>>> +    unsigned int buf_size;
>>> +    struct ffa_address_range range;
>>> +};
>> 
>> same here, at a first glance it is not quite clear why
>> a fragment needs that much info. Some seem to only
>> be needed during the transaction but do not need to be saved.
> 
> This struct is only used during the transaction, so this is freed once
> the entire memory transaction descriptor has been processed.
> 
>> 
>>> +
>>> /* Negotiated FF-A version to use with the SPMC */
>>> static uint32_t ffa_version __ro_after_init;
>>> 
>>> @@ -512,6 +525,36 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
>>>    }
>>> }
>>> 
>>> +static int32_t ffa_mem_frag_tx(uint64_t handle, uint32_t frag_len,
>>> +                               uint16_t sender_id)
>>> +{
>>> +    struct arm_smccc_1_2_regs arg = {
>>> +        .a0 = FFA_MEM_FRAG_TX,
>>> +        .a1 = handle & UINT32_MAX,
>>> +        .a2 = handle >> 32,
>>> +        .a3 = frag_len,
>>> +        .a4 = (uint32_t)sender_id << 16,
>>> +    };
>>> +    struct arm_smccc_1_2_regs resp;
>>> +
>>> +    arm_smccc_1_2_smc(&arg, &resp);
>>> +
>>> +    switch ( resp.a0 )
>>> +    {
>>> +    case FFA_ERROR:
>>> +        if ( resp.a2 )
>>> +            return resp.a2;
>>> +        else
>>> +            return FFA_RET_NOT_SUPPORTED;
>>> +    case FFA_SUCCESS_32:
>>> +        return FFA_RET_OK;
>>> +    case FFA_MEM_FRAG_RX:
>>> +        return resp.a3;
>>> +    default:
>>> +            return FFA_RET_NOT_SUPPORTED;
>>> +    }
>>> +}
>>> +
>>> static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
>>>                               uint32_t flags)
>>> {
>>> @@ -586,6 +629,14 @@ static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
>>>    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
>>> }
>>> 
>>> +static void set_regs_frag_rx(struct cpu_user_regs *regs, uint32_t handle_lo,
>>> +                             uint32_t handle_hi, uint32_t frag_offset,
>>> +                             uint16_t sender_id)
>>> +{
>>> +    set_regs(regs, FFA_MEM_FRAG_RX, handle_lo, handle_hi, frag_offset,
>>> +             (uint32_t)sender_id << 16, 0, 0, 0);
>>> +}
>>> +
>>> static void handle_version(struct cpu_user_regs *regs)
>>> {
>>>    struct domain *d = current->domain;
>>> @@ -955,6 +1006,8 @@ static int share_shm(struct ffa_shm_mem *shm)
>>>    paddr_t last_pa;
>>>    unsigned int n;
>>>    paddr_t pa;
>>> +    bool first;
>>> +    int ret;
>>> 
>>>    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
>>>    if ( !shm->page_count )
>>> @@ -994,13 +1047,23 @@ static int share_shm(struct ffa_shm_mem *shm)
>>> 
>>>    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
>>>                                region_descr->address_range_count);
>>> -    if ( tot_len > max_frag_len )
>>> -        return FFA_RET_NOT_SUPPORTED;
>>> 
>>> +    /*
>>> +     * Sharing memory with secure world may have to be done with multiple
>>> +     * calls depending on how many address ranges will be needed. If we're
>>> +     * sharing physically contiguous memory we will only need one range but
>>> +     * we will also need to deal with the worst case where all physical
>>> +     * pages are non-contiguous. For the first batch of address ranges we
>>> +     * call ffa_mem_share() and for all that follows ffa_mem_frag_tx().
>>> +     *
>>> +     * We use frag_len to keep track of how far into the transmit buffer we
>>> +     * have gone.
>>> +     */
>>>    addr_range = region_descr->address_range_array;
>>>    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
>>>    last_pa = page_to_maddr(shm->pages[0]);
>>>    init_range(addr_range, last_pa);
>>> +    first = true;
>>>    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
>>>    {
>>>        pa = page_to_maddr(shm->pages[n]);
>>> @@ -1010,12 +1073,34 @@ static int share_shm(struct ffa_shm_mem *shm)
>>>            continue;
>>>        }
>>> 
>>> -        frag_len += sizeof(*addr_range);
>>> -        addr_range++;
>>> +        if ( frag_len == max_frag_len )
>>> +        {
>>> +            if ( first )
>>> +            {
>>> +                ret = ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
>>> +                first = false;
>>> +            }
>>> +            else
>>> +            {
>>> +                ret = ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
>>> +            }
>>> +            if ( ret <= 0 )
>>> +                return ret;
>>> +            frag_len = sizeof(*addr_range);
>>> +            addr_range = buf;
>>> +        }
>>> +        else
>>> +        {
>>> +            frag_len += sizeof(*addr_range);
>>> +            addr_range++;
>>> +        }
>>>        init_range(addr_range, pa);
>>>    }
>>> 
>>> -    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
>>> +    if ( first )
>>> +        return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
>>> +    else
>>> +        return ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
>>> }
>>> 
>>> static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
>>> @@ -1092,8 +1177,53 @@ static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
>>>    return 0;
>>> }
>>> 
>>> +static int add_mem_share_frag(struct mem_frag_state *s, unsigned int offs,
>>> +                              unsigned int frag_len)
>>> +{
>>> +    struct domain *d = current->domain;
>>> +    unsigned int o = offs;
>>> +    unsigned int l;
>>> +    int ret;
>>> +
>>> +    if ( frag_len < o )
>>> +        return FFA_RET_INVALID_PARAMETERS;
>>> +
>>> +    /* Fill up the first struct ffa_address_range */
>>> +    l = min_t(unsigned int, frag_len - o, sizeof(s->range) - s->range_offset);
>>> +    memcpy((uint8_t *)&s->range + s->range_offset, s->buf + o, l);
>>> +    s->range_offset += l;
>>> +    o += l;
>>> +    if ( s->range_offset != sizeof(s->range) )
>>> +        goto out;
>>> +    s->range_offset = 0;
>>> +
>>> +    while ( true )
>>> +    {
>>> +        ret = get_shm_pages(d, s->shm, &s->range, 1, s->current_page_idx,
>>> +                            &s->current_page_idx);
>>> +        if ( ret )
>>> +            return ret;
>>> +        if ( s->range_count == 1 )
>>> +            return 0;
>>> +        s->range_count--;
>>> +        if ( frag_len - o < sizeof(s->range) )
>>> +            break;
>>> +        memcpy(&s->range, s->buf + o, sizeof(s->range));
>>> +        o += sizeof(s->range);
>>> +    }
>>> +
>>> +    /* Collect any remaining bytes for the next struct ffa_address_range */
>>> +    s->range_offset = frag_len - o;
>>> +    memcpy(&s->range, s->buf + o, frag_len - o);
>>> +out:
>>> +    s->frag_offset += frag_len;
>>> +
>>> +    return s->frag_offset;
>>> +}
>>> +
>>> static void handle_mem_share(struct cpu_user_regs *regs)
>>> {
>>> +    static uint64_t next_handle = FFA_HANDLE_HYP_FLAG;
>>>    uint32_t tot_len = get_user_reg(regs, 1);
>>>    uint32_t frag_len = get_user_reg(regs, 2);
>>>    uint64_t addr = get_user_reg(regs, 3);
>>> @@ -1128,13 +1258,6 @@ static void handle_mem_share(struct cpu_user_regs *regs)
>>>        goto out_set_ret;
>>>    }
>>> 
>>> -    /* We currently only support a single fragment */
>>> -    if ( frag_len != tot_len )
>>> -    {
>>> -        ret = FFA_RET_NOT_SUPPORTED;
>>> -        goto out_set_ret;
>>> -    }
>>> -
>>>    spin_lock(&ctx->lock);
>>> 
>>>    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
>>> @@ -1195,11 +1318,41 @@ static void handle_mem_share(struct cpu_user_regs *regs)
>>>    if ( !shm )
>>>    {
>>>        ret = FFA_RET_NO_MEMORY;
>>> -        goto out_unlock;
>>> +        goto out;
>> 
>> Why is this changed ?
>> You still have no shm here so call free_sha_shm does not make sense
> 
> Good catch, I'll fix it.
> 
>> 
>>>    }
>>>    shm->sender_id = trans.sender_id;
>>>    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
>>> 
>>> +    if ( frag_len != tot_len )
>>> +    {
>>> +        struct mem_frag_state *s = xzalloc(struct mem_frag_state);
>>> +
>>> +        if ( !s )
>>> +        {
>>> +            ret = FFA_RET_NO_MEMORY;
>>> +            goto out;
>>> +        }
>>> +        s->shm = shm;
>>> +        s->range_count = range_count;
>>> +        s->buf = ctx->tx;
>>> +        s->buf_size = ffa_page_count * FFA_PAGE_SIZE;
>>> +        ret = add_mem_share_frag(s, sizeof(*region_descr)  + region_offs,
>>> +                                 frag_len);
>>> +        if ( ret <= 0 )
>>> +        {
>>> +            xfree(s);
>>> +            if ( ret < 0 )
>>> +                goto out;
>>> +        }
>>> +        else
>>> +        {
>>> +            shm->handle = next_handle++;
>>> +            uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
>>> +            list_add_tail(&s->list, &ctx->frag_list);
>>> +        }
>>> +        goto out_unlock;
>>> +    }
>>> +
>>>    /*
>>>     * Check that the Composite memory region descriptor fits.
>>>     */
>>> @@ -1238,7 +1391,75 @@ out_unlock:
>>>    spin_unlock(&ctx->lock);
>>> 
>>> out_set_ret:
>>> -    if ( ret == 0)
>>> +    if ( ret > 0 )
>>> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, trans.sender_id);
>>> +    else if ( ret == 0)
>>> +            set_regs_success(regs, handle_lo, handle_hi);
>>> +    else
>>> +            set_regs_error(regs, ret);
>>> +}
>>> +
>>> +static struct mem_frag_state *find_frag_state(struct ffa_ctx *ctx,
>>> +                                              uint64_t handle)
>>> +{
>>> +    struct mem_frag_state *s;
>>> +
>>> +    list_for_each_entry(s, &ctx->frag_list, list)
>>> +        if ( s->shm->handle == handle )
>>> +            return s;
>>> +
>>> +    return NULL;
>>> +}
>>> +
>>> +static void handle_mem_frag_tx(struct cpu_user_regs *regs)
>>> +{
>>> +    struct domain *d = current->domain;
>>> +    struct ffa_ctx *ctx = d->arch.tee;
>>> +    uint32_t frag_len = get_user_reg(regs, 3);
>>> +    uint32_t handle_lo = get_user_reg(regs, 1);
>>> +    uint32_t handle_hi = get_user_reg(regs, 2);
>>> +    uint64_t handle = regpair_to_uint64(handle_hi, handle_lo);
>>> +    struct mem_frag_state *s;
>>> +    uint16_t sender_id = 0;
>>> +    int ret;
>>> +
>>> +    spin_lock(&ctx->lock);
>>> +    s = find_frag_state(ctx, handle);
>>> +    if ( !s )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +    sender_id = s->shm->sender_id;
>>> +
>>> +    if ( frag_len > s->buf_size )
>>> +    {
>>> +        ret = FFA_RET_INVALID_PARAMETERS;
>>> +        goto out;
>>> +    }
>>> +
>>> +    ret = add_mem_share_frag(s, 0, frag_len);
>>> +    if ( ret == 0 )
>>> +    {
>>> +        /* Note that share_shm() uses our tx buffer */
>>> +        spin_lock(&ffa_tx_buffer_lock);
>>> +        ret = share_shm(s->shm);
>>> +        spin_unlock(&ffa_tx_buffer_lock);
>>> +        if ( ret == 0 )
>>> +            list_add_tail(&s->shm->list, &ctx->shm_list);
>>> +        else
>>> +            free_ffa_shm_mem(ctx, s->shm);
>>> +    }
>>> +    else if ( ret < 0 )
>>> +        free_ffa_shm_mem(ctx, s->shm);
>> 
>> 
>> If there is not error the stuff allocated are kept but i do not see
>> where/when they would be freed or used.
>> Could you explain why we need to save all those ?
> 
> s->shm is the final shared memory object which is added to the list of
> shared memory objects when the transaction is completed successfully.
> The fragment state, s, is kept as long as the transaction is ongoing.
> Once the transaction is completed successfully or due to a failure
> it's freed.
> 
> The specification doesn't say what to do if an invalid frag_len is
> detected, except that we should return FFA_RET_INVALID_PARAMETERS.
> Currently, we just return an error, but keep the state. Another option
> is to free the state instead since the caller might have lost track of
> the state.

There is no solution for the client to "continue" anyway so I think we should
properly cleanup in all exit conditions.

Cheers
Bertrand

> 
> Thanks,
> Jens
> 
>> 
>> Cheers
>> Bertrand
>> 
>>> +    list_del(&s->list);
>>> +    xfree(s);
>>> +out:
>>> +    spin_unlock(&ctx->lock);
>>> +
>>> +    if ( ret > 0 )
>>> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, sender_id);
>>> +    else if ( ret == 0)
>>>            set_regs_success(regs, handle_lo, handle_hi);
>>>    else
>>>            set_regs_error(regs, ret);
>>> @@ -1357,6 +1578,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
>>>        else
>>>            set_regs_success(regs, 0, 0);
>>>        return true;
>>> +    case FFA_MEM_FRAG_TX:
>>> +        handle_mem_frag_tx(regs);
>>> +        return true;
>>> 
>>>    default:
>>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
>>> @@ -1396,6 +1620,7 @@ static int ffa_domain_init(struct domain *d)
>>>        }
>>>    }
>>> 
>>> +    INIT_LIST_HEAD(&ctx->frag_list);
>>>    INIT_LIST_HEAD(&ctx->shm_list);
>>> 
>>>    d->arch.tee = ctx;
>>> @@ -1560,6 +1785,7 @@ static bool ffa_probe(void)
>>> #endif
>>>         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
>>>         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
>>> +         !check_mandatory_feature(FFA_MEM_FRAG_TX) ||
>>>         !check_mandatory_feature(FFA_MEM_RECLAIM) ||
>>>         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
>>>        return false;
>>> --
>>> 2.34.1

Re: [XEN PATCH v7 18/20] xen/arm: ffa: support sharing memory

[Jens Wiklander]

Hi Bertrand,

On Wed, Mar 15, 2023 at 2:24 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 14 Mar 2023, at 18:56, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Mon, Mar 13, 2023 at 9:49 AM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support for a guest to share memory with an SP using FFA_MEM_SHARE
> >>> and FFA_MEM_RECLAIM. Only small memory regions can be shared using a
> >>> single call to FFA_MEM_SHARE are supported.
> >>
> >> This sentence needs a bit of rephrasing and to add more details: what is "small".
> >
> > OK, how about "Only memory regions small enough to be shared with a
> > single call..."
>
> Ok
>
> >
> >>
> >>>
> >>> A memory region that doesn't need to be shared any longer can be
> >>> reclaimed with FFA_MEM_RECLAIM once the SP doesn't use it any longer.
> >>> This is checked by the SPMC and not in control of the mediator.
> >>
> >> This explanation would make more sense in the following patch adding support
> >> for Reclaim.
> >
> > Quite right, I'll move it to the next patch.
> >
> >>
> >>>
> >>> With this commit we have a FF-A version 1.1 [1] mediator able to
> >>> communicate with a Secure Partition in secure world using shared memory.
> >>> The secure world must use FF-A version 1.1, but the guest is free to use
> >>> version 1.0 or version 1.1.
> >>
> >> I do not see anything limiting that in the code.
> >> During init we accept 1.0 or 1.1 versions of the secure world.
> >
> > Good catch, I'll update to only accept version 1.1 in the secure world.
> >
> >>
> >>>
> >>> Adds a check that the SP supports the needed FF-A features
> >>> FFA_MEM_SHARE_64 or FFA_MEM_SHARE_32.
> >>>
> >>> [1] https://developer.arm.com/documentation/den0077/latest
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 491 +++++++++++++++++++++++++++++++++++++++++
> >>> 1 file changed, 491 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 94c90b252454..cdc286caf62c 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -270,6 +270,38 @@ struct ffa_mem_transaction_1_1 {
> >>>    uint8_t reserved[12];
> >>> };
> >>>
> >>> +/* Calculate offset of struct ffa_mem_access from start of buffer */
> >>> +#define MEM_ACCESS_OFFSET(access_idx) \
> >>> +    ( sizeof(struct ffa_mem_transaction_1_1) + \
> >>> +      ( access_idx ) * sizeof(struct ffa_mem_access) )
> >>> +
> >>> +/* Calculate offset of struct ffa_mem_region from start of buffer */
> >>> +#define REGION_OFFSET(access_count, region_idx) \
> >>> +    ( MEM_ACCESS_OFFSET(access_count) + \
> >>> +      ( region_idx ) * sizeof(struct ffa_mem_region) )
> >>> +
> >>> +/* Calculate offset of struct ffa_address_range from start of buffer */
> >>> +#define ADDR_RANGE_OFFSET(access_count, region_count, range_idx) \
> >>> +    ( REGION_OFFSET(access_count, region_count) + \
> >>> +      ( range_idx ) * sizeof(struct ffa_address_range) )
> >>> +
> >>> +/*
> >>> + * The parts needed from struct ffa_mem_transaction_1_0 or struct
> >>> + * ffa_mem_transaction_1_1, used to provide an abstraction of difference in
> >>> + * data structures between version 1.0 and 1.1. This is just an internal
> >>> + * interface and can be changed without changing any ABI.
> >>> + */
> >>> +struct ffa_mem_transaction_x {
> >>
> >> I would suggest to s/_x/_int/ in the name here
> >
> > OK, I'll update
> >
> >>
> >>> +    uint16_t sender_id;
> >>> +    uint8_t mem_reg_attr;
> >>> +    uint8_t flags;
> >>> +    uint8_t mem_access_size;
> >>> +    uint8_t mem_access_count;
> >>> +    uint16_t mem_access_offs;
> >>> +    uint64_t global_handle;
> >>> +    uint64_t tag;
> >>> +};
> >>> +
> >>> /* Endpoint RX/TX descriptor */
> >>> struct ffa_endpoint_rxtx_descriptor_1_0 {
> >>>    uint16_t sender_id;
> >>> @@ -294,8 +326,20 @@ struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>>    bool tx_is_mine;
> >>>    bool interrupted;
> >>> +    struct list_head shm_list;
> >>> +    unsigned int shm_count;
> >>>    spinlock_t lock;
> >>> };
> >>> +
> >>> +struct ffa_shm_mem {
> >>> +    struct list_head list;
> >>> +    uint16_t sender_id;
> >>> +    uint16_t ep_id;     /* endpoint, the one lending */
> >>> +    uint64_t handle;    /* FFA_HANDLE_INVALID if not set yet */
> >>> +    unsigned int page_count;
> >>> +    struct page_info *pages[];
> >>> +};
> >>> +
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> @@ -310,6 +354,8 @@ static unsigned int subscr_vm_destroyed_count __read_mostly;
> >>> *
> >>> * ffa_page_count is the number of pages used in each of these buffers.
> >>> *
> >>> + * The TX buffer is protected from concurrent usage with ffa_tx_buffer_lock.
> >>> + *
> >>> * The RX buffer is protected from concurrent usage with ffa_rx_buffer_lock.
> >>> * Note that the SPMC is also tracking the ownership of our RX buffer so
> >>> * for calls which uses our RX buffer to deliver a result we must call
> >>> @@ -319,6 +365,7 @@ static void *ffa_rx __read_mostly;
> >>> static void *ffa_tx __read_mostly;
> >>> static unsigned int ffa_page_count __read_mostly;
> >>> static DEFINE_SPINLOCK(ffa_rx_buffer_lock);
> >>> +static DEFINE_SPINLOCK(ffa_tx_buffer_lock);
> >>>
> >>> static bool ffa_get_version(uint32_t *vers)
> >>> {
> >>> @@ -429,6 +476,42 @@ static int32_t ffa_rx_release(void)
> >>>    return ffa_simple_call(FFA_RX_RELEASE, 0, 0, 0, 0);
> >>> }
> >>>
> >>> +static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
> >>> +                             register_t addr, uint32_t pg_count,
> >>> +                             uint64_t *handle)
> >>> +{
> >>> +    struct arm_smccc_1_2_regs arg = {
> >>> +        .a0 = FFA_MEM_SHARE_32,
> >>> +        .a1 = tot_len,
> >>> +        .a2 = frag_len,
> >>> +        .a3 = addr,
> >>> +        .a4 = pg_count,
> >>> +    };
> >>> +    struct arm_smccc_1_2_regs resp;
> >>> +
> >>> +    if ( IS_ENABLED(CONFIG_ARM_64) )
> >>> +        arg.a0 = FFA_MEM_SHARE_64;
> >>> +
> >>> +    arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +    switch ( resp.a0 )
> >>> +    {
> >>> +    case FFA_ERROR:
> >>> +        if ( resp.a2 )
> >>> +            return resp.a2;
> >>> +        else
> >>> +            return FFA_RET_NOT_SUPPORTED;
> >>> +    case FFA_SUCCESS_32:
> >>> +        *handle = regpair_to_uint64(resp.a3, resp.a2);
> >>> +        return FFA_RET_OK;
> >>> +    case FFA_MEM_FRAG_RX:
> >>> +        *handle = regpair_to_uint64(resp.a2, resp.a1);
> >>> +        return resp.a3;
> >>
> >> You are using an int32_t type to cast something that is uint32_t from the spec
> >> and here could in fact be a uint64_t.
> >
> > In practice only much smaller values can be valid, however, I
> > understand that that's not your point. What's the best recovery if the
> > SPMC gives an invalid value for FFA_MEM_FRAG_RX? The SPMC has
> > allocated a memory-sharing state when it returns FFA_MEM_FRAG_RX. The
> > specification doesn't say how to remove that state apart from either
> > completing it successfully or if it's terminated earlier by the SPMC.
> > One option is to do a FFA_MEM_FRAG_TX call with invalid arguments so
> > that the SPMC can free up the memory-sharing state. Thoughts?
>
> I do not think it is Xen responsability to fix SPMC problems here.
> If we detect something of an error we should just return back an error to the client.
>
> If the SPMC is returning an invalid value, we cannot really make much assumptions.
>
> An other solution here would just be to mask out to prevent the implicit cast.

OK, I'll do that.

>
> >
> >>
> >>
> >>> +    default:
> >>> +        return FFA_RET_NOT_SUPPORTED;
> >>> +    }
> >>> +}
> >>> +
> >>> static int32_t ffa_direct_req_send_vm(uint16_t sp_id, uint16_t vm_id,
> >>>                                      uint8_t msg)
> >>> {
> >>> @@ -757,6 +840,404 @@ out:
> >>>             resp.a4 & mask, resp.a5 & mask, resp.a6 & mask, resp.a7 & mask);
> >>> }
> >>>
> >>> +/*
> >>> + * Gets all page and assigns them to the supplied shared memory object. If
> >>> + * this function fails then the caller is still expected to call
> >>> + * put_shm_pages() as a cleanup.
> >>> + */
> >>> +static int get_shm_pages(struct domain *d, struct ffa_shm_mem *shm,
> >>> +                         const struct ffa_address_range *range,
> >>> +                         uint32_t range_count, unsigned int start_page_idx,
> >>> +                         unsigned int *last_page_idx)
> >>> +{
> >>> +    unsigned int pg_idx = start_page_idx;
> >>> +    gfn_t gfn;
> >>> +    unsigned int n;
> >>> +    unsigned int m;
> >>> +    p2m_type_t t;
> >>> +    uint64_t addr;
> >>> +
> >>> +    for ( n = 0; n < range_count; n++ )
> >>> +    {
> >>> +        for ( m = 0; m < range[n].page_count; m++ )
> >>> +        {
> >>> +            if ( pg_idx >= shm->page_count )
> >>> +                return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +            addr = read_atomic(&range[n].address);
> >>> +            gfn = gaddr_to_gfn(addr + m * FFA_PAGE_SIZE);
> >>> +            shm->pages[pg_idx] = get_page_from_gfn(d, gfn_x(gfn), &t,
> >>> +   P2M_ALLOC);
> >>> +            if ( !shm->pages[pg_idx] )
> >>> +                return FFA_RET_DENIED;
> >>> +            pg_idx++;
> >>
> >> This increment could be done at the end, why here ?
> >
> > Do you mean after the p2m_is_ram() check? I'll move it there.
>
> yes that would be more natural i think.
>
> >
> >>
> >>> +            /* Only normal RAM for now */
> >>> +            if ( !p2m_is_ram(t) )
> >>> +                return FFA_RET_DENIED;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    *last_page_idx = pg_idx;
> >>> +
> >>> +    return FFA_RET_OK;
> >>> +}
> >>> +
> >>> +static void put_shm_pages(struct ffa_shm_mem *shm)
> >>> +{
> >>> +    unsigned int n;
> >>> +
> >>> +    for ( n = 0; n < shm->page_count && shm->pages[n]; n++ )
> >>> +    {
> >>> +        put_page(shm->pages[n]);
> >>> +        shm->pages[n] = NULL;
> >>
> >> If an error occured during the generation you might have part
> >> of the pages which are NULL already.
> >>
> >> So you should do a if (pages[n] != NULL) here
> >
> > I'm doing that above in the head of the loop, the loop is terminated
> > at the first pages[n] == NULL.
>
> Right, sorry i missed that.
>
> >
> >>
> >>> +    }
> >>> +}
> >>> +
> >>> +static struct ffa_shm_mem *alloc_ffa_shm_mem(struct ffa_ctx *ctx,
> >>> +                                             unsigned int page_count)
> >>> +{
> >>> +    struct ffa_shm_mem *shm;
> >>> +
> >>> +    if ( page_count >= FFA_MAX_SHM_PAGE_COUNT ||
> >>> +         ctx->shm_count >= FFA_MAX_SHM_COUNT )
> >>> +        return NULL;
> >>
> >> Shouldn't you also filter out for page_count = 0 ?
> >
> > Sure, 0 doesn't make sense. But I should probably do it before this
> > function is called since I suppose we'd like to return something
> > different from FFA_RET_NO_MEMORY.
>
> Very true.
>
> >
> >>
> >>> +
> >>> +    shm = xzalloc_flex_struct(struct ffa_shm_mem, pages, page_count);
> >>> +    if ( shm )
> >>> +    {
> >>> +        ctx->shm_count++;
> >>> +        shm->page_count = page_count;
> >>> +    }
> >>> +
> >>> +    return shm;
> >>> +}
> >>> +
> >>> +static void free_ffa_shm_mem(struct ffa_ctx *ctx, struct ffa_shm_mem *shm)
> >>> +{
> >>> +    if ( shm ) {
> >>> +        ASSERT(ctx->shm_count > 0);
> >>> +        ctx->shm_count--;
> >>> +        put_shm_pages(shm);
> >>> +        xfree(shm);
> >>> +    }
> >>> +}
> >>> +
> >>> +static void init_range(struct ffa_address_range *addr_range,
> >>> +                       paddr_t pa)
> >>> +{
> >>> +    memset(addr_range, 0, sizeof(*addr_range));
> >>> +    addr_range->address = pa;
> >>> +    addr_range->page_count = 1;
> >>> +}
> >>> +
> >>> +/*
> >>> + * This function uses the ffa_tx buffer to transmit the memory transaction
> >>> + * descriptor. The function depends ffa_tx_buffer_lock to be used to guard
> >>> + * the buffer from concurrent use.
> >>> + */
> >>> +static int share_shm(struct ffa_shm_mem *shm)
> >>> +{
> >>> +    const uint32_t max_frag_len = ffa_page_count * FFA_PAGE_SIZE;
> >>> +    struct ffa_mem_access *mem_access_array;
> >>> +    struct ffa_mem_transaction_1_1 *descr;
> >>> +    struct ffa_address_range *addr_range;
> >>> +    struct ffa_mem_region *region_descr;
> >>> +    const unsigned int region_count = 1;
> >>> +    void *buf = ffa_tx;
> >>> +    uint32_t frag_len;
> >>> +    uint32_t tot_len;
> >>> +    paddr_t last_pa;
> >>> +    unsigned int n;
> >>> +    paddr_t pa;
> >>> +
> >>> +    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
> >>> +    if ( !shm->page_count )
> >>> +    {
> >>> +        ASSERT_UNREACHABLE();
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>
> >> page_count = 0 should be filtered out before reaching this and this should
> >> only be an assert if you want but no unreachable with a return.
> >
> > I'm adding code to filter out page_count = 0. I'm not sure what you
> > expect here, should I remove the entire check here or what do you
> > want?
>
> As this is checked before, this could just be an assert and not something with a return.

OK, I'll do that.

>
> >
> >>
> >>> +    }
> >>> +
> >>> +    descr = buf;
> >>> +    memset(descr, 0, sizeof(*descr));
> >>> +    descr->sender_id = shm->sender_id;
> >>> +    descr->global_handle = shm->handle;
> >>> +    descr->mem_reg_attr = FFA_NORMAL_MEM_REG_ATTR;
> >>> +    descr->mem_access_count = 1;
> >>> +    descr->mem_access_size = sizeof(*mem_access_array);
> >>> +    descr->mem_access_offs = MEM_ACCESS_OFFSET(0);
> >>> +
> >>> +    mem_access_array = buf + descr->mem_access_offs;
> >>> +    memset(mem_access_array, 0, sizeof(*mem_access_array));
> >>> +    mem_access_array[0].access_perm.endpoint_id = shm->ep_id;
> >>> +    mem_access_array[0].access_perm.perm = FFA_MEM_ACC_RW;
> >>> +    mem_access_array[0].region_offs = REGION_OFFSET(descr->mem_access_count, 0);
> >>> +
> >>> +    region_descr = buf + mem_access_array[0].region_offs;
> >>> +    memset(region_descr, 0, sizeof(*region_descr));
> >>> +    region_descr->total_page_count = shm->page_count;
> >>> +
> >>> +    region_descr->address_range_count = 1;
> >>> +    last_pa = page_to_maddr(shm->pages[0]);
> >>> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> >>> +    {
> >>> +        pa = page_to_maddr(shm->pages[n]);
> >>> +        if ( last_pa + FFA_PAGE_SIZE == pa )
> >>> +            continue;
> >>> +        region_descr->address_range_count++;
> >>> +    }
> >>> +
> >>> +    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
> >>> +                                region_descr->address_range_count);
> >>> +    if ( tot_len > max_frag_len )
> >>> +        return FFA_RET_NOT_SUPPORTED;
> >>> +
> >>> +    addr_range = region_descr->address_range_array;
> >>> +    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
> >>> +    last_pa = page_to_maddr(shm->pages[0]);
> >>> +    init_range(addr_range, last_pa);
> >>> +    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> >>> +    {
> >>> +        pa = page_to_maddr(shm->pages[n]);
> >>> +        if ( last_pa + FFA_PAGE_SIZE == pa )
> >>> +        {
> >>> +            addr_range->page_count++;
> >>> +            continue;
> >>> +        }
> >>> +
> >>> +        frag_len += sizeof(*addr_range);
> >>> +        addr_range++;
> >>> +        init_range(addr_range, pa);
> >>> +    }
> >>> +
> >>> +    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> >>> +}
> >>> +
> >>> +static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> >>> +                                struct ffa_mem_transaction_x *trans)
> >>> +{
> >>> +    uint16_t mem_reg_attr;
> >>> +    uint32_t flags;
> >>> +    uint32_t count;
> >>> +    uint32_t offs;
> >>> +    uint32_t size;
> >>> +
> >>> +    if ( ffa_vers >= FFA_VERSION_1_1 )
> >>> +    {
> >>> +        const struct ffa_mem_transaction_1_1 *descr;
> >>> +
> >>> +        if ( blen < sizeof(*descr) )
> >>> +            return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +        descr = buf;
> >>> +        trans->sender_id = descr->sender_id;
> >>> +        mem_reg_attr = descr->mem_reg_attr;
> >>> +        flags = descr->flags;
> >>> +        trans->global_handle = descr->global_handle;
> >>> +        trans->tag = descr->tag;
> >>> +
> >>> +        count = descr->mem_access_count;
> >>> +        size = descr->mem_access_size;
> >>> +        offs = descr->mem_access_offs;
> >>> +    }
> >>> +    else
> >>> +    {
> >>> +        const struct ffa_mem_transaction_1_0 *descr;
> >>> +
> >>> +        if ( blen < sizeof(*descr) )
> >>> +            return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +        descr = buf;
> >>> +        trans->sender_id = descr->sender_id;
> >>> +        mem_reg_attr = descr->mem_reg_attr;
> >>> +        flags = descr->flags;
> >>> +        trans->global_handle = descr->global_handle;
> >>> +        trans->tag = descr->tag;
> >>> +
> >>> +        count = descr->mem_access_count;
> >>> +        size = sizeof(struct ffa_mem_access);
> >>> +        offs = offsetof(struct ffa_mem_transaction_1_0, mem_access_array);
> >>> +    }
> >>> +    /*
> >>> +     * Make sure that "descr" which is shared with the guest isn't accessed
> >>> +     * again after this point.
> >>> +     */
> >>> +    barrier();
> >>
> >> I am not really following the comment here. You accessed the content of descr
> >> before and it is in the rxtx buffer so why is this needed ?
> >
> > I'm making sure that the compiler doesn't optimize and reorders the
> > reads from memory in funny ways, for instance, reading again after the
> > ifs just below. The RXTX buffer is shared with the guest so it can
> > potentially be updated concurrently by another CPU.
>
> The client is not suppose to modify the buffer during the call, using atomic
> operations here is not really solving any issue if this is modified while we use it right ?

We are guarding against a TOCTOU (time-of-check to time-of-use) type
of attack or bug.

>
> >
> >>
> >>> +
> >>> +    /*
> >>> +     * We're doing a rough check to see that no information is lost when
> >>> +     * tranfering the values into a struct ffa_mem_transaction_x below. The
> >>> +     * fields in struct ffa_mem_transaction_x are wide enough to hold any
> >>> +     * valid value so being out of range means that something is wrong.
> >>> +     */
> >>> +    if ( mem_reg_attr > UINT8_MAX || flags > UINT8_MAX || size > UINT8_MAX ||
> >>> +        count > UINT8_MAX || offs > UINT16_MAX )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    /* Check that the endpoint memory access descriptor array fits */
> >>> +    if ( size * count + offs > blen )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    trans->mem_reg_attr = mem_reg_attr;
> >>> +    trans->flags = flags;
> >>> +    trans->mem_access_size = size;
> >>> +    trans->mem_access_count = count;
> >>> +    trans->mem_access_offs = offs;
> >>> +
> >>> +    return 0;
> >>> +}
> >>> +
> >>> +static void handle_mem_share(struct cpu_user_regs *regs)
> >>> +{
> >>> +    uint32_t tot_len = get_user_reg(regs, 1);
> >>> +    uint32_t frag_len = get_user_reg(regs, 2);
> >>> +    uint64_t addr = get_user_reg(regs, 3);
> >>> +    uint32_t page_count = get_user_reg(regs, 4);
> >>> +    const struct ffa_mem_region *region_descr;
> >>> +    const struct ffa_mem_access *mem_access;
> >>> +    struct ffa_mem_transaction_x trans;
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    struct ffa_shm_mem *shm = NULL;
> >>> +    unsigned int last_page_idx = 0;
> >>> +    register_t handle_hi = 0;
> >>> +    register_t handle_lo = 0;
> >>> +    int ret = FFA_RET_DENIED;
> >>> +    uint32_t range_count;
> >>> +    uint32_t region_offs;
> >>> +
> >>> +    /*
> >>> +     * We're only accepting memory transaction descriptors via the rx/tx
> >>> +     * buffer.
> >>
> >> Is this a limitation coming fomr the spec or from the implementation ?
> >
> > This is just a limitation in the implementation.
> >
> >>
> >>> +     */
> >>> +    if ( addr )
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_set_ret;
> >>> +    }
> >>> +
> >>> +    /* Check that fragment length doesn't exceed total length */
> >>> +    if ( frag_len > tot_len )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out_set_ret;
> >>> +    }
> >>> +
> >>> +    /* We currently only support a single fragment */
> >>
> >> It would make sense to add some text at the beginning of the files listing
> >> the current limitations of the implementation.
> >
> > That's quite a bit to keep track of, especially since it will change
> > with each patch. If it's important perhaps we can summarize that in a
> > final commit instead. By the way, this particular limitation is
> > removed in a later patch.
>
> I am more thinking at the end of the serie to have one place with the current
> state and limitations of the implementation.
>
> We cannot really expect someone to browse all comments to get what is
>  supported or not.

OK

>
> >
> >>
> >>> +    if ( frag_len != tot_len )
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_set_ret;
> >>> +    }
> >>> +
> >>> +    spin_lock(&ctx->lock);
> >>> +
> >>> +    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> >>> +        goto out_unlock;
> >>> +
> >>> +    if ( !ffa_page_count )
> >>> +    {
> >>> +        ret = FFA_RET_NO_MEMORY;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    ret = read_mem_transaction(ctx->guest_vers, ctx->tx, frag_len, &trans);
> >>> +    if ( ret )
> >>> +        goto out_unlock;
> >>> +
> >>> +    if ( trans.mem_reg_attr != FFA_NORMAL_MEM_REG_ATTR )
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    /* Only supports sharing it with one SP for now */
> >>
> >> Also a limitation to list.
> >
> > OK
> >
> >>
> >>> +    if ( trans.mem_access_count != 1 )
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    if ( trans.sender_id != get_vm_id(d) )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    /* Check that it fits in the supplied data */
> >>> +    if ( trans.mem_access_offs + trans.mem_access_size > frag_len )
> >>> +        goto out_unlock;
> >>> +
> >>
> >> Why are you using atomic operations to access rxtx buffer after here ?
> >
> > To limit how the compiler can reorder the reads from memory.
>
> As stated before, don't we assume that the buffer is not modified by the client
>  while we use it ?

No, as I'm saying above we're guarding against it.

Thanks,
Jens

>
> >
> >>
> >>> +    mem_access = ctx->tx + trans.mem_access_offs;
> >>> +    if ( read_atomic(&mem_access->access_perm.perm) != FFA_MEM_ACC_RW )
> >>
> >> Also a limitation to list.
> >
> > OK
>
> Cheers
> Bertrand
>
> >
> > Thanks,
> > Jens
> >
> >>
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    region_offs = read_atomic(&mem_access->region_offs);
> >>> +    if ( sizeof(*region_descr) + region_offs > frag_len )
> >>> +    {
> >>> +        ret = FFA_RET_NOT_SUPPORTED;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>> +    region_descr = ctx->tx + region_offs;
> >>> +    range_count = read_atomic(&region_descr->address_range_count);
> >>> +    page_count = read_atomic(&region_descr->total_page_count);
> >>> +
> >>> +    shm = alloc_ffa_shm_mem(ctx, page_count);
> >>> +    if ( !shm )
> >>> +    {
> >>> +        ret = FFA_RET_NO_MEMORY;
> >>> +        goto out_unlock;
> >>> +    }
> >>> +    shm->sender_id = trans.sender_id;
> >>> +    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> >>> +
> >>> +    /*
> >>> +     * Check that the Composite memory region descriptor fits.
> >>> +     */
> >>> +    if ( sizeof(*region_descr) + region_offs +
> >>> +         range_count * sizeof(struct ffa_address_range) > frag_len )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +
> >>> +    ret = get_shm_pages(d, shm, region_descr->address_range_array, range_count,
> >>> +                        0, &last_page_idx);
> >>> +    if ( ret )
> >>> +        goto out;
> >>> +    if ( last_page_idx != shm->page_count )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +
> >>> +    /* Note that share_shm() uses our tx buffer */
> >>> +    spin_lock(&ffa_tx_buffer_lock);
> >>> +    ret = share_shm(shm);
> >>> +    spin_unlock(&ffa_tx_buffer_lock);
> >>> +    if ( ret )
> >>> +        goto out;
> >>> +
> >>> +    list_add_tail(&shm->list, &ctx->shm_list);
> >>> +
> >>> +    uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> >>> +
> >>> +out:
> >>> +    if ( ret )
> >>> +        free_ffa_shm_mem(ctx, shm);
> >>> +out_unlock:
> >>> +    spin_unlock(&ctx->lock);
> >>> +
> >>> +out_set_ret:
> >>> +    if ( ret == 0)
> >>> +            set_regs_success(regs, handle_lo, handle_hi);
> >>> +    else
> >>> +            set_regs_error(regs, ret);
> >>> +}
> >>> +
> >>> static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>> {
> >>>    uint32_t fid = get_user_reg(regs, 0);
> >>> @@ -818,6 +1299,12 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>> #endif
> >>>        handle_msg_send_direct_req(regs, fid);
> >>>        return true;
> >>> +    case FFA_MEM_SHARE_32:
> >>> +#ifdef CONFIG_ARM_64
> >>> +    case FFA_MEM_SHARE_64:
> >>> +#endif
> >>> +        handle_mem_share(regs);
> >>> +        return true;
> >>>
> >>>    default:
> >>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> >>> @@ -857,6 +1344,8 @@ static int ffa_domain_init(struct domain *d)
> >>>        }
> >>>    }
> >>>
> >>> +    INIT_LIST_HEAD(&ctx->shm_list);
> >>> +
> >>>    d->arch.tee = ctx;
> >>>
> >>>    return 0;
> >>> @@ -1012,11 +1501,13 @@ static bool ffa_probe(void)
> >>>         !check_mandatory_feature(FFA_RX_RELEASE) ||
> >>> #ifdef CONFIG_ARM_64
> >>>         !check_mandatory_feature(FFA_RXTX_MAP_64) ||
> >>> +         !check_mandatory_feature(FFA_MEM_SHARE_64) ||
> >>> #endif
> >>> #ifdef CONFIG_ARM_32
> >>>         !check_mandatory_feature(FFA_RXTX_MAP_32) ||
> >>> #endif
> >>>         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> >>> +         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> >>>         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >>>        return false;
> >>>
> >>> --
> >>> 2.34.1
> >>
> >>
> >> Cheers
> >> Bertrand
>
>

Re: [XEN PATCH v7 20/20] xen/arm: ffa: support sharing large memory ranges

[Jens Wiklander]

Hi Bertrand,

On Wed, Mar 15, 2023 at 2:35 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Jens,
>
> > On 15 Mar 2023, at 12:47, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > Hi Bertrand,
> >
> > On Wed, Mar 15, 2023 at 11:13 AM Bertrand Marquis
> > <Bertrand.Marquis@arm.com> wrote:
> >>
> >> Hi Jens,
> >>
> >>> On 22 Feb 2023, at 16:33, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>
> >>> Adds support for sharing large memory ranges transmitted in fragments
> >>> using FFA_MEM_FRAG_TX.
> >>>
> >>> The implementation is the bare minimum to be able to communicate with
> >>> OP-TEE running as an SPMC at S-EL1.
> >>>
> >>> Adds a check that the SP supports the needed FF-A feature
> >>> FFA_MEM_FRAG_TX.
> >>>
> >>> Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
> >>> ---
> >>> xen/arch/arm/tee/ffa.c | 254 ++++++++++++++++++++++++++++++++++++++---
> >>> 1 file changed, 240 insertions(+), 14 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/tee/ffa.c b/xen/arch/arm/tee/ffa.c
> >>> index 3557edc455d0..72c0249d8cad 100644
> >>> --- a/xen/arch/arm/tee/ffa.c
> >>> +++ b/xen/arch/arm/tee/ffa.c
> >>> @@ -326,6 +326,7 @@ struct ffa_ctx {
> >>>    uint32_t guest_vers;
> >>>    bool tx_is_mine;
> >>>    bool interrupted;
> >>> +    struct list_head frag_list;
> >>>    struct list_head shm_list;
> >>>    unsigned int shm_count;
> >>>    spinlock_t lock;
> >>> @@ -340,6 +341,18 @@ struct ffa_shm_mem {
> >>>    struct page_info *pages[];
> >>> };
> >>
> >> We start to have a lot of fields here.
> >> It might be useful to have some quick documentation
> >> in comment here as some names are a bit generic.
> >> For example "frag_list" does not say much.
> >
> > I'll add some comments.
> >
> >>
> >>>
> >>> +struct mem_frag_state {
> >>> +    struct list_head list;
> >>> +    struct ffa_shm_mem *shm;
> >>> +    uint32_t range_count;
> >>> +    unsigned int current_page_idx;
> >>> +    unsigned int frag_offset;
> >>> +    unsigned int range_offset;
> >>> +    const uint8_t *buf;
> >>> +    unsigned int buf_size;
> >>> +    struct ffa_address_range range;
> >>> +};
> >>
> >> same here, at a first glance it is not quite clear why
> >> a fragment needs that much info. Some seem to only
> >> be needed during the transaction but do not need to be saved.
> >
> > This struct is only used during the transaction, so this is freed once
> > the entire memory transaction descriptor has been processed.
> >
> >>
> >>> +
> >>> /* Negotiated FF-A version to use with the SPMC */
> >>> static uint32_t ffa_version __ro_after_init;
> >>>
> >>> @@ -512,6 +525,36 @@ static int32_t ffa_mem_share(uint32_t tot_len, uint32_t frag_len,
> >>>    }
> >>> }
> >>>
> >>> +static int32_t ffa_mem_frag_tx(uint64_t handle, uint32_t frag_len,
> >>> +                               uint16_t sender_id)
> >>> +{
> >>> +    struct arm_smccc_1_2_regs arg = {
> >>> +        .a0 = FFA_MEM_FRAG_TX,
> >>> +        .a1 = handle & UINT32_MAX,
> >>> +        .a2 = handle >> 32,
> >>> +        .a3 = frag_len,
> >>> +        .a4 = (uint32_t)sender_id << 16,
> >>> +    };
> >>> +    struct arm_smccc_1_2_regs resp;
> >>> +
> >>> +    arm_smccc_1_2_smc(&arg, &resp);
> >>> +
> >>> +    switch ( resp.a0 )
> >>> +    {
> >>> +    case FFA_ERROR:
> >>> +        if ( resp.a2 )
> >>> +            return resp.a2;
> >>> +        else
> >>> +            return FFA_RET_NOT_SUPPORTED;
> >>> +    case FFA_SUCCESS_32:
> >>> +        return FFA_RET_OK;
> >>> +    case FFA_MEM_FRAG_RX:
> >>> +        return resp.a3;
> >>> +    default:
> >>> +            return FFA_RET_NOT_SUPPORTED;
> >>> +    }
> >>> +}
> >>> +
> >>> static int32_t ffa_mem_reclaim(uint32_t handle_lo, uint32_t handle_hi,
> >>>                               uint32_t flags)
> >>> {
> >>> @@ -586,6 +629,14 @@ static void set_regs_success(struct cpu_user_regs *regs, uint32_t w2,
> >>>    set_regs(regs, FFA_SUCCESS_32, 0, w2, w3, 0, 0, 0, 0);
> >>> }
> >>>
> >>> +static void set_regs_frag_rx(struct cpu_user_regs *regs, uint32_t handle_lo,
> >>> +                             uint32_t handle_hi, uint32_t frag_offset,
> >>> +                             uint16_t sender_id)
> >>> +{
> >>> +    set_regs(regs, FFA_MEM_FRAG_RX, handle_lo, handle_hi, frag_offset,
> >>> +             (uint32_t)sender_id << 16, 0, 0, 0);
> >>> +}
> >>> +
> >>> static void handle_version(struct cpu_user_regs *regs)
> >>> {
> >>>    struct domain *d = current->domain;
> >>> @@ -955,6 +1006,8 @@ static int share_shm(struct ffa_shm_mem *shm)
> >>>    paddr_t last_pa;
> >>>    unsigned int n;
> >>>    paddr_t pa;
> >>> +    bool first;
> >>> +    int ret;
> >>>
> >>>    ASSERT(spin_is_locked(&ffa_tx_buffer_lock));
> >>>    if ( !shm->page_count )
> >>> @@ -994,13 +1047,23 @@ static int share_shm(struct ffa_shm_mem *shm)
> >>>
> >>>    tot_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count,
> >>>                                region_descr->address_range_count);
> >>> -    if ( tot_len > max_frag_len )
> >>> -        return FFA_RET_NOT_SUPPORTED;
> >>>
> >>> +    /*
> >>> +     * Sharing memory with secure world may have to be done with multiple
> >>> +     * calls depending on how many address ranges will be needed. If we're
> >>> +     * sharing physically contiguous memory we will only need one range but
> >>> +     * we will also need to deal with the worst case where all physical
> >>> +     * pages are non-contiguous. For the first batch of address ranges we
> >>> +     * call ffa_mem_share() and for all that follows ffa_mem_frag_tx().
> >>> +     *
> >>> +     * We use frag_len to keep track of how far into the transmit buffer we
> >>> +     * have gone.
> >>> +     */
> >>>    addr_range = region_descr->address_range_array;
> >>>    frag_len = ADDR_RANGE_OFFSET(descr->mem_access_count, region_count, 1);
> >>>    last_pa = page_to_maddr(shm->pages[0]);
> >>>    init_range(addr_range, last_pa);
> >>> +    first = true;
> >>>    for ( n = 1; n < shm->page_count; last_pa = pa, n++ )
> >>>    {
> >>>        pa = page_to_maddr(shm->pages[n]);
> >>> @@ -1010,12 +1073,34 @@ static int share_shm(struct ffa_shm_mem *shm)
> >>>            continue;
> >>>        }
> >>>
> >>> -        frag_len += sizeof(*addr_range);
> >>> -        addr_range++;
> >>> +        if ( frag_len == max_frag_len )
> >>> +        {
> >>> +            if ( first )
> >>> +            {
> >>> +                ret = ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> >>> +                first = false;
> >>> +            }
> >>> +            else
> >>> +            {
> >>> +                ret = ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> >>> +            }
> >>> +            if ( ret <= 0 )
> >>> +                return ret;
> >>> +            frag_len = sizeof(*addr_range);
> >>> +            addr_range = buf;
> >>> +        }
> >>> +        else
> >>> +        {
> >>> +            frag_len += sizeof(*addr_range);
> >>> +            addr_range++;
> >>> +        }
> >>>        init_range(addr_range, pa);
> >>>    }
> >>>
> >>> -    return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> >>> +    if ( first )
> >>> +        return ffa_mem_share(tot_len, frag_len, 0, 0, &shm->handle);
> >>> +    else
> >>> +        return ffa_mem_frag_tx(shm->handle, frag_len, shm->sender_id);
> >>> }
> >>>
> >>> static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> >>> @@ -1092,8 +1177,53 @@ static int read_mem_transaction(uint32_t ffa_vers, const void *buf, size_t blen,
> >>>    return 0;
> >>> }
> >>>
> >>> +static int add_mem_share_frag(struct mem_frag_state *s, unsigned int offs,
> >>> +                              unsigned int frag_len)
> >>> +{
> >>> +    struct domain *d = current->domain;
> >>> +    unsigned int o = offs;
> >>> +    unsigned int l;
> >>> +    int ret;
> >>> +
> >>> +    if ( frag_len < o )
> >>> +        return FFA_RET_INVALID_PARAMETERS;
> >>> +
> >>> +    /* Fill up the first struct ffa_address_range */
> >>> +    l = min_t(unsigned int, frag_len - o, sizeof(s->range) - s->range_offset);
> >>> +    memcpy((uint8_t *)&s->range + s->range_offset, s->buf + o, l);
> >>> +    s->range_offset += l;
> >>> +    o += l;
> >>> +    if ( s->range_offset != sizeof(s->range) )
> >>> +        goto out;
> >>> +    s->range_offset = 0;
> >>> +
> >>> +    while ( true )
> >>> +    {
> >>> +        ret = get_shm_pages(d, s->shm, &s->range, 1, s->current_page_idx,
> >>> +                            &s->current_page_idx);
> >>> +        if ( ret )
> >>> +            return ret;
> >>> +        if ( s->range_count == 1 )
> >>> +            return 0;
> >>> +        s->range_count--;
> >>> +        if ( frag_len - o < sizeof(s->range) )
> >>> +            break;
> >>> +        memcpy(&s->range, s->buf + o, sizeof(s->range));
> >>> +        o += sizeof(s->range);
> >>> +    }
> >>> +
> >>> +    /* Collect any remaining bytes for the next struct ffa_address_range */
> >>> +    s->range_offset = frag_len - o;
> >>> +    memcpy(&s->range, s->buf + o, frag_len - o);
> >>> +out:
> >>> +    s->frag_offset += frag_len;
> >>> +
> >>> +    return s->frag_offset;
> >>> +}
> >>> +
> >>> static void handle_mem_share(struct cpu_user_regs *regs)
> >>> {
> >>> +    static uint64_t next_handle = FFA_HANDLE_HYP_FLAG;
> >>>    uint32_t tot_len = get_user_reg(regs, 1);
> >>>    uint32_t frag_len = get_user_reg(regs, 2);
> >>>    uint64_t addr = get_user_reg(regs, 3);
> >>> @@ -1128,13 +1258,6 @@ static void handle_mem_share(struct cpu_user_regs *regs)
> >>>        goto out_set_ret;
> >>>    }
> >>>
> >>> -    /* We currently only support a single fragment */
> >>> -    if ( frag_len != tot_len )
> >>> -    {
> >>> -        ret = FFA_RET_NOT_SUPPORTED;
> >>> -        goto out_set_ret;
> >>> -    }
> >>> -
> >>>    spin_lock(&ctx->lock);
> >>>
> >>>    if ( frag_len > ctx->page_count * FFA_PAGE_SIZE )
> >>> @@ -1195,11 +1318,41 @@ static void handle_mem_share(struct cpu_user_regs *regs)
> >>>    if ( !shm )
> >>>    {
> >>>        ret = FFA_RET_NO_MEMORY;
> >>> -        goto out_unlock;
> >>> +        goto out;
> >>
> >> Why is this changed ?
> >> You still have no shm here so call free_sha_shm does not make sense
> >
> > Good catch, I'll fix it.
> >
> >>
> >>>    }
> >>>    shm->sender_id = trans.sender_id;
> >>>    shm->ep_id = read_atomic(&mem_access->access_perm.endpoint_id);
> >>>
> >>> +    if ( frag_len != tot_len )
> >>> +    {
> >>> +        struct mem_frag_state *s = xzalloc(struct mem_frag_state);
> >>> +
> >>> +        if ( !s )
> >>> +        {
> >>> +            ret = FFA_RET_NO_MEMORY;
> >>> +            goto out;
> >>> +        }
> >>> +        s->shm = shm;
> >>> +        s->range_count = range_count;
> >>> +        s->buf = ctx->tx;
> >>> +        s->buf_size = ffa_page_count * FFA_PAGE_SIZE;
> >>> +        ret = add_mem_share_frag(s, sizeof(*region_descr)  + region_offs,
> >>> +                                 frag_len);
> >>> +        if ( ret <= 0 )
> >>> +        {
> >>> +            xfree(s);
> >>> +            if ( ret < 0 )
> >>> +                goto out;
> >>> +        }
> >>> +        else
> >>> +        {
> >>> +            shm->handle = next_handle++;
> >>> +            uint64_to_regpair(&handle_hi, &handle_lo, shm->handle);
> >>> +            list_add_tail(&s->list, &ctx->frag_list);
> >>> +        }
> >>> +        goto out_unlock;
> >>> +    }
> >>> +
> >>>    /*
> >>>     * Check that the Composite memory region descriptor fits.
> >>>     */
> >>> @@ -1238,7 +1391,75 @@ out_unlock:
> >>>    spin_unlock(&ctx->lock);
> >>>
> >>> out_set_ret:
> >>> -    if ( ret == 0)
> >>> +    if ( ret > 0 )
> >>> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, trans.sender_id);
> >>> +    else if ( ret == 0)
> >>> +            set_regs_success(regs, handle_lo, handle_hi);
> >>> +    else
> >>> +            set_regs_error(regs, ret);
> >>> +}
> >>> +
> >>> +static struct mem_frag_state *find_frag_state(struct ffa_ctx *ctx,
> >>> +                                              uint64_t handle)
> >>> +{
> >>> +    struct mem_frag_state *s;
> >>> +
> >>> +    list_for_each_entry(s, &ctx->frag_list, list)
> >>> +        if ( s->shm->handle == handle )
> >>> +            return s;
> >>> +
> >>> +    return NULL;
> >>> +}
> >>> +
> >>> +static void handle_mem_frag_tx(struct cpu_user_regs *regs)
> >>> +{
> >>> +    struct domain *d = current->domain;
> >>> +    struct ffa_ctx *ctx = d->arch.tee;
> >>> +    uint32_t frag_len = get_user_reg(regs, 3);
> >>> +    uint32_t handle_lo = get_user_reg(regs, 1);
> >>> +    uint32_t handle_hi = get_user_reg(regs, 2);
> >>> +    uint64_t handle = regpair_to_uint64(handle_hi, handle_lo);
> >>> +    struct mem_frag_state *s;
> >>> +    uint16_t sender_id = 0;
> >>> +    int ret;
> >>> +
> >>> +    spin_lock(&ctx->lock);
> >>> +    s = find_frag_state(ctx, handle);
> >>> +    if ( !s )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +    sender_id = s->shm->sender_id;
> >>> +
> >>> +    if ( frag_len > s->buf_size )
> >>> +    {
> >>> +        ret = FFA_RET_INVALID_PARAMETERS;
> >>> +        goto out;
> >>> +    }
> >>> +
> >>> +    ret = add_mem_share_frag(s, 0, frag_len);
> >>> +    if ( ret == 0 )
> >>> +    {
> >>> +        /* Note that share_shm() uses our tx buffer */
> >>> +        spin_lock(&ffa_tx_buffer_lock);
> >>> +        ret = share_shm(s->shm);
> >>> +        spin_unlock(&ffa_tx_buffer_lock);
> >>> +        if ( ret == 0 )
> >>> +            list_add_tail(&s->shm->list, &ctx->shm_list);
> >>> +        else
> >>> +            free_ffa_shm_mem(ctx, s->shm);
> >>> +    }
> >>> +    else if ( ret < 0 )
> >>> +        free_ffa_shm_mem(ctx, s->shm);
> >>
> >>
> >> If there is not error the stuff allocated are kept but i do not see
> >> where/when they would be freed or used.
> >> Could you explain why we need to save all those ?
> >
> > s->shm is the final shared memory object which is added to the list of
> > shared memory objects when the transaction is completed successfully.
> > The fragment state, s, is kept as long as the transaction is ongoing.
> > Once the transaction is completed successfully or due to a failure
> > it's freed.
> >
> > The specification doesn't say what to do if an invalid frag_len is
> > detected, except that we should return FFA_RET_INVALID_PARAMETERS.
> > Currently, we just return an error, but keep the state. Another option
> > is to free the state instead since the caller might have lost track of
> > the state.
>
> There is no solution for the client to "continue" anyway so I think we should
> properly cleanup in all exit conditions.

OK, I'll do that.

Thanks,
Jens

>
> Cheers
> Bertrand
>
> >
> > Thanks,
> > Jens
> >
> >>
> >> Cheers
> >> Bertrand
> >>
> >>> +    list_del(&s->list);
> >>> +    xfree(s);
> >>> +out:
> >>> +    spin_unlock(&ctx->lock);
> >>> +
> >>> +    if ( ret > 0 )
> >>> +            set_regs_frag_rx(regs, handle_lo, handle_hi, ret, sender_id);
> >>> +    else if ( ret == 0)
> >>>            set_regs_success(regs, handle_lo, handle_hi);
> >>>    else
> >>>            set_regs_error(regs, ret);
> >>> @@ -1357,6 +1578,9 @@ static bool ffa_handle_call(struct cpu_user_regs *regs)
> >>>        else
> >>>            set_regs_success(regs, 0, 0);
> >>>        return true;
> >>> +    case FFA_MEM_FRAG_TX:
> >>> +        handle_mem_frag_tx(regs);
> >>> +        return true;
> >>>
> >>>    default:
> >>>        gprintk(XENLOG_ERR, "ffa: unhandled fid 0x%x\n", fid);
> >>> @@ -1396,6 +1620,7 @@ static int ffa_domain_init(struct domain *d)
> >>>        }
> >>>    }
> >>>
> >>> +    INIT_LIST_HEAD(&ctx->frag_list);
> >>>    INIT_LIST_HEAD(&ctx->shm_list);
> >>>
> >>>    d->arch.tee = ctx;
> >>> @@ -1560,6 +1785,7 @@ static bool ffa_probe(void)
> >>> #endif
> >>>         !check_mandatory_feature(FFA_RXTX_UNMAP) ||
> >>>         !check_mandatory_feature(FFA_MEM_SHARE_32) ||
> >>> +         !check_mandatory_feature(FFA_MEM_FRAG_TX) ||
> >>>         !check_mandatory_feature(FFA_MEM_RECLAIM) ||
> >>>         !check_mandatory_feature(FFA_MSG_SEND_DIRECT_REQ_32) )
> >>>        return false;
> >>> --
> >>> 2.34.1
>
>