[PATCH v2 00/11] Drivers for gunyah hypervisor

[PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

Gunyah is a Type-1 hypervisor independent of any
high-level OS kernel, and runs in a higher CPU privilege level. It does
not depend on any lower-privileged OS kernel/code for its core
functionality. This increases its security and can support a much smaller
trusted computing base than a Type-2 hypervisor.

Gunyah is an open source hypervisor. The source repo is available at
https://github.com/quic/gunyah-hypervisor.

The diagram below shows the architecture.

::

        Primary VM           Secondary VMs
     +-----+ +-----+  | +-----+ +-----+ +-----+
     |     | |     |  | |     | |     | |     |
 EL0 | APP | | APP |  | | APP | | APP | | APP |
     |     | |     |  | |     | |     | |     |
     +-----+ +-----+  | +-----+ +-----+ +-----+
 ---------------------|-------------------------
     +--------------+ | +----------------------+
     |              | | |                      |
 EL1 | Linux Kernel | | |Linux kernel/Other OS |   ...
     |              | | |                      |
     +--------------+ | +----------------------+
 --------hvc/smc------|------hvc/smc------------
     +----------------------------------------+
     |                                        |
 EL2 |            Gunyah Hypervisor           |
     |                                        |
     +----------------------------------------+

Gunyah provides these following features.

- Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on
physical CPUs and enables time-sharing of the CPUs.
- Memory Management: Gunyah tracks memory ownership and use of all memory
under its control. Memory partitioning between VMs is a fundamental
security feature.
- Interrupt Virtualization: All interrupts are handled in the hypervisor
and routed to the assigned VM.
- Inter-VM Communication: There are several different mechanisms provided
for communicating between VMs.
- Device Virtualization: Para-virtualization of devices is supported using
inter-VM communication. Low level system features and devices such as
interrupt controllers are supported with emulation where required.

Elliot Berman (11):
  docs: gunyah: Introduce Gunyah Hypervisor
  dt-bindings: Add binding for gunyah hypervisor
  arm64: gunyah: Add Gunyah hypercalls ABI
  gunyah: Common types and error codes for Gunyah hypercalls
  virt: gunyah: Add sysfs nodes
  virt: gunyah: Add capabilities bus and devices
  gunyah: msgq: Add Gunyah message queues
  gunyah: rsc_mgr: Add resource manager RPC core
  gunyah: rsc_mgr: Add auxiliary devices for console
  gunyah: rsc_mgr: Add RPC for console services
  gunyah: Add tty console driver for RM Console Serivces

 .../ABI/testing/sysfs-hypervisor-gunyah       |  37 +
 .../bindings/firmware/gunyah-hypervisor.yaml  |  84 +++
 Documentation/virt/gunyah/index.rst           |  99 +++
 Documentation/virt/gunyah/message-queue.rst   |  52 ++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |  12 +
 arch/arm64/include/asm/gunyah.h               | 142 ++++
 drivers/virt/Kconfig                          |   1 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/gunyah/Kconfig                   |  24 +
 drivers/virt/gunyah/Makefile                  |   8 +
 drivers/virt/gunyah/device.c                  | 108 +++
 drivers/virt/gunyah/gunyah_private.h          |  18 +
 drivers/virt/gunyah/msgq.c                    | 223 ++++++
 drivers/virt/gunyah/rsc_mgr.c                 | 682 ++++++++++++++++++
 drivers/virt/gunyah/rsc_mgr.h                 |  56 ++
 drivers/virt/gunyah/rsc_mgr_console.c         | 405 +++++++++++
 drivers/virt/gunyah/rsc_mgr_rpc.c             | 151 ++++
 drivers/virt/gunyah/sysfs.c                   | 176 +++++
 include/linux/gunyah.h                        | 133 ++++
 include/linux/gunyah_rsc_mgr.h                |  45 ++
 21 files changed, 2458 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-hypervisor-gunyah
 create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 create mode 100644 Documentation/virt/gunyah/index.rst
 create mode 100644 Documentation/virt/gunyah/message-queue.rst
 create mode 100644 arch/arm64/include/asm/gunyah.h
 create mode 100644 drivers/virt/gunyah/Kconfig
 create mode 100644 drivers/virt/gunyah/Makefile
 create mode 100644 drivers/virt/gunyah/device.c
 create mode 100644 drivers/virt/gunyah/gunyah_private.h
 create mode 100644 drivers/virt/gunyah/msgq.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.h
 create mode 100644 drivers/virt/gunyah/rsc_mgr_console.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c
 create mode 100644 drivers/virt/gunyah/sysfs.c
 create mode 100644 include/linux/gunyah.h
 create mode 100644 include/linux/gunyah_rsc_mgr.h

-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/11] docs: gunyah: Introduce Gunyah Hypervisor

[Elliot Berman]

Gunyah is an open-source Type-1 hypervisor developed by Qualcomm. It
does not depend on any lower-privileged OS/kernel code for its core
functionality. This increases its security and can support a smaller
trusted computing based when compared to Type-2 hypervisors.

Add documentation describing the Gunyah hypervisor and the main
components of the Gunyah hypervisor which are of interest to Linux
virtualization development.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 Documentation/virt/gunyah/index.rst         | 92 +++++++++++++++++++++
 Documentation/virt/gunyah/message-queue.rst | 52 ++++++++++++
 Documentation/virt/index.rst                |  1 +
 MAINTAINERS                                 |  7 ++
 4 files changed, 152 insertions(+)
 create mode 100644 Documentation/virt/gunyah/index.rst
 create mode 100644 Documentation/virt/gunyah/message-queue.rst

diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst
new file mode 100644
index 000000000000..e7bb2b14543e
--- /dev/null
+++ b/Documentation/virt/gunyah/index.rst
@@ -0,0 +1,92 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================
+Gunyah Hypervisor
+=================
+
+.. toctree::
+   :maxdepth: 1
+
+   message-queue
+
+Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in
+a higher CPU privilege level. It does not depend on any lower-privileged operating system
+for its core functionality. This increases its security and can support a much smaller
+trusted computing base than a Type-2 hypervisor.
+
+Gunyah is an open source hypervisor. The source repo is available at
+https://github.com/quic/gunyah-hypervisor.
+
+Gunyah provides these following features.
+
+- Scheduling:
+  A scheduler for virtual CPUs (VCPUs) on physical CPUs and enables time-sharing
+  of the CPUs.
+- Memory Management:
+  APIs handling memory, abstracted as objects, limiting direct use of physical
+  addresses. Memory ownership and usage tracking of all memory under its control.
+  Memory partitioning between VMs is a fundamental security feature.
+- Interrupt Virtualization:
+  Uses CPU hardware interrupt virtualization capabilities. Interrupts are handled
+  in the hypervisor and routed to the assigned VM.
+- Inter-VM Communication:
+  There are several different mechanisms provided for communicating between VMs.
+- Virtual platform:
+  Architectural devices such as interrupt controllers and CPU timers are directly provided
+  by the hypervisor as well as core virtual platform devices and system APIs such as ARM PSCI.
+- Device Virtualization:
+  Para-virtualization of devices is supported using inter-VM communication.
+
+Architectures supported
+=======================
+AArch64 with a GIC
+
+Resources and Capabilities
+==========================
+
+Some services or resources provided by the Gunyah hypervisor are described by capability IDs.
+For instance, inter-VM communication is performed with doorbells and message queues. The specific
+instance of a doorbell is described by a capability ID. These devices are described in Linux as a
+struct gunyah_device.
+
+High level management of these resources is performed by the resource manager VM. RM informs a
+guest VM about resources it can access through either the device tree or via guest-initiated RPC.
+
+Resource Manager
+================
+
+The resource manager (RM) is a privileged application VM supporting the Gunyah Hypervisor.
+It provides policy enforcement aspects of the virtualization system. The resource manager can
+be treated as an extension of the Hypervisor but is separated to its own partition to ensure
+that the hypervisor layer itself remains small and secure and to maintain a separation of policy
+and mechanism in the platform. On arm64, RM runs at NS-EL1 similar to other virtual machines.
+
+Communication with the resource manager from each guest VM happens with message-queue.rst. Details
+about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c
+
+::
+  +-------+   +--------+   +--------+
+  |  RM   |   |  VM_A  |   |  VM_B  |
+  +-.-.-.-+   +---.----+   +---.----+
+    | |           |            |
+  +-.-.-----------.------------.----+
+  | | \==========/             |    |
+  |  \========================/     |
+  |            Gunyah               |
+  +---------------------------------+
+
+The source for the resource manager is available at https://github.com/quic/gunyah-resource-manager.
+
+The resource manager provides the following features:
+
+- Generate device-tree overlay
+- VM creation and deletion
+- VM device-tree management
+- VM access control policy
+- Interrupt routing configuration
+
+When booting a virtual machine which uses a devicetree, resource manager overlays a
+/hypervisor node. This node can let Linux know it is running as a Gunyah guest VM,
+how to communicate with resource manager, and basic description and capabilities of
+this VM. See Documentation/devicetree/bindings/gunyah/qcom,hypervisor.yml for a description
+of this node.
diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst
new file mode 100644
index 000000000000..afd405f3a5e1
--- /dev/null
+++ b/Documentation/virt/gunyah/message-queue.rst
@@ -0,0 +1,52 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Message Queues
+==============
+Message queue is a simple low-capacity IPC channel between two VMs. It is
+intended for sending small control and configuration messages. Each message
+queue object is unidirectional, so a full-duplex IPC channel requires a pair of
+objects.
+
+Messages can be up to 240 bytes in length. Longer messages require a further
+protocol on top of the message queue messages themselves. For instance, communication
+with the resource manager adds a header field for sending longer messages via multiple
+message fragments.
+
+The diagram below shows how message queue works. A typical configuration involves
+2 message queues. Message queue 1 allows VM_A to send messages to VM_B. Message
+queue 2 allows VM_B to send messages to VM_A.
+
+1. VM_A sends a message of up to 240 bytes in length. It raises a hypercall
+   with the message to inform the hypervisor to add the message to
+   message queue 1's queue.
+2. Gunyah raises the corresponding interrupt for VM_B when any of these happens:
+   a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case.
+   b. Explicility with gh_msgq_push command from VM_A.
+   c. Message queue has reached a threshold depth.
+3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer.
+
+For VM_B to send a message to VM_A, the process is identical, except that hypercalls
+reference message queue 2's capability ID.
+
+::
+
+      +---------------+         +-----------------+         +---------------+
+      |      VM_A     |         |Gunyah hypervisor|         |      VM_B     |
+      |               |         |                 |         |               |
+      |               |         |                 |         |               |
+      |               |   Tx    |                 |         |               |
+      |               |-------->|                 | Rx vIRQ |               |
+      |gh_msgq_send() | Tx vIRQ |Message queue 1  |-------->|gh_msgq_recv() |
+      |               |<------- |                 |         |               |
+      |               |         |                 |         |               |
+      | Message Queue |         |                 |         | Message Queue |
+      | driver        |         |                 |         | driver        |
+      |               |         |                 |         |               |
+      |               |         |                 |         |               |
+      |               |         |                 |   Tx    |               |
+      |               | Rx vIRQ |                 |<--------|               |
+      |gh_msgq_recv() |<--------|Message queue 2  | Tx vIRQ |gh_msgq_send() |
+      |               |         |                 |-------->|               |
+      |               |         |                 |         |               |
+      |               |         |                 |         |               |
+      +---------------+         +-----------------+         +---------------+
diff --git a/Documentation/virt/index.rst b/Documentation/virt/index.rst
index 492f0920b988..dd4e8ef284eb 100644
--- a/Documentation/virt/index.rst
+++ b/Documentation/virt/index.rst
@@ -14,6 +14,7 @@ Linux Virtualization Support
    ne_overview
    acrn/index
    coco/sev-guest
+   gunyah/index
 
 .. only:: html and subproject
 
diff --git a/MAINTAINERS b/MAINTAINERS
index 04ec80ee7352..18fb034526e1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8736,6 +8736,13 @@ L:	linux-efi@vger.kernel.org
 S:	Maintained
 F:	block/partitions/efi.*
 
+GUNYAH HYPERVISOR DRIVER
+M:	Elliot Berman <quic_eberman@quicinc.com>
+M:	Murali Nalajala <quic_mnalajal@quicinc.com>
+L:	linux-arm-msm@vger.kernel.org
+S:	Maintained
+F:	Documentation/virt/gunyah/
+
 HABANALABS PCI DRIVER
 M:	Oded Gabbay <ogabbay@kernel.org>
 S:	Supported
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/11] docs: gunyah: Introduce Gunyah Hypervisor

[Jeffrey Hugo]

On Mon, Aug 1, 2022 at 3:16 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 04ec80ee7352..18fb034526e1 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8736,6 +8736,13 @@ L:       linux-efi@vger.kernel.org
>  S:     Maintained
>  F:     block/partitions/efi.*
>
> +GUNYAH HYPERVISOR DRIVER
> +M:     Elliot Berman <quic_eberman@quicinc.com>
> +M:     Murali Nalajala <quic_mnalajal@quicinc.com>
> +L:     linux-arm-msm@vger.kernel.org
> +S:     Maintained

Supported?  Sure seems like something you'd be paid to maintain....

> +F:     Documentation/virt/gunyah/
> +
>  HABANALABS PCI DRIVER
>  M:     Oded Gabbay <ogabbay@kernel.org>
>  S:     Supported
> --
> 2.25.1
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/11] docs: gunyah: Introduce Gunyah Hypervisor

[Bagas Sanjaya]

[-- Attachment #1.1: Type: text/plain, Size: 1508 bytes --]

On Mon, Aug 01, 2022 at 02:12:30PM -0700, Elliot Berman wrote:
> +Communication with the resource manager from each guest VM happens with message-queue.rst. Details
> +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c
> +
> +::
> +  +-------+   +--------+   +--------+
> +  |  RM   |   |  VM_A  |   |  VM_B  |
> +  +-.-.-.-+   +---.----+   +---.----+
> +    | |           |            |
> +  +-.-.-----------.------------.----+
> +  | | \==========/             |    |
> +  |  \========================/     |
> +  |            Gunyah               |
> +  +---------------------------------+
> +
 
Hi,

The diagram above triggers htmldocs warnings:

Documentation/virt/gunyah/index.rst:71: WARNING: Unexpected indentation.
Documentation/virt/gunyah/index.rst:72: WARNING: Block quote ends without a blank line; unexpected unindent.

I have applied the fixup:

diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst
index 95ba9b71ab30d2..b74f7a6f9d4904 100644
--- a/Documentation/virt/gunyah/index.rst
+++ b/Documentation/virt/gunyah/index.rst
@@ -65,6 +65,7 @@ Communication with the resource manager from each guest VM happens with message-
 about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c
 
 ::
+
   +-------+   +--------+   +--------+
   |  RM   |   |  VM_A  |   |  VM_B  |
   +-.-.-.-+   +---.----+   +---.----+

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 273 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/11] docs: gunyah: Introduce Gunyah Hypervisor

[Elliot Berman]

On 8/4/2022 8:18 PM, Bagas Sanjaya wrote:
> On Mon, Aug 01, 2022 at 02:12:30PM -0700, Elliot Berman wrote:
>> +Communication with the resource manager from each guest VM happens with message-queue.rst. Details
>> +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c
>> +
>> +::
>> +  +-------+   +--------+   +--------+
>> +  |  RM   |   |  VM_A  |   |  VM_B  |
>> +  +-.-.-.-+   +---.----+   +---.----+
>> +    | |           |            |
>> +  +-.-.-----------.------------.----+
>> +  | | \==========/             |    |
>> +  |  \========================/     |
>> +  |            Gunyah               |
>> +  +---------------------------------+
>> +
>   
> Hi,
> 
> The diagram above triggers htmldocs warnings:
> 
> Documentation/virt/gunyah/index.rst:71: WARNING: Unexpected indentation.
> Documentation/virt/gunyah/index.rst:72: WARNING: Block quote ends without a blank line; unexpected unindent.
> 
> I have applied the fixup:
> 
> diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst
> index 95ba9b71ab30d2..b74f7a6f9d4904 100644
> --- a/Documentation/virt/gunyah/index.rst
> +++ b/Documentation/virt/gunyah/index.rst
> @@ -65,6 +65,7 @@ Communication with the resource manager from each guest VM happens with message-
>   about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c
>   
>   ::
> +
>     +-------+   +--------+   +--------+
>     |  RM   |   |  VM_A  |   |  VM_B  |
>     +-.-.-.-+   +---.----+   +---.----+
> 
> Thanks.
> 

Thanks! Updated for v3.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 01/11] docs: gunyah: Introduce Gunyah Hypervisor

[kernel test robot]

Hi Elliot,

I love your patch! Perhaps something to improve:

[auto build test WARNING on arm64/for-next/core]
[also build test WARNING on v5.19]
[cannot apply to linus/master next-20220805]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Elliot-Berman/Drivers-for-gunyah-hypervisor/20220802-051534
base:   https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
reproduce: make htmldocs

If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> Documentation/virt/gunyah/index.rst:71: WARNING: Unexpected indentation.
>> Documentation/virt/gunyah/index.rst:72: WARNING: Block quote ends without a blank line; unexpected unindent.

vim +71 Documentation/virt/gunyah/index.rst

    66	
    67	::
    68	  +-------+   +--------+   +--------+
    69	  |  RM   |   |  VM_A  |   |  VM_B  |
    70	  +-.-.-.-+   +---.----+   +---.----+
  > 71	    | |           |            |
  > 72	  +-.-.-----------.------------.----+
    73	  | | \==========/             |    |
    74	  |  \========================/     |
    75	  |            Gunyah               |
    76	  +---------------------------------+
    77	

-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 02/11] dt-bindings: Add binding for gunyah hypervisor

[Elliot Berman]

When Linux is booted as a guest under the Gunyah hypervisor, Gunyah
applies a devicetree overlay describing the virtual platform
configuration of the guest VM, such as the message queue capability IDs
for communicating with the Resource Manager. Add the DT bindings that
Gunyah adheres for the hypervisor node and message queues.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 .../bindings/firmware/gunyah-hypervisor.yaml  | 84 +++++++++++++++++++
 MAINTAINERS                                   |  1 +
 2 files changed, 85 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml

diff --git a/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
new file mode 100644
index 000000000000..e50d932e768c
--- /dev/null
+++ b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
@@ -0,0 +1,84 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/firmware/gunyah-hypervisor.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Gunyah Hypervisor
+
+maintainers:
+  - Murali Nalajala <quic_mnalajal@quicinc.com>
+  - Elliot Berman <quic_eberman@quicinc.com>
+
+description: |+
+  On systems which support devicetree, Gunyah generates and overlays a deviceetree overlay which
+  describes the basic configuration of the hypervisor. Virtual machines use this information for
+  initial discovery that they are running as a Gunyah guest VM.
+  See also: https://github.com/quic/gunyah-resource-manager/blob/develop/src/vm_creation/dto_construct.c
+
+properties:
+  compatible:
+    oneOf:
+      - items:
+          - const: gunyah-hypervisor-1.0
+          - const: gunyah-hypervisor
+
+  "#address-cells":
+    description: Number of cells needed to represent 64-bit capability IDs.
+    const: 2
+  "#size-cells":
+    description: must be 0, because capability IDs are not memory address
+                  ranges and do not have a size.
+    const: 0
+
+patternProperties:
+  "^gunyah-resource-mgr(@.*)?":
+    type: object
+    description:
+      Resource Manager node which is required to communicate to Resource
+      Manager VM using Gunyah Message Queues.
+
+    properties:
+      compatible:
+        oneOf:
+          - items:
+              - const: gunyah-resource-manager-1-0
+              - const: gunyah-resource-manager
+      reg:
+        items:
+          - description: Gunyah capability ID of the TX message queue
+          - description: Gunyah capability ID of the RX message queue
+      interrupts:
+        items:
+          - description: Interrupt for the TX message queue
+          - description: Interrupt for the RX message queue
+    additionalProperties: false
+    required:
+      - compatible
+      - reg
+      - interrupts
+
+additionalProperties: false
+
+required:
+  - compatible
+  - "#address-cells"
+  - "#size-cells"
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+
+    hypervisor {
+        #address-cells = <2>;
+        #size-cells = <0>;
+        compatible = "gunyah-hypervisor-1.0", "gunyah-hypervisor";
+
+        gunyah-resource-mgr@1 {
+            compatible = "gunyah-resource-manager-1-0", "gunyah-resource-manager";
+            interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>, /* TX full IRQ */
+                         <GIC_SPI 4 IRQ_TYPE_EDGE_RISING>; /* RX empty IRQ */
+            reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>;
+                  /* TX, RX cap ids */
+        };
+    };
diff --git a/MAINTAINERS b/MAINTAINERS
index 18fb034526e1..0cd12ea6c11c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8741,6 +8741,7 @@ M:	Elliot Berman <quic_eberman@quicinc.com>
 M:	Murali Nalajala <quic_mnalajal@quicinc.com>
 L:	linux-arm-msm@vger.kernel.org
 S:	Maintained
+F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
 
 HABANALABS PCI DRIVER
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 02/11] dt-bindings: Add binding for gunyah hypervisor

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> When Linux is booted as a guest under the Gunyah hypervisor, Gunyah
> applies a devicetree overlay describing the virtual platform
> configuration of the guest VM, such as the message queue capability IDs
> for communicating with the Resource Manager. Add the DT bindings that
> Gunyah adheres for the hypervisor node and message queues.
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   .../bindings/firmware/gunyah-hypervisor.yaml  | 84 +++++++++++++++++++
>   MAINTAINERS                                   |  1 +
>   2 files changed, 85 insertions(+)
>   create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> 
> diff --git a/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> new file mode 100644
> index 000000000000..e50d932e768c
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> @@ -0,0 +1,84 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/firmware/gunyah-hypervisor.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Gunyah Hypervisor
> +
> +maintainers:
> +  - Murali Nalajala <quic_mnalajal@quicinc.com>
> +  - Elliot Berman <quic_eberman@quicinc.com>
> +
> +description: |+
> +  On systems which support devicetree, Gunyah generates and overlays a deviceetree overlay which
> +  describes the basic configuration of the hypervisor. Virtual machines use this information for
> +  initial discovery that they are running as a Gunyah guest VM.
> +  See also: https://github.com/quic/gunyah-resource-manager/blob/develop/src/vm_creation/dto_construct.c
> +
> +properties:
> +  compatible:
> +    oneOf:
> +      - items:
> +          - const: gunyah-hypervisor-1.0
> +          - const: gunyah-hypervisor

Just:
  - items:
   - const: gunyah-hypervisor-1.0
   - const: gunyah-hypervisor

oneOf is not necessary

> +
> +  "#address-cells":
> +    description: Number of cells needed to represent 64-bit capability IDs.
> +    const: 2
> +  "#size-cells":
> +    description: must be 0, because capability IDs are not memory address
> +                  ranges and do not have a size.
> +    const: 0
> +
> +patternProperties:
> +  "^gunyah-resource-mgr(@.*)?":
> +    type: object
> +    description:
> +      Resource Manager node which is required to communicate to Resource
> +      Manager VM using Gunyah Message Queues.
> +
> +    properties:
> +      compatible:
> +        oneOf:

Again, please drop the oneOf.

> +          - items:
> +              - const: gunyah-resource-manager-1-0
> +              - const: gunyah-resource-manager
> +      reg:
> +        items:
> +          - description: Gunyah capability ID of the TX message queue
> +          - description: Gunyah capability ID of the RX message queue
> +      interrupts:
> +        items:
> +          - description: Interrupt for the TX message queue
> +          - description: Interrupt for the RX message queue
> +    additionalProperties: false
> +    required:
> +      - compatible
> +      - reg
> +      - interrupts
> +
> +additionalProperties: false
> +
> +required:
> +  - compatible
> +  - "#address-cells"
> +  - "#size-cells"
> +
> +examples:
> +  - |
> +    #include <dt-bindings/interrupt-controller/arm-gic.h>
> +
> +    hypervisor {
> +        #address-cells = <2>;
> +        #size-cells = <0>;
> +        compatible = "gunyah-hypervisor-1.0", "gunyah-hypervisor";
> +
> +        gunyah-resource-mgr@1 {
> +            compatible = "gunyah-resource-manager-1-0", "gunyah-resource-manager";
> +            interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>, /* TX full IRQ */
> +                         <GIC_SPI 4 IRQ_TYPE_EDGE_RISING>; /* RX empty IRQ */
> +            reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>;
> +                  /* TX, RX cap ids */

Do these cap ids persist from platform to platform? Then you might want 
to define generic values for these caps.

Also it feels like you are misusing the #address-cells/reg properties. 
For example, the device should have its @NNNN to be equal to the first 
addres. But you are using the second reg.

I think that using a separate property with /bits/ 64 might be a better 
solution.

> +        };
> +    };
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 18fb034526e1..0cd12ea6c11c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8741,6 +8741,7 @@ M:	Elliot Berman <quic_eberman@quicinc.com>
>   M:	Murali Nalajala <quic_mnalajal@quicinc.com>
>   L:	linux-arm-msm@vger.kernel.org
>   S:	Maintained
> +F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>   F:	Documentation/virt/gunyah/
>   
>   HABANALABS PCI DRIVER


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 02/11] dt-bindings: Add binding for gunyah hypervisor

[Krzysztof Kozlowski]

On 01/08/2022 23:12, Elliot Berman wrote:
> When Linux is booted as a guest under the Gunyah hypervisor, Gunyah
> applies a devicetree overlay describing the virtual platform
> configuration of the guest VM, such as the message queue capability IDs
> for communicating with the Resource Manager. Add the DT bindings that
> Gunyah adheres for the hypervisor node and message queues.

Previously you explained that this cannot be discoverable, because EL2
hypervisor follows micro-kernel architecture and it is small. Size of
your code is not a reason to push things to DT... This is a kind of
explanation like: we did not add auto-discoverability, because we don't
want to add more code.

> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>  .../bindings/firmware/gunyah-hypervisor.yaml  | 84 +++++++++++++++++++
>  MAINTAINERS                                   |  1 +
>  2 files changed, 85 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> 
> diff --git a/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> new file mode 100644
> index 000000000000..e50d932e768c
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
> @@ -0,0 +1,84 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/firmware/gunyah-hypervisor.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Gunyah Hypervisor
> +
> +maintainers:
> +  - Murali Nalajala <quic_mnalajal@quicinc.com>
> +  - Elliot Berman <quic_eberman@quicinc.com>
> +
> +description: |+
> +  On systems which support devicetree, Gunyah generates and overlays a deviceetree overlay which
> +  describes the basic configuration of the hypervisor. Virtual machines use this information for
> +  initial discovery that they are running as a Gunyah guest VM.
> +  See also: https://github.com/quic/gunyah-resource-manager/blob/develop/src/vm_creation/dto_construct.c
> +
> +properties:
> +  compatible:
> +    oneOf:
> +      - items:
> +          - const: gunyah-hypervisor-1.0
> +          - const: gunyah-hypervisor
> +
> +  "#address-cells":
> +    description: Number of cells needed to represent 64-bit capability IDs.
> +    const: 2

Blank line between definition of each property.

> +  "#size-cells":
> +    description: must be 0, because capability IDs are not memory address
> +                  ranges and do not have a size.
> +    const: 0
> +
> +patternProperties:
> +  "^gunyah-resource-mgr(@.*)?":
> +    type: object
> +    description:
> +      Resource Manager node which is required to communicate to Resource
> +      Manager VM using Gunyah Message Queues.
> +
> +    properties:
> +      compatible:
> +        oneOf:
> +          - items:
> +              - const: gunyah-resource-manager-1-0
> +              - const: gunyah-resource-manager

Blank line.

> +      reg:
> +        items:
> +          - description: Gunyah capability ID of the TX message queue
> +          - description: Gunyah capability ID of the RX message queue

Blank line.

> +      interrupts:
> +        items:
> +          - description: Interrupt for the TX message queue
> +          - description: Interrupt for the RX message queue

Blank line.

> +    additionalProperties: false

Blank line.

> +    required:
> +      - compatible
> +      - reg
> +      - interrupts
> +
> +additionalProperties: false
> +
> +required:
> +  - compatible
> +  - "#address-cells"
> +  - "#size-cells"
> +
> +examples:
> +  - |
> +    #include <dt-bindings/interrupt-controller/arm-gic.h>
> +
> +    hypervisor {
> +        #address-cells = <2>;
> +        #size-cells = <0>;
> +        compatible = "gunyah-hypervisor-1.0", "gunyah-hypervisor";
> +
> +        gunyah-resource-mgr@1 {
> +            compatible = "gunyah-resource-manager-1-0", "gunyah-resource-manager";
> +            interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>, /* TX full IRQ */
> +                         <GIC_SPI 4 IRQ_TYPE_EDGE_RISING>; /* RX empty IRQ */
> +            reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>;
> +                  /* TX, RX cap ids */
> +        };
> +    };


Best regards,
Krzysztof

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 03/11] arm64: gunyah: Add Gunyah hypercalls ABI

[Elliot Berman]

Add initial support to perform Gunyah hypercalls. The arm64 ABI for
Gunyah hypercalls generally follows the SMC Calling Convention.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 MAINTAINERS                     |   1 +
 arch/arm64/include/asm/gunyah.h | 134 ++++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+)
 create mode 100644 arch/arm64/include/asm/gunyah.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 0cd12ea6c11c..02f97ac90cdf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8743,6 +8743,7 @@ L:	linux-arm-msm@vger.kernel.org
 S:	Maintained
 F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
+F:	arch/arm64/include/asm/gunyah.h
 
 HABANALABS PCI DRIVER
 M:	Oded Gabbay <ogabbay@kernel.org>
diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h
new file mode 100644
index 000000000000..4820e9389f40
--- /dev/null
+++ b/arch/arm64/include/asm/gunyah.h
@@ -0,0 +1,134 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+#ifndef __ASM_GUNYAH_H
+#define __ASM_GUNYAH_H
+
+#include <linux/arm-smccc.h>
+#include <linux/types.h>
+
+#define GH_CALL_TYPE_PLATFORM_CALL		0
+#define GH_CALL_TYPE_HYPERCALL			2
+#define GH_CALL_TYPE_SERVICE			3
+#define GH_CALL_TYPE_SHIFT			14
+#define GH_CALL_FUNCTION_NUM_MASK		0x3fff
+
+#define GH_SERVICE(fn)		ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_32, \
+						   ARM_SMCCC_OWNER_VENDOR_HYP, \
+						   (GH_CALL_TYPE_SERVICE << GH_CALL_TYPE_SHIFT) \
+							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
+
+#define GH_HYPERCALL(fn)	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \
+						   ARM_SMCCC_OWNER_VENDOR_HYP, \
+						   (GH_CALL_TYPE_HYPERCALL << GH_CALL_TYPE_SHIFT) \
+							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
+
+#define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
+
+#define __gh_count_args(...)						\
+	___gh_count_args(_, ## __VA_ARGS__, 8, 7, 6, 5, 4, 3, 2, 1, 0)
+
+#define __gh_skip_0(...)		__VA_ARGS__
+#define __gh_skip_1(a, ...)	__VA_ARGS__
+#define __gh_skip_2(a, b, ...)	__VA_ARGS__
+#define __gh_skip_3(a, b, c, ...)	__VA_ARGS__
+#define __gh_skip_4(a, b, c, d, ...)	__VA_ARGS__
+#define __gh_skip_5(a, b, c, d, e, ...)	__VA_ARGS__
+#define __gh_skip_6(a, b, c, d, e, f, ...)	__VA_ARGS__
+#define __gh_skip_7(a, b, c, d, e, f, g, ...)	__VA_ARGS__
+#define __gh_skip_8(a, b, c, d, e, f, g, h, ...)	__VA_ARGS__
+#define __gh_to_res(nargs, ...)		__gh_skip_ ## nargs (__VA_ARGS__)
+
+#define __gh_declare_arg_0(...)
+
+#define __gh_declare_arg_1(arg1, ...)						\
+	.a1 = (arg1)
+
+#define __gh_declare_arg_2(arg1, arg2, ...)					\
+	__gh_declare_arg_1(arg1),						\
+	.a2 = (arg2)
+
+#define __gh_declare_arg_3(arg1, arg2, arg3, ...)				\
+	__gh_declare_arg_2(arg1, arg2),						\
+	.a3 = (arg3)
+
+#define __gh_declare_arg_4(arg1, arg2, arg3, arg4, ...)				\
+	__gh_declare_arg_3(arg1, arg2, arg3),					\
+	.a4 = (arg4)
+
+#define __gh_declare_arg_5(arg1, arg2, arg3, arg4, arg5, ...)			\
+	__gh_declare_arg_4(arg1, arg2, arg3, arg4),				\
+	.a5 = (arg5)
+
+#define __gh_declare_arg_6(arg1, arg2, arg3, arg4, arg5, arg6, ...)		\
+	__gh_declare_arg_5(arg1, arg2, arg3, arg4, arg5),			\
+	.a6 = (arg6)
+
+#define __gh_declare_arg_7(arg1, arg2, arg3, arg4, arg5, arg6, arg7, ...)	\
+	__gh_declare_arg_6(arg1, arg2, arg3, arg4, arg5, arg6),			\
+	.a7 = (arg7)
+
+#define __gh_declare_arg_8(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, ...)	\
+	__gh_declare_arg_7(arg1, arg2, arg3, arg4, arg5, arg6, arg7),		\
+	.a8 = (arg8)
+
+#define ___gh_declare_args(nargs)	__gh_declare_arg_ ## nargs
+#define __gh_declare_args(nargs)	___gh_declare_args(nargs)
+#define _gh_declare_args(nargs, ...) __gh_declare_args(nargs)(__VA_ARGS__)
+
+#define __gh_assign_res_0(...)
+
+#define __gh_assign_res_1(r1)					\
+	r1 = __res.a0
+
+#define __gh_assign_res_2(r1, r2)				\
+	__gh_assign_res_1(r1);					\
+	r2 = __res.a1
+
+#define __gh_assign_res_3(r1, r2, r3)				\
+	__gh_assign_res_2(r1, r2);				\
+	r3 = __res.a2
+
+#define __gh_assign_res_4(r1, r2, r3, r4)			\
+	__gh_assign_res_3(r1, r2, r3);				\
+	r4 = __res.a3
+
+#define __gh_assign_res_5(r1, r2, r3, r4, r5)			\
+	__gh_assign_res_4(r1, r2, r3, r4);			\
+	r5 = __res.a4
+
+#define __gh_assign_res_6(r1, r2, r3, r4, r5, r6)		\
+	__gh_assign_res_5(r1, r2, r3, r4, r5);			\
+	r6 = __res.a5
+
+#define __gh_assign_res_7(r1, r2, r3, r4, r5, r6, r7)		\
+	__gh_assign_res_6(r1, r2, r3, r4, r5, r6);		\
+	r7 = __res.a6
+
+#define __gh_assign_res_8(r1, r2, r3, r4, r5, r6, r7, r8)	\
+	__gh_assign_res_7(r1, r2, r3, r4, r5, r6, r7);		\
+	r8 = __res.a7
+
+#define ___gh_assign_res(nargs)	__gh_assign_res_ ## nargs
+#define __gh_assign_res(nargs)	___gh_assign_res(nargs)
+#define _gh_assign_res(...) __gh_assign_res(__gh_count_args(__VA_ARGS__))(__VA_ARGS__)
+
+/**
+ * arch_gh_hypercall() - Performs an AArch64-specific call into hypervisor using Gunyah ABI
+ * @hcall_num: Hypercall function ID to invoke
+ * @nargs: Number of input arguments
+ * @...: First nargs are the input arguments. Remaining arguments are output variables.
+ */
+#define arch_gh_hypercall(hcall_num, nargs, ...)				\
+	do {									\
+		struct arm_smccc_1_2_regs __res;					\
+		struct arm_smccc_1_2_regs __args = {				\
+			.a0 = hcall_num,					\
+			_gh_declare_args(nargs, __VA_ARGS__)			\
+		};								\
+		arm_smccc_1_2_hvc(&__args, &__res);					\
+		_gh_assign_res(__gh_to_res(nargs, __VA_ARGS__));		\
+	} while (0)
+
+#endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 03/11] arm64: gunyah: Add Gunyah hypercalls ABI

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Add initial support to perform Gunyah hypercalls. The arm64 ABI for
> Gunyah hypercalls generally follows the SMC Calling Convention.

Seeing a c&p (or c&rework) from arm-smccc.h, could you please describe:

1) Why can't you use the existing arm_smccc_1_1_hvc()? I checked, you 
don't seem to be getting more than 4 values back.

2) If #1 is not possible, why can't you add necessary glue code to the 
arm-smccc.h (as your code to support nargs/multiple return values is 
generic enough) and use corresponding macro in asm/gunyah.h ?

-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 03/11] arm64: gunyah: Add Gunyah hypercalls ABI

[Elliot Berman]

Hi Dmitry,

On 8/2/2022 6:34 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Add initial support to perform Gunyah hypercalls. The arm64 ABI for
>> Gunyah hypercalls generally follows the SMC Calling Convention.
> 
> Seeing a c&p (or c&rework) from arm-smccc.h, could you please describe:
> 
> 1) Why can't you use the existing arm_smccc_1_1_hvc()? I checked, you 
> don't seem to be getting more than 4 values back.
> 

The Gunyah APIs can return up to 8 values. As you observed though, these 
initial patches are only using the first 4 values back. I'd like to use 
the larger v1.2 so I don't need to update later.

> 2) If #1 is not possible, why can't you add necessary glue code to the 
> arm-smccc.h (as your code to support nargs/multiple return values is 
> generic enough) and use corresponding macro in asm/gunyah.h ?
> 

I think the code here may be considered Gunyah-specific as I am limiting 
to 8 arguments and return values. If I add to arm-smccc.h, I would need 
to expand out to x17. Does it make sense to add another SMCCC 1.2 
interface to arm-smccc.h?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 03/11] arm64: gunyah: Add Gunyah hypercalls ABI

[Dmitry Baryshkov]

On Thu, 4 Aug 2022 at 00:15, Elliot Berman <quic_eberman@quicinc.com> wrote:
>
> Hi Dmitry,
>
> On 8/2/2022 6:34 AM, Dmitry Baryshkov wrote:
> > On 02/08/2022 00:12, Elliot Berman wrote:
> >> Add initial support to perform Gunyah hypercalls. The arm64 ABI for
> >> Gunyah hypercalls generally follows the SMC Calling Convention.
> >
> > Seeing a c&p (or c&rework) from arm-smccc.h, could you please describe:
> >
> > 1) Why can't you use the existing arm_smccc_1_1_hvc()? I checked, you
> > don't seem to be getting more than 4 values back.
> >
>
> The Gunyah APIs can return up to 8 values. As you observed though, these
> initial patches are only using the first 4 values back. I'd like to use
> the larger v1.2 so I don't need to update later.

I'd suggest following the hyperv example here. It uses arm_smccc_1_1
when possible and 1_2 only when required. Note, that if you are using
the 1_2 call all the times, you are somewhat wasting the cpu cycles by
always copying x0...x17 instead of just x0...x3.

> > 2) If #1 is not possible, why can't you add necessary glue code to the
> > arm-smccc.h (as your code to support nargs/multiple return values is
> > generic enough) and use corresponding macro in asm/gunyah.h ?
> >
>
> I think the code here may be considered Gunyah-specific as I am limiting
> to 8 arguments and return values. If I add to arm-smccc.h, I would need
> to expand out to x17. Does it make sense to add another SMCCC 1.2
> interface to arm-smccc.h?

You do not need to handle 8 arguments at this moment, until the moment
you really need this code, I'd suggest postponing this change. And
when you need it, having the generic code is better than having the
gh-specific code.


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 04/11] gunyah: Common types and error codes for Gunyah hypercalls

[Elliot Berman]

Add architecture-independent standard error codes, types, and macros for
Gunyah hypercalls.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 MAINTAINERS            |  1 +
 include/linux/gunyah.h | 75 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)
 create mode 100644 include/linux/gunyah.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 02f97ac90cdf..2e4f1d9ed47b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8744,6 +8744,7 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
 F:	arch/arm64/include/asm/gunyah.h
+F:	include/linux/gunyah.h
 
 HABANALABS PCI DRIVER
 M:	Oded Gabbay <ogabbay@kernel.org>
diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
new file mode 100644
index 000000000000..69931a0f5736
--- /dev/null
+++ b/include/linux/gunyah.h
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#ifndef _GUNYAH_H
+#define _GUNYAH_H
+
+#include <linux/types.h>
+#include <linux/errno.h>
+#include <asm/gunyah.h>
+
+typedef u64 gh_capid_t;
+
+/* Common Gunyah macros */
+#define GH_CAPID_INVAL	U64_MAX
+
+#define GH_ERROR_OK			0
+#define GH_ERROR_UNIMPLEMENTED		-1
+#define GH_ERROR_RETRY			-2
+
+#define GH_ERROR_ARG_INVAL		1
+#define GH_ERROR_ARG_SIZE		2
+#define GH_ERROR_ARG_ALIGN		3
+
+#define GH_ERROR_NOMEM			10
+
+#define GH_ERROR_ADDR_OVFL		20
+#define GH_ERROR_ADDR_UNFL		21
+#define GH_ERROR_ADDR_INVAL		22
+
+#define GH_ERROR_DENIED			30
+#define GH_ERROR_BUSY			31
+#define GH_ERROR_IDLE			32
+
+#define GH_ERROR_IRQ_BOUND		40
+#define GH_ERROR_IRQ_UNBOUND		41
+
+#define GH_ERROR_CSPACE_CAP_NULL	50
+#define GH_ERROR_CSPACE_CAP_REVOKED	51
+#define GH_ERROR_CSPACE_WRONG_OBJ_TYPE	52
+#define GH_ERROR_CSPACE_INSUF_RIGHTS	53
+#define GH_ERROR_CSPACE_FULL		54
+
+#define GH_ERROR_MSGQUEUE_EMPTY		60
+#define GH_ERROR_MSGQUEUE_FULL		61
+
+static inline int gh_remap_error(int gh_error)
+{
+	switch (gh_error) {
+	case GH_ERROR_OK:
+		return 0;
+	case GH_ERROR_NOMEM:
+		return -ENOMEM;
+	case GH_ERROR_DENIED:
+	case GH_ERROR_CSPACE_CAP_NULL:
+	case GH_ERROR_CSPACE_CAP_REVOKED:
+	case GH_ERROR_CSPACE_WRONG_OBJ_TYPE:
+	case GH_ERROR_CSPACE_INSUF_RIGHTS:
+	case GH_ERROR_CSPACE_FULL:
+		return -EACCES;
+	case GH_ERROR_BUSY:
+	case GH_ERROR_IDLE:
+		return -EBUSY;
+	case GH_ERROR_IRQ_BOUND:
+	case GH_ERROR_IRQ_UNBOUND:
+	case GH_ERROR_MSGQUEUE_FULL:
+	case GH_ERROR_MSGQUEUE_EMPTY:
+		return -EPERM;
+	default:
+		return -EINVAL;
+	}
+}
+
+#endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 04/11] gunyah: Common types and error codes for Gunyah hypercalls

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Add architecture-independent standard error codes, types, and macros for
> Gunyah hypercalls.
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   MAINTAINERS            |  1 +
>   include/linux/gunyah.h | 75 ++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 76 insertions(+)
>   create mode 100644 include/linux/gunyah.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 02f97ac90cdf..2e4f1d9ed47b 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8744,6 +8744,7 @@ S:	Maintained
>   F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>   F:	Documentation/virt/gunyah/
>   F:	arch/arm64/include/asm/gunyah.h
> +F:	include/linux/gunyah.h
>   
>   HABANALABS PCI DRIVER
>   M:	Oded Gabbay <ogabbay@kernel.org>
> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
> new file mode 100644
> index 000000000000..69931a0f5736
> --- /dev/null
> +++ b/include/linux/gunyah.h
> @@ -0,0 +1,75 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#ifndef _GUNYAH_H
> +#define _GUNYAH_H
> +
> +#include <linux/types.h>
> +#include <linux/errno.h>
> +#include <asm/gunyah.h>
> +
> +typedef u64 gh_capid_t;
> +
> +/* Common Gunyah macros */
> +#define GH_CAPID_INVAL	U64_MAX
> +
> +#define GH_ERROR_OK			0

Is there any semantic difference between GH_ERROR_foo < 0 and 
GH_ERROR_bar > 0 ?

> +#define GH_ERROR_UNIMPLEMENTED		-1
> +#define GH_ERROR_RETRY			-2
> +
> +#define GH_ERROR_ARG_INVAL		1
> +#define GH_ERROR_ARG_SIZE		2
> +#define GH_ERROR_ARG_ALIGN		3
> +
> +#define GH_ERROR_NOMEM			10
> +
> +#define GH_ERROR_ADDR_OVFL		20
> +#define GH_ERROR_ADDR_UNFL		21
> +#define GH_ERROR_ADDR_INVAL		22
> +
> +#define GH_ERROR_DENIED			30
> +#define GH_ERROR_BUSY			31
> +#define GH_ERROR_IDLE			32
> +
> +#define GH_ERROR_IRQ_BOUND		40
> +#define GH_ERROR_IRQ_UNBOUND		41
> +
> +#define GH_ERROR_CSPACE_CAP_NULL	50
> +#define GH_ERROR_CSPACE_CAP_REVOKED	51
> +#define GH_ERROR_CSPACE_WRONG_OBJ_TYPE	52
> +#define GH_ERROR_CSPACE_INSUF_RIGHTS	53
> +#define GH_ERROR_CSPACE_FULL		54
> +
> +#define GH_ERROR_MSGQUEUE_EMPTY		60
> +#define GH_ERROR_MSGQUEUE_FULL		61
> +
> +static inline int gh_remap_error(int gh_error)
> +{
> +	switch (gh_error) {
> +	case GH_ERROR_OK:
> +		return 0;
> +	case GH_ERROR_NOMEM:
> +		return -ENOMEM;
> +	case GH_ERROR_DENIED:
> +	case GH_ERROR_CSPACE_CAP_NULL:
> +	case GH_ERROR_CSPACE_CAP_REVOKED:
> +	case GH_ERROR_CSPACE_WRONG_OBJ_TYPE:
> +	case GH_ERROR_CSPACE_INSUF_RIGHTS:
> +	case GH_ERROR_CSPACE_FULL:
> +		return -EACCES;
> +	case GH_ERROR_BUSY:
> +	case GH_ERROR_IDLE:
> +		return -EBUSY;
> +	case GH_ERROR_IRQ_BOUND:
> +	case GH_ERROR_IRQ_UNBOUND:
> +	case GH_ERROR_MSGQUEUE_FULL:
> +	case GH_ERROR_MSGQUEUE_EMPTY:
> +		return -EPERM;
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +#endif


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 04/11] gunyah: Common types and error codes for Gunyah hypercalls

[Elliot Berman]

On 8/2/2022 12:33 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Add architecture-independent standard error codes, types, and macros for
>> Gunyah hypercalls.
>>
>> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
>> ---
>>   MAINTAINERS            |  1 +
>>   include/linux/gunyah.h | 75 ++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 76 insertions(+)
>>   create mode 100644 include/linux/gunyah.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 02f97ac90cdf..2e4f1d9ed47b 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -8744,6 +8744,7 @@ S:    Maintained
>>   F:    Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>>   F:    Documentation/virt/gunyah/
>>   F:    arch/arm64/include/asm/gunyah.h
>> +F:    include/linux/gunyah.h
>>   HABANALABS PCI DRIVER
>>   M:    Oded Gabbay <ogabbay@kernel.org>
>> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
>> new file mode 100644
>> index 000000000000..69931a0f5736
>> --- /dev/null
>> +++ b/include/linux/gunyah.h
>> @@ -0,0 +1,75 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights 
>> reserved.
>> + */
>> +
>> +#ifndef _GUNYAH_H
>> +#define _GUNYAH_H
>> +
>> +#include <linux/types.h>
>> +#include <linux/errno.h>
>> +#include <asm/gunyah.h>
>> +
>> +typedef u64 gh_capid_t;
>> +
>> +/* Common Gunyah macros */
>> +#define GH_CAPID_INVAL    U64_MAX
>> +
>> +#define GH_ERROR_OK            0
> 
> Is there any semantic difference between GH_ERROR_foo < 0 and 
> GH_ERROR_bar > 0 ?
> 

GH_ERROR_foo < 0 comes from Gunyah's plumbing for handling hypercalls. 
GH_ERROR_bar > 0 comes from the hypercall itself.

>> +#define GH_ERROR_UNIMPLEMENTED        -1
>> +#define GH_ERROR_RETRY            -2
>> +
>> +#define GH_ERROR_ARG_INVAL        1
>> +#define GH_ERROR_ARG_SIZE        2
>> +#define GH_ERROR_ARG_ALIGN        3
>> +
>> +#define GH_ERROR_NOMEM            10
>> +
>> +#define GH_ERROR_ADDR_OVFL        20
>> +#define GH_ERROR_ADDR_UNFL        21
>> +#define GH_ERROR_ADDR_INVAL        22
>> +
>> +#define GH_ERROR_DENIED            30
>> +#define GH_ERROR_BUSY            31
>> +#define GH_ERROR_IDLE            32
>> +
>> +#define GH_ERROR_IRQ_BOUND        40
>> +#define GH_ERROR_IRQ_UNBOUND        41
>> +
>> +#define GH_ERROR_CSPACE_CAP_NULL    50
>> +#define GH_ERROR_CSPACE_CAP_REVOKED    51
>> +#define GH_ERROR_CSPACE_WRONG_OBJ_TYPE    52
>> +#define GH_ERROR_CSPACE_INSUF_RIGHTS    53
>> +#define GH_ERROR_CSPACE_FULL        54
>> +
>> +#define GH_ERROR_MSGQUEUE_EMPTY        60
>> +#define GH_ERROR_MSGQUEUE_FULL        61
>> +
>> +static inline int gh_remap_error(int gh_error)
>> +{
>> +    switch (gh_error) {
>> +    case GH_ERROR_OK:
>> +        return 0;
>> +    case GH_ERROR_NOMEM:
>> +        return -ENOMEM;
>> +    case GH_ERROR_DENIED:
>> +    case GH_ERROR_CSPACE_CAP_NULL:
>> +    case GH_ERROR_CSPACE_CAP_REVOKED:
>> +    case GH_ERROR_CSPACE_WRONG_OBJ_TYPE:
>> +    case GH_ERROR_CSPACE_INSUF_RIGHTS:
>> +    case GH_ERROR_CSPACE_FULL:
>> +        return -EACCES;
>> +    case GH_ERROR_BUSY:
>> +    case GH_ERROR_IDLE:
>> +        return -EBUSY;
>> +    case GH_ERROR_IRQ_BOUND:
>> +    case GH_ERROR_IRQ_UNBOUND:
>> +    case GH_ERROR_MSGQUEUE_FULL:
>> +    case GH_ERROR_MSGQUEUE_EMPTY:
>> +        return -EPERM;
>> +    default:
>> +        return -EINVAL;
>> +    }
>> +}
>> +
>> +#endif
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 04/11] gunyah: Common types and error codes for Gunyah hypercalls

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Add architecture-independent standard error codes, types, and macros for
> Gunyah hypercalls.
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   MAINTAINERS            |  1 +
>   include/linux/gunyah.h | 75 ++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 76 insertions(+)
>   create mode 100644 include/linux/gunyah.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 02f97ac90cdf..2e4f1d9ed47b 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8744,6 +8744,7 @@ S:	Maintained
>   F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>   F:	Documentation/virt/gunyah/
>   F:	arch/arm64/include/asm/gunyah.h
> +F:	include/linux/gunyah.h
>   
>   HABANALABS PCI DRIVER
>   M:	Oded Gabbay <ogabbay@kernel.org>
> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
> new file mode 100644
> index 000000000000..69931a0f5736
> --- /dev/null
> +++ b/include/linux/gunyah.h
> @@ -0,0 +1,75 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#ifndef _GUNYAH_H
> +#define _GUNYAH_H
> +
> +#include <linux/types.h>
> +#include <linux/errno.h>
> +#include <asm/gunyah.h>
> +
> +typedef u64 gh_capid_t;

I think there was a rule on typedefs? Maybe I'm mistaken, couldn't find 
one. Why do you need it in the first place? Just use u64. Or 'enum 
gh_capid'.

> +
> +/* Common Gunyah macros */
> +#define GH_CAPID_INVAL	U64_MAX
> +



-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 04/11] gunyah: Common types and error codes for Gunyah hypercalls

[Elliot Berman]

On 8/2/2022 12:51 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Add architecture-independent standard error codes, types, and macros for
>> Gunyah hypercalls.
>>
>> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
>> ---
>>   MAINTAINERS            |  1 +
>>   include/linux/gunyah.h | 75 ++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 76 insertions(+)
>>   create mode 100644 include/linux/gunyah.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 02f97ac90cdf..2e4f1d9ed47b 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -8744,6 +8744,7 @@ S:    Maintained
>>   F:    Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>>   F:    Documentation/virt/gunyah/
>>   F:    arch/arm64/include/asm/gunyah.h
>> +F:    include/linux/gunyah.h
>>   HABANALABS PCI DRIVER
>>   M:    Oded Gabbay <ogabbay@kernel.org>
>> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
>> new file mode 100644
>> index 000000000000..69931a0f5736
>> --- /dev/null
>> +++ b/include/linux/gunyah.h
>> @@ -0,0 +1,75 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights 
>> reserved.
>> + */
>> +
>> +#ifndef _GUNYAH_H
>> +#define _GUNYAH_H
>> +
>> +#include <linux/types.h>
>> +#include <linux/errno.h>
>> +#include <asm/gunyah.h>
>> +
>> +typedef u64 gh_capid_t;
> 
> I think there was a rule on typedefs? Maybe I'm mistaken, couldn't find 
> one. Why do you need it in the first place? Just use u64. Or 'enum 
> gh_capid'.
> 

The rules are in Documentation/process/coding-style.rst. gh_capid_it is 
totally opaque to Linux, but I will switch to use u64 throughout.

>> +
>> +/* Common Gunyah macros */
>> +#define GH_CAPID_INVAL    U64_MAX
>> +
> 
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 05/11] virt: gunyah: Add sysfs nodes

[Elliot Berman]

Add /sys/hypervisor support when detecting that Linux is running in a
Gunyah environment. Export the version of Gunyah which is reported via
the hyp_identify hypercall.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 .../ABI/testing/sysfs-hypervisor-gunyah       |  37 +++++
 MAINTAINERS                                   |   2 +
 arch/arm64/include/asm/gunyah.h               |   4 +
 drivers/virt/Kconfig                          |   1 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/gunyah/Kconfig                   |  13 ++
 drivers/virt/gunyah/Makefile                  |   4 +
 drivers/virt/gunyah/sysfs.c                   | 139 ++++++++++++++++++
 8 files changed, 201 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-hypervisor-gunyah
 create mode 100644 drivers/virt/gunyah/Kconfig
 create mode 100644 drivers/virt/gunyah/Makefile
 create mode 100644 drivers/virt/gunyah/sysfs.c

diff --git a/Documentation/ABI/testing/sysfs-hypervisor-gunyah b/Documentation/ABI/testing/sysfs-hypervisor-gunyah
new file mode 100644
index 000000000000..41bce1965c91
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-hypervisor-gunyah
@@ -0,0 +1,37 @@
+What:		/sys/hypervisor/type
+Date:		August 2022
+KernelVersion:	5.20
+Contact:	linux-arm-msm@vger.kernel.org
+Description:	If running under Gunyah:
+		Type of hypervisor:
+		"gunyah": Gunyah hypervisor
+
+What:		/sys/hypervisor/features
+Date:		August 2022
+KernelVersion:	5.20
+Contact:	linux-arm-msm@vger.kernel.org
+Description:	If running under Gunyah:
+		Space separated list of features supported by Linux and Gunyah:
+		"cspace": Gunyah devices
+		"doorbell": Sending/receiving virtual interrupts via Gunyah doorbells
+		"message-queue": Sending/receiving messages via Gunyah message queues
+		"vic": Interrupt lending
+		"vpm": Virtual platform management
+		"vcpu": Virtual CPU management
+		"memextent": Memory lending/management
+		"trace": Gunyah hypervisor tracing
+
+
+What:		/sys/hypervisor/version/api
+Date:		August 2022
+KernelVersion:	5.20
+Contact:	linux-arm-msm@vger.kernel.org
+Description:	If running under Gunyah:
+		The Gunyah API version.
+
+What:		/sys/hypervisor/version/variant
+Date:		August 2022
+KernelVersion:	5.20
+Contact:	linux-arm-msm@vger.kernel.org
+Description:	If running under Gunyah:
+		The Gunyah variant (build) version.
\ No newline at end of file
diff --git a/MAINTAINERS b/MAINTAINERS
index 2e4f1d9ed47b..e63c51ee1a2a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8741,9 +8741,11 @@ M:	Elliot Berman <quic_eberman@quicinc.com>
 M:	Murali Nalajala <quic_mnalajal@quicinc.com>
 L:	linux-arm-msm@vger.kernel.org
 S:	Maintained
+F:	Documentation/ABI/testing/sysfs-hypervisor-gunyah
 F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
 F:	arch/arm64/include/asm/gunyah.h
+F:	drivers/virt/gunyah/
 F:	include/linux/gunyah.h
 
 HABANALABS PCI DRIVER
diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h
index 4820e9389f40..3aee35009910 100644
--- a/arch/arm64/include/asm/gunyah.h
+++ b/arch/arm64/include/asm/gunyah.h
@@ -19,11 +19,15 @@
 						   (GH_CALL_TYPE_SERVICE << GH_CALL_TYPE_SHIFT) \
 							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
 
+#define GH_HYPERCALL_CALL_UID			GH_SERVICE(0x3f01)
+
 #define GH_HYPERCALL(fn)	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \
 						   ARM_SMCCC_OWNER_VENDOR_HYP, \
 						   (GH_CALL_TYPE_HYPERCALL << GH_CALL_TYPE_SHIFT) \
 							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
 
+#define GH_HYPERCALL_HYP_IDENTIFY		GH_HYPERCALL(0x0000)
+
 #define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
 
 #define __gh_count_args(...)						\
diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig
index 87ef258cec64..259dc2be6cad 100644
--- a/drivers/virt/Kconfig
+++ b/drivers/virt/Kconfig
@@ -52,4 +52,5 @@ source "drivers/virt/coco/efi_secret/Kconfig"
 
 source "drivers/virt/coco/sev-guest/Kconfig"
 
+source "drivers/virt/gunyah/Kconfig"
 endif
diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile
index 093674e05c40..ec4fdfa8eef1 100644
--- a/drivers/virt/Makefile
+++ b/drivers/virt/Makefile
@@ -6,6 +6,7 @@
 obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
 obj-$(CONFIG_VMGENID)		+= vmgenid.o
 obj-y				+= vboxguest/
+obj-$(CONFIG_GUNYAH)		+= gunyah/
 
 obj-$(CONFIG_NITRO_ENCLAVES)	+= nitro_enclaves/
 obj-$(CONFIG_ACRN_HSM)		+= acrn/
diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig
new file mode 100644
index 000000000000..1e493017885e
--- /dev/null
+++ b/drivers/virt/gunyah/Kconfig
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config GUNYAH
+	tristate "Gunyah Virtualization drivers"
+	depends on ARM64
+	select SYS_HYPERVISOR
+	help
+	  The Gunyah drivers are the helper interfaces that runs in a guest VM
+	  such as basic inter-VM IPC and signaling mechanisms and higher level
+	  services such as memory/device sharing, IRQ sharing, and so on.
+
+	  Say Y here to enable the drivers needed to interact in a Gunyah
+	  virtual environment.
diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
new file mode 100644
index 000000000000..0aa086f9149f
--- /dev/null
+++ b/drivers/virt/gunyah/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+gunyah-y += sysfs.o
+obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
new file mode 100644
index 000000000000..253433a939cf
--- /dev/null
+++ b/drivers/virt/gunyah/sysfs.c
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "gunyah: " fmt
+
+#include <linux/kobject.h>
+#include <linux/gunyah.h>
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/init.h>
+#include <linux/of.h>
+
+#define QC_HYP_UID0 0x19bd54bd
+#define QC_HYP_UID1 0x0b37571b
+#define QC_HYP_UID2 0x946f609b
+#define QC_HYP_UID3 0x54539de6
+
+#define GUNYAH_UID0 0x673d5f14
+#define GUNYAH_UID1 0x9265ce36
+#define GUNYAH_UID2 0xa4535fdb
+#define GUNYAH_UID3 0xc1d58fcd
+
+#define gh_uid_matches(prefix, uid)	\
+	((uid)[0] == prefix ## _UID0 && (uid)[1] == prefix ## _UID1 && \
+	 (uid)[2] == prefix ## _UID2 && (uid)[3] == prefix ## _UID3)
+
+#define GH_API_INFO_API_VERSION(x)	(((x) >> 0) & 0x3fff)
+#define GH_API_INFO_BIG_ENDIAN(x)	(((x) >> 14) & 1)
+#define GH_API_INFO_IS_64BIT(x)		(((x) >> 15) & 1)
+#define GH_API_INFO_VARIANT(x)		(((x) >> 56) & 0xff)
+
+#define GH_IDENTIFY_PARTITION_CSPACE(flags)	(((flags)[0] >> 0) & 1)
+#define GH_IDENTIFY_DOORBELL(flags)		(((flags)[0] >> 1) & 1)
+#define GH_IDENTIFY_MSGQUEUE(flags)		(((flags)[0] >> 2) & 1)
+#define GH_IDENTIFY_VIC(flags)			(((flags)[0] >> 3) & 1)
+#define GH_IDENTIFY_VPM(flags)			(((flags)[0] >> 4) & 1)
+#define GH_IDENTIFY_VCPU(flags)			(((flags)[0] >> 5) & 1)
+#define GH_IDENTIFY_MEMEXTENT(flags)		(((flags)[0] >> 6) & 1)
+#define GH_IDENTIFY_TRACE_CTRL(flags)		(((flags)[0] >> 7) & 1)
+
+struct gh_hypercall_hyp_identify_resp {
+	u64 api_info;
+	u64 flags[3];
+};
+
+static struct gh_hypercall_hyp_identify_resp gunyah_api;
+
+static ssize_t type_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
+{
+	return sysfs_emit(buffer, "gunyah\n");
+}
+static struct kobj_attribute type_attr = __ATTR_RO(type);
+
+static ssize_t api_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
+{
+	return sysfs_emit(buffer, "%d\n", (int)GH_API_INFO_API_VERSION(gunyah_api.api_info));
+}
+static struct kobj_attribute api_attr = __ATTR_RO(api);
+
+static ssize_t variant_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
+{
+	return sysfs_emit(buffer, "%d\n", (int)GH_API_INFO_VARIANT(gunyah_api.api_info));
+}
+static struct kobj_attribute variant_attr = __ATTR_RO(variant);
+
+static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
+{
+	return sysfs_emit(buffer, "\n");
+}
+static struct kobj_attribute features_attr = __ATTR_RO(features);
+
+static struct attribute *version_attrs[] = {
+	&api_attr.attr,
+	&variant_attr.attr,
+	NULL
+};
+
+static const struct attribute_group version_group = {
+	.name = "version",
+	.attrs = version_attrs,
+};
+
+static int __init gh_sysfs_register(void)
+{
+	int ret;
+
+	ret = sysfs_create_file(hypervisor_kobj, &type_attr.attr);
+	if (ret)
+		return ret;
+
+	ret = sysfs_create_group(hypervisor_kobj, &version_group);
+	if (ret)
+		return ret;
+
+	return sysfs_create_file(hypervisor_kobj, &features_attr.attr);
+}
+
+static void gh_sysfs_unregister(void)
+{
+	sysfs_remove_file(hypervisor_kobj, &type_attr.attr);
+	sysfs_remove_group(hypervisor_kobj, &version_group);
+}
+
+static int __init gunyah_init(void)
+{
+	unsigned long uid[4];
+
+	arch_gh_hypercall(GH_HYPERCALL_CALL_UID, 0, uid[0], uid[1], uid[2], uid[3]);
+
+	if (!(gh_uid_matches(GUNYAH, uid) || gh_uid_matches(QC_HYP, uid)))
+		return 0;
+
+	arch_gh_hypercall(GH_HYPERCALL_HYP_IDENTIFY, 0, gunyah_api.api_info,
+		gunyah_api.flags[0], gunyah_api.flags[1], gunyah_api.flags[2]);
+
+	if (GH_API_INFO_API_VERSION(gunyah_api.api_info) != 1) {
+		pr_warn("Unrecognized gunyah version: %llu. Currently supported: 1\n",
+			GH_API_INFO_API_VERSION(gunyah_api.api_info));
+		return 0;
+	}
+
+	pr_notice("Running under Gunyah hypervisor v%lld/%llx\n",
+		  GH_API_INFO_API_VERSION(gunyah_api.api_info),
+		  GH_API_INFO_VARIANT(gunyah_api.api_info));
+
+	return gh_sysfs_register();
+}
+module_init(gunyah_init);
+
+static void __exit gunyah_exit(void)
+{
+	gh_sysfs_unregister();
+}
+module_exit(gunyah_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Gunyah Hypervisor Driver");
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 05/11] virt: gunyah: Add sysfs nodes

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Add /sys/hypervisor support when detecting that Linux is running in a
> Gunyah environment. Export the version of Gunyah which is reported via
> the hyp_identify hypercall.
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   .../ABI/testing/sysfs-hypervisor-gunyah       |  37 +++++
>   MAINTAINERS                                   |   2 +
>   arch/arm64/include/asm/gunyah.h               |   4 +
>   drivers/virt/Kconfig                          |   1 +
>   drivers/virt/Makefile                         |   1 +
>   drivers/virt/gunyah/Kconfig                   |  13 ++
>   drivers/virt/gunyah/Makefile                  |   4 +
>   drivers/virt/gunyah/sysfs.c                   | 139 ++++++++++++++++++
>   8 files changed, 201 insertions(+)
>   create mode 100644 Documentation/ABI/testing/sysfs-hypervisor-gunyah
>   create mode 100644 drivers/virt/gunyah/Kconfig
>   create mode 100644 drivers/virt/gunyah/Makefile
>   create mode 100644 drivers/virt/gunyah/sysfs.c
> 
> diff --git a/Documentation/ABI/testing/sysfs-hypervisor-gunyah b/Documentation/ABI/testing/sysfs-hypervisor-gunyah
> new file mode 100644
> index 000000000000..41bce1965c91
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-hypervisor-gunyah
> @@ -0,0 +1,37 @@
> +What:		/sys/hypervisor/type

These whole set of entries doesn't look right. You are defining 
gunyah-specific entries in the generic /sys/hypervisor directory. Other 
hypervisors will have their own set of ABI/sysfs files/etc/.

I would suggest using /sys/hypervisor/gunyah as a base dir. Then you can 
define any gunyah-specific file in it without caring if it applies to 
other hypervisors or not.

> +Date:		August 2022
> +KernelVersion:	5.20
> +Contact:	linux-arm-msm@vger.kernel.org
> +Description:	If running under Gunyah:
> +		Type of hypervisor:
> +		"gunyah": Gunyah hypervisor

And this will be gone if you adopt /sys/hypervisor/gunyah approach

> +
> +What:		/sys/hypervisor/features
> +Date:		August 2022
> +KernelVersion:	5.20
> +Contact:	linux-arm-msm@vger.kernel.org
> +Description:	If running under Gunyah:
> +		Space separated list of features supported by Linux and Gunyah:
> +		"cspace": Gunyah devices
> +		"doorbell": Sending/receiving virtual interrupts via Gunyah doorbells
> +		"message-queue": Sending/receiving messages via Gunyah message queues
> +		"vic": Interrupt lending
> +		"vpm": Virtual platform management
> +		"vcpu": Virtual CPU management
> +		"memextent": Memory lending/management
> +		"trace": Gunyah hypervisor tracing
> +
> +
> +What:		/sys/hypervisor/version/api
> +Date:		August 2022
> +KernelVersion:	5.20
> +Contact:	linux-arm-msm@vger.kernel.org
> +Description:	If running under Gunyah:
> +		The Gunyah API version.
> +
> +What:		/sys/hypervisor/version/variant
> +Date:		August 2022
> +KernelVersion:	5.20
> +Contact:	linux-arm-msm@vger.kernel.org
> +Description:	If running under Gunyah:
> +		The Gunyah variant (build) version.

If this is a build version, please call it a build version rather than 
variant.

> \ No newline at end of file
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 2e4f1d9ed47b..e63c51ee1a2a 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8741,9 +8741,11 @@ M:	Elliot Berman <quic_eberman@quicinc.com>
>   M:	Murali Nalajala <quic_mnalajal@quicinc.com>
>   L:	linux-arm-msm@vger.kernel.org
>   S:	Maintained
> +F:	Documentation/ABI/testing/sysfs-hypervisor-gunyah
>   F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
>   F:	Documentation/virt/gunyah/
>   F:	arch/arm64/include/asm/gunyah.h
> +F:	drivers/virt/gunyah/
>   F:	include/linux/gunyah.h
>   
>   HABANALABS PCI DRIVER
> diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h
> index 4820e9389f40..3aee35009910 100644
> --- a/arch/arm64/include/asm/gunyah.h
> +++ b/arch/arm64/include/asm/gunyah.h
> @@ -19,11 +19,15 @@
>   						   (GH_CALL_TYPE_SERVICE << GH_CALL_TYPE_SHIFT) \
>   							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
>   
> +#define GH_HYPERCALL_CALL_UID			GH_SERVICE(0x3f01)
> +
>   #define GH_HYPERCALL(fn)	ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \
>   						   ARM_SMCCC_OWNER_VENDOR_HYP, \
>   						   (GH_CALL_TYPE_HYPERCALL << GH_CALL_TYPE_SHIFT) \
>   							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
>   
> +#define GH_HYPERCALL_HYP_IDENTIFY		GH_HYPERCALL(0x0000)
> +
>   #define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
>   
>   #define __gh_count_args(...)						\
> diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig
> index 87ef258cec64..259dc2be6cad 100644
> --- a/drivers/virt/Kconfig
> +++ b/drivers/virt/Kconfig
> @@ -52,4 +52,5 @@ source "drivers/virt/coco/efi_secret/Kconfig"
>   
>   source "drivers/virt/coco/sev-guest/Kconfig"
>   
> +source "drivers/virt/gunyah/Kconfig"
>   endif
> diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile
> index 093674e05c40..ec4fdfa8eef1 100644
> --- a/drivers/virt/Makefile
> +++ b/drivers/virt/Makefile
> @@ -6,6 +6,7 @@
>   obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
>   obj-$(CONFIG_VMGENID)		+= vmgenid.o
>   obj-y				+= vboxguest/
> +obj-$(CONFIG_GUNYAH)		+= gunyah/
>   
>   obj-$(CONFIG_NITRO_ENCLAVES)	+= nitro_enclaves/
>   obj-$(CONFIG_ACRN_HSM)		+= acrn/
> diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig
> new file mode 100644
> index 000000000000..1e493017885e
> --- /dev/null
> +++ b/drivers/virt/gunyah/Kconfig
> @@ -0,0 +1,13 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +config GUNYAH
> +	tristate "Gunyah Virtualization drivers"
> +	depends on ARM64
> +	select SYS_HYPERVISOR
> +	help
> +	  The Gunyah drivers are the helper interfaces that runs in a guest VM
> +	  such as basic inter-VM IPC and signaling mechanisms and higher level
> +	  services such as memory/device sharing, IRQ sharing, and so on.
> +
> +	  Say Y here to enable the drivers needed to interact in a Gunyah
> +	  virtual environment.
> diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
> new file mode 100644
> index 000000000000..0aa086f9149f
> --- /dev/null
> +++ b/drivers/virt/gunyah/Makefile
> @@ -0,0 +1,4 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +gunyah-y += sysfs.o
> +obj-$(CONFIG_GUNYAH) += gunyah.o
> \ No newline at end of file
> diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
> new file mode 100644
> index 000000000000..253433a939cf
> --- /dev/null
> +++ b/drivers/virt/gunyah/sysfs.c
> @@ -0,0 +1,139 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#define pr_fmt(fmt) "gunyah: " fmt
> +
> +#include <linux/kobject.h>
> +#include <linux/gunyah.h>
> +#include <linux/module.h>
> +#include <linux/printk.h>
> +#include <linux/init.h>
> +#include <linux/of.h>
> +
> +#define QC_HYP_UID0 0x19bd54bd
> +#define QC_HYP_UID1 0x0b37571b
> +#define QC_HYP_UID2 0x946f609b
> +#define QC_HYP_UID3 0x54539de6
> +
> +#define GUNYAH_UID0 0x673d5f14
> +#define GUNYAH_UID1 0x9265ce36
> +#define GUNYAH_UID2 0xa4535fdb
> +#define GUNYAH_UID3 0xc1d58fcd
> +
> +#define gh_uid_matches(prefix, uid)	\
> +	((uid)[0] == prefix ## _UID0 && (uid)[1] == prefix ## _UID1 && \
> +	 (uid)[2] == prefix ## _UID2 && (uid)[3] == prefix ## _UID3)
> +
> +#define GH_API_INFO_API_VERSION(x)	(((x) >> 0) & 0x3fff)
> +#define GH_API_INFO_BIG_ENDIAN(x)	(((x) >> 14) & 1)
> +#define GH_API_INFO_IS_64BIT(x)		(((x) >> 15) & 1)
> +#define GH_API_INFO_VARIANT(x)		(((x) >> 56) & 0xff)
> +
> +#define GH_IDENTIFY_PARTITION_CSPACE(flags)	(((flags)[0] >> 0) & 1)
> +#define GH_IDENTIFY_DOORBELL(flags)		(((flags)[0] >> 1) & 1)
> +#define GH_IDENTIFY_MSGQUEUE(flags)		(((flags)[0] >> 2) & 1)
> +#define GH_IDENTIFY_VIC(flags)			(((flags)[0] >> 3) & 1)
> +#define GH_IDENTIFY_VPM(flags)			(((flags)[0] >> 4) & 1)
> +#define GH_IDENTIFY_VCPU(flags)			(((flags)[0] >> 5) & 1)
> +#define GH_IDENTIFY_MEMEXTENT(flags)		(((flags)[0] >> 6) & 1)
> +#define GH_IDENTIFY_TRACE_CTRL(flags)		(((flags)[0] >> 7) & 1)
> +
> +struct gh_hypercall_hyp_identify_resp {
> +	u64 api_info;
> +	u64 flags[3];
> +};
> +
> +static struct gh_hypercall_hyp_identify_resp gunyah_api;
> +
> +static ssize_t type_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
> +{
> +	return sysfs_emit(buffer, "gunyah\n");
> +}
> +static struct kobj_attribute type_attr = __ATTR_RO(type);
> +
> +static ssize_t api_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
> +{
> +	return sysfs_emit(buffer, "%d\n", (int)GH_API_INFO_API_VERSION(gunyah_api.api_info));
> +}
> +static struct kobj_attribute api_attr = __ATTR_RO(api);
> +
> +static ssize_t variant_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
> +{
> +	return sysfs_emit(buffer, "%d\n", (int)GH_API_INFO_VARIANT(gunyah_api.api_info));
> +}
> +static struct kobj_attribute variant_attr = __ATTR_RO(variant);
> +
> +static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
> +{
> +	return sysfs_emit(buffer, "\n");

empty features? oook. At least it deserves a comment here, if you plan 
to add features later in these series. But strictly speaking it might be 
better to define the features ABI once you start populating it.

> +}
> +static struct kobj_attribute features_attr = __ATTR_RO(features);
> +
> +static struct attribute *version_attrs[] = {
> +	&api_attr.attr,
> +	&variant_attr.attr,
> +	NULL
> +};
> +
> +static const struct attribute_group version_group = {
> +	.name = "version",
> +	.attrs = version_attrs,
> +};
> +
> +static int __init gh_sysfs_register(void)
> +{
> +	int ret;
> +
> +	ret = sysfs_create_file(hypervisor_kobj, &type_attr.attr);
> +	if (ret)
> +		return ret;
> +
> +	ret = sysfs_create_group(hypervisor_kobj, &version_group);
> +	if (ret)
> +		return ret;
> +
> +	return sysfs_create_file(hypervisor_kobj, &features_attr.attr);

error path?


> +}
> +
> +static void gh_sysfs_unregister(void)
> +{
> +	sysfs_remove_file(hypervisor_kobj, &type_attr.attr);
> +	sysfs_remove_group(hypervisor_kobj, &version_group);

Please unwind in a backwards manner. Also one sysfs_remove_ is missing.

> +}
> +
> +static int __init gunyah_init(void)
> +{
> +	unsigned long uid[4];
> +
> +	arch_gh_hypercall(GH_HYPERCALL_CALL_UID, 0, uid[0], uid[1], uid[2], uid[3]);
> +
> +	if (!(gh_uid_matches(GUNYAH, uid) || gh_uid_matches(QC_HYP, uid)))
> +		return 0;

if (!gh_uid_matched(GUNYAH, uid) &&
     !gh_uid_matches(QC_HYP, uid))
     return -ENODEV;


BTW: what is the difference between GUNYAH and QC_HYP here?

> +
> +	arch_gh_hypercall(GH_HYPERCALL_HYP_IDENTIFY, 0, gunyah_api.api_info,
> +		gunyah_api.flags[0], gunyah_api.flags[1], gunyah_api.flags[2]);
> +
> +	if (GH_API_INFO_API_VERSION(gunyah_api.api_info) != 1) {
> +		pr_warn("Unrecognized gunyah version: %llu. Currently supported: 1\n",
> +			GH_API_INFO_API_VERSION(gunyah_api.api_info));
> +		return 0;
> +	}
> +
> +	pr_notice("Running under Gunyah hypervisor v%lld/%llx\n",
> +		  GH_API_INFO_API_VERSION(gunyah_api.api_info),
> +		  GH_API_INFO_VARIANT(gunyah_api.api_info));
> +
> +	return gh_sysfs_register();
> +}
> +module_init(gunyah_init);
> +
> +static void __exit gunyah_exit(void)
> +{
> +	gh_sysfs_unregister();
> +}
> +module_exit(gunyah_exit);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_DESCRIPTION("Gunyah Hypervisor Driver");

Sysfs driver?


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 06/11] virt: gunyah: Add capabilities bus and devices

[Elliot Berman]

Some resources provided by the Gunyah hypervisor are described as
objects. The objects are identified with a capability ID. For instance,
Inter-VM communication is performed with doorbells and message queues.
Each doorbell and message queue endpoint can be described consisely as a
Linux device.

These resources are discovered either on the devicetree or reported by
the Resource Manager. Devices on the Gunyah bus are matched with drivers
according to the type ID reported by resource manager. Most resources
will be discovered directly from the resource manager, so matching
directly on type ID seems like sensible design.

Each resource may also optionally have an interrupt associated with it
and a known partner VM (e.g. which VM is the receiver of a message
queue).

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 drivers/virt/gunyah/Makefile         |   2 +-
 drivers/virt/gunyah/device.c         | 108 +++++++++++++++++++++++++++
 drivers/virt/gunyah/gunyah_private.h |  12 +++
 drivers/virt/gunyah/sysfs.c          |  25 ++++++-
 include/linux/gunyah.h               |  45 +++++++++++
 5 files changed, 189 insertions(+), 3 deletions(-)
 create mode 100644 drivers/virt/gunyah/device.c
 create mode 100644 drivers/virt/gunyah/gunyah_private.h

diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index 0aa086f9149f..3869fb7371df 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
-gunyah-y += sysfs.o
+gunyah-y += sysfs.o device.o
 obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
diff --git a/drivers/virt/gunyah/device.c b/drivers/virt/gunyah/device.c
new file mode 100644
index 000000000000..93595f9a65b9
--- /dev/null
+++ b/drivers/virt/gunyah/device.c
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "ghdev: " fmt
+
+#include <linux/interrupt.h>
+#include <linux/gunyah.h>
+#include <linux/device.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+
+#include "gunyah_private.h"
+
+static int gunyah_match(struct device *dev, struct device_driver *drv)
+{
+	struct gunyah_device *ghdev = to_gunyah_device(dev);
+	struct gunyah_driver *ghdrv = to_gunyah_driver(drv);
+
+	return ghdev->type == ghdrv->type;
+}
+
+static int gunyah_probe(struct device *dev)
+{
+	struct gunyah_device *ghdev = to_gunyah_device(dev);
+	struct gunyah_driver *ghdrv = to_gunyah_driver(dev->driver);
+
+	return ghdrv->probe ? ghdrv->probe(ghdev) : 0;
+}
+
+static void gunyah_remove(struct device *dev)
+{
+	struct gunyah_device *ghdev = to_gunyah_device(dev);
+	struct gunyah_driver *ghdrv = to_gunyah_driver(dev->driver);
+
+	if (ghdrv->remove)
+		ghdrv->remove(ghdev);
+}
+
+static struct bus_type gunyah_bus = {
+	.name	= "gunyah",
+	.match	= gunyah_match,
+	.probe	= gunyah_probe,
+	.remove	= gunyah_remove,
+};
+
+int gunyah_register_driver(struct gunyah_driver *ghdrv)
+{
+	ghdrv->driver.bus = &gunyah_bus;
+	return driver_register(&ghdrv->driver);
+}
+
+void gunyah_unregister_driver(struct gunyah_driver *ghdrv)
+{
+	driver_unregister(&ghdrv->driver);
+}
+
+static void gunyah_device_release(struct device *dev)
+{
+	struct gunyah_device *ghdev = to_gunyah_device(dev);
+
+	kfree(ghdev);
+}
+
+struct gunyah_device *gunyah_device_alloc(struct device *parent, gh_capid_t capid, u8 type)
+{
+	struct gunyah_device *ghdev;
+
+	ghdev = kzalloc(sizeof(*ghdev), GFP_KERNEL);
+	if (!ghdev)
+		return NULL;
+
+	ghdev->capid = capid;
+	ghdev->type = type;
+	ghdev->irq = IRQ_NOTCONNECTED;
+	ghdev->dev.parent = parent;
+	ghdev->dev.release = gunyah_device_release;
+	ghdev->dev.bus = &gunyah_bus;
+	device_initialize(&ghdev->dev);
+	return ghdev;
+}
+
+int gunyah_device_add(struct gunyah_device *ghdev)
+{
+	int ret;
+
+	ret = dev_set_name(&ghdev->dev, "%u.%08llx", ghdev->type, ghdev->capid);
+	if (ret)
+		return ret;
+
+	return device_add(&ghdev->dev);
+}
+
+void gunyah_device_remove(struct gunyah_device *ghdev)
+{
+	device_unregister(&ghdev->dev);
+}
+
+int __init gunyah_bus_init(void)
+{
+	return bus_register(&gunyah_bus);
+}
+
+void gunyah_bus_exit(void)
+{
+	bus_unregister(&gunyah_bus);
+}
diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
new file mode 100644
index 000000000000..5f3832608020
--- /dev/null
+++ b/drivers/virt/gunyah/gunyah_private.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#ifndef _GUNYAH_PRIVATE_H
+#define _GUNYAH_PRIVATE_H
+
+int __init gunyah_bus_init(void);
+void gunyah_bus_exit(void);
+
+#endif
diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
index 253433a939cf..220560cb3b1c 100644
--- a/drivers/virt/gunyah/sysfs.c
+++ b/drivers/virt/gunyah/sysfs.c
@@ -12,6 +12,8 @@
 #include <linux/init.h>
 #include <linux/of.h>
 
+#include "gunyah_private.h"
+
 #define QC_HYP_UID0 0x19bd54bd
 #define QC_HYP_UID1 0x0b37571b
 #define QC_HYP_UID2 0x946f609b
@@ -67,7 +69,13 @@ static struct kobj_attribute variant_attr = __ATTR_RO(variant);
 
 static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
 {
-	return sysfs_emit(buffer, "\n");
+	int len = 0;
+
+	if (GH_IDENTIFY_PARTITION_CSPACE(gunyah_api.flags))
+		len += sysfs_emit_at(buffer, len, "cspace ");
+
+	len += sysfs_emit_at(buffer, len, "\n");
+	return len;
 }
 static struct kobj_attribute features_attr = __ATTR_RO(features);
 
@@ -105,6 +113,7 @@ static void gh_sysfs_unregister(void)
 
 static int __init gunyah_init(void)
 {
+	int ret;
 	unsigned long uid[4];
 
 	arch_gh_hypercall(GH_HYPERCALL_CALL_UID, 0, uid[0], uid[1], uid[2], uid[3]);
@@ -125,12 +134,24 @@ static int __init gunyah_init(void)
 		  GH_API_INFO_API_VERSION(gunyah_api.api_info),
 		  GH_API_INFO_VARIANT(gunyah_api.api_info));
 
-	return gh_sysfs_register();
+	ret = gh_sysfs_register();
+	if (ret)
+		return ret;
+
+	ret = gunyah_bus_init();
+	if (ret)
+		goto err_sysfs;
+
+	return ret;
+err_sysfs:
+	gh_sysfs_unregister();
+	return ret;
 }
 module_init(gunyah_init);
 
 static void __exit gunyah_exit(void)
 {
+	gunyah_bus_exit();
 	gh_sysfs_unregister();
 }
 module_exit(gunyah_exit);
diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
index 69931a0f5736..ce35f4491773 100644
--- a/include/linux/gunyah.h
+++ b/include/linux/gunyah.h
@@ -6,6 +6,7 @@
 #ifndef _GUNYAH_H
 #define _GUNYAH_H
 
+#include <linux/device.h>
 #include <linux/types.h>
 #include <linux/errno.h>
 #include <asm/gunyah.h>
@@ -72,4 +73,48 @@ static inline int gh_remap_error(int gh_error)
 	}
 }
 
+/* Follows resource manager's resource types for VM_GET_HYP_RESOURCES */
+#define GUNYAH_DEVICE_TYPE_BELL_TX	0
+#define GUNYAH_DEVICE_TYPE_BELL_RX	1
+#define GUNYAH_DEVICE_TYPE_MSGQ_TX	2
+#define GUNYAH_DEVICE_TYPE_MSGQ_RX	3
+#define GUNYAH_DEVICE_TYPE_VCPU		4
+
+struct gunyah_device {
+	u8 type;
+	gh_capid_t capid;
+	int irq;
+
+	struct device dev;
+};
+
+#define to_gunyah_device(dev) container_of(dev, struct gunyah_device, dev)
+
+static inline void *ghdev_get_drvdata(const struct gunyah_device *ghdev)
+{
+	return dev_get_drvdata(&ghdev->dev);
+}
+
+static inline void ghdev_set_drvdata(struct gunyah_device *ghdev, void *data)
+{
+	dev_set_drvdata(&ghdev->dev, data);
+}
+
+struct gunyah_device *gunyah_device_alloc(struct device *parent, gh_capid_t capid, u8 type);
+
+int gunyah_device_add(struct gunyah_device *ghdev);
+void gunyah_device_remove(struct gunyah_device *ghdev);
+
+struct gunyah_driver {
+	struct device_driver driver;
+	u8 type;
+	int (*probe)(struct gunyah_device *ghdev);
+	int (*remove)(struct gunyah_device *ghdev);
+};
+
+#define to_gunyah_driver(drv) container_of(drv, struct gunyah_driver, driver)
+
+int gunyah_register_driver(struct gunyah_driver *ghdrv);
+void gunyah_unregister_driver(struct gunyah_driver *ghdrv);
+
 #endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 06/11] virt: gunyah: Add capabilities bus and devices

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Some resources provided by the Gunyah hypervisor are described as
> objects. The objects are identified with a capability ID. For instance,
> Inter-VM communication is performed with doorbells and message queues.
> Each doorbell and message queue endpoint can be described consisely as a
> Linux device.
> 
> These resources are discovered either on the devicetree or reported by
> the Resource Manager. Devices on the Gunyah bus are matched with drivers
> according to the type ID reported by resource manager. Most resources
> will be discovered directly from the resource manager, so matching
> directly on type ID seems like sensible design.
> 
> Each resource may also optionally have an interrupt associated with it
> and a known partner VM (e.g. which VM is the receiver of a message
> queue).
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   drivers/virt/gunyah/Makefile         |   2 +-
>   drivers/virt/gunyah/device.c         | 108 +++++++++++++++++++++++++++
>   drivers/virt/gunyah/gunyah_private.h |  12 +++
>   drivers/virt/gunyah/sysfs.c          |  25 ++++++-
>   include/linux/gunyah.h               |  45 +++++++++++
>   5 files changed, 189 insertions(+), 3 deletions(-)
>   create mode 100644 drivers/virt/gunyah/device.c
>   create mode 100644 drivers/virt/gunyah/gunyah_private.h
> 
> diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
> index 0aa086f9149f..3869fb7371df 100644
> --- a/drivers/virt/gunyah/Makefile
> +++ b/drivers/virt/gunyah/Makefile
> @@ -1,4 +1,4 @@
>   # SPDX-License-Identifier: GPL-2.0-only
>   
> -gunyah-y += sysfs.o
> +gunyah-y += sysfs.o device.o
>   obj-$(CONFIG_GUNYAH) += gunyah.o
> \ No newline at end of file

Even git-format-patch warns you that should be a newline at the end of 
the file.

> diff --git a/drivers/virt/gunyah/device.c b/drivers/virt/gunyah/device.c
> new file mode 100644
> index 000000000000..93595f9a65b9
> --- /dev/null
> +++ b/drivers/virt/gunyah/device.c
> @@ -0,0 +1,108 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#define pr_fmt(fmt) "ghdev: " fmt
> +
> +#include <linux/interrupt.h>
> +#include <linux/gunyah.h>
> +#include <linux/device.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +
> +#include "gunyah_private.h"
> +
> +static int gunyah_match(struct device *dev, struct device_driver *drv)
> +{
> +	struct gunyah_device *ghdev = to_gunyah_device(dev);
> +	struct gunyah_driver *ghdrv = to_gunyah_driver(drv);
> +
> +	return ghdev->type == ghdrv->type;
> +}
> +
> +static int gunyah_probe(struct device *dev)
> +{
> +	struct gunyah_device *ghdev = to_gunyah_device(dev);
> +	struct gunyah_driver *ghdrv = to_gunyah_driver(dev->driver);
> +
> +	return ghdrv->probe ? ghdrv->probe(ghdev) : 0;
> +}
> +
> +static void gunyah_remove(struct device *dev)
> +{
> +	struct gunyah_device *ghdev = to_gunyah_device(dev);
> +	struct gunyah_driver *ghdrv = to_gunyah_driver(dev->driver);
> +
> +	if (ghdrv->remove)
> +		ghdrv->remove(ghdev);
> +}
> +
> +static struct bus_type gunyah_bus = {
> +	.name	= "gunyah",
> +	.match	= gunyah_match,
> +	.probe	= gunyah_probe,
> +	.remove	= gunyah_remove,
> +};
> +
> +int gunyah_register_driver(struct gunyah_driver *ghdrv)
> +{
> +	ghdrv->driver.bus = &gunyah_bus;
> +	return driver_register(&ghdrv->driver);
> +}
> +
> +void gunyah_unregister_driver(struct gunyah_driver *ghdrv)
> +{
> +	driver_unregister(&ghdrv->driver);
> +}
> +
> +static void gunyah_device_release(struct device *dev)
> +{
> +	struct gunyah_device *ghdev = to_gunyah_device(dev);
> +
> +	kfree(ghdev);
> +}
> +
> +struct gunyah_device *gunyah_device_alloc(struct device *parent, gh_capid_t capid, u8 type)
> +{
> +	struct gunyah_device *ghdev;
> +
> +	ghdev = kzalloc(sizeof(*ghdev), GFP_KERNEL);
> +	if (!ghdev)
> +		return NULL;
> +
> +	ghdev->capid = capid;
> +	ghdev->type = type;
> +	ghdev->irq = IRQ_NOTCONNECTED;
> +	ghdev->dev.parent = parent;
> +	ghdev->dev.release = gunyah_device_release;
> +	ghdev->dev.bus = &gunyah_bus;
> +	device_initialize(&ghdev->dev);
> +	return ghdev;
> +}
> +
> +int gunyah_device_add(struct gunyah_device *ghdev)
> +{
> +	int ret;
> +
> +	ret = dev_set_name(&ghdev->dev, "%u.%08llx", ghdev->type, ghdev->capid);
> +	if (ret)
> +		return ret;
> +
> +	return device_add(&ghdev->dev);
> +}

Does your device need any additional setup between _alloc() and _add(). 
  If not, it may be better to squash them together.

> +
> +void gunyah_device_remove(struct gunyah_device *ghdev)
> +{
> +	device_unregister(&ghdev->dev);
> +}
> +
> +int __init gunyah_bus_init(void)
> +{
> +	return bus_register(&gunyah_bus);
> +}
> +
> +void gunyah_bus_exit(void)
> +{
> +	bus_unregister(&gunyah_bus);
> +}
> diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
> new file mode 100644
> index 000000000000..5f3832608020
> --- /dev/null
> +++ b/drivers/virt/gunyah/gunyah_private.h
> @@ -0,0 +1,12 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#ifndef _GUNYAH_PRIVATE_H
> +#define _GUNYAH_PRIVATE_H
> +
> +int __init gunyah_bus_init(void);
> +void gunyah_bus_exit(void);
> +
> +#endif
> diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
> index 253433a939cf..220560cb3b1c 100644
> --- a/drivers/virt/gunyah/sysfs.c
> +++ b/drivers/virt/gunyah/sysfs.c
> @@ -12,6 +12,8 @@
>   #include <linux/init.h>
>   #include <linux/of.h>
>   
> +#include "gunyah_private.h"
> +
>   #define QC_HYP_UID0 0x19bd54bd
>   #define QC_HYP_UID1 0x0b37571b
>   #define QC_HYP_UID2 0x946f609b
> @@ -67,7 +69,13 @@ static struct kobj_attribute variant_attr = __ATTR_RO(variant);
>   
>   static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, char *buffer)
>   {
> -	return sysfs_emit(buffer, "\n");
> +	int len = 0;
> +
> +	if (GH_IDENTIFY_PARTITION_CSPACE(gunyah_api.flags))
> +		len += sysfs_emit_at(buffer, len, "cspace ");

How is this related to Linux gunyah bus implementation? Please move to 
the sysfs patch.

> +
> +	len += sysfs_emit_at(buffer, len, "\n");
> +	return len;
>   }
>   static struct kobj_attribute features_attr = __ATTR_RO(features);
>   
> @@ -105,6 +113,7 @@ static void gh_sysfs_unregister(void)
>   
>   static int __init gunyah_init(void)
>   {
> +	int ret;
>   	unsigned long uid[4];
>   
>   	arch_gh_hypercall(GH_HYPERCALL_CALL_UID, 0, uid[0], uid[1], uid[2], uid[3]);
> @@ -125,12 +134,24 @@ static int __init gunyah_init(void)
>   		  GH_API_INFO_API_VERSION(gunyah_api.api_info),
>   		  GH_API_INFO_VARIANT(gunyah_api.api_info));
>   
> -	return gh_sysfs_register();
> +	ret = gh_sysfs_register();
> +	if (ret)
> +		return ret;
> +
> +	ret = gunyah_bus_init();
> +	if (ret)
> +		goto err_sysfs;
> +
> +	return ret;
> +err_sysfs:
> +	gh_sysfs_unregister();
> +	return ret;
>   }
>   module_init(gunyah_init);
>   
>   static void __exit gunyah_exit(void)
>   {
> +	gunyah_bus_exit();
>   	gh_sysfs_unregister();
>   }
>   module_exit(gunyah_exit);
> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
> index 69931a0f5736..ce35f4491773 100644
> --- a/include/linux/gunyah.h
> +++ b/include/linux/gunyah.h
> @@ -6,6 +6,7 @@
>   #ifndef _GUNYAH_H
>   #define _GUNYAH_H
>   
> +#include <linux/device.h>
>   #include <linux/types.h>
>   #include <linux/errno.h>
>   #include <asm/gunyah.h>
> @@ -72,4 +73,48 @@ static inline int gh_remap_error(int gh_error)
>   	}
>   }
>   
> +/* Follows resource manager's resource types for VM_GET_HYP_RESOURCES */
> +#define GUNYAH_DEVICE_TYPE_BELL_TX	0
> +#define GUNYAH_DEVICE_TYPE_BELL_RX	1
> +#define GUNYAH_DEVICE_TYPE_MSGQ_TX	2
> +#define GUNYAH_DEVICE_TYPE_MSGQ_RX	3
> +#define GUNYAH_DEVICE_TYPE_VCPU		4
> +
> +struct gunyah_device {
> +	u8 type;
> +	gh_capid_t capid;
> +	int irq;
> +
> +	struct device dev;
> +};
> +
> +#define to_gunyah_device(dev) container_of(dev, struct gunyah_device, dev)
> +
> +static inline void *ghdev_get_drvdata(const struct gunyah_device *ghdev)
> +{
> +	return dev_get_drvdata(&ghdev->dev);
> +}
> +
> +static inline void ghdev_set_drvdata(struct gunyah_device *ghdev, void *data)
> +{
> +	dev_set_drvdata(&ghdev->dev, data);
> +}
> +
> +struct gunyah_device *gunyah_device_alloc(struct device *parent, gh_capid_t capid, u8 type);
> +
> +int gunyah_device_add(struct gunyah_device *ghdev);
> +void gunyah_device_remove(struct gunyah_device *ghdev);
> +
> +struct gunyah_driver {
> +	struct device_driver driver;
> +	u8 type;
> +	int (*probe)(struct gunyah_device *ghdev);
> +	int (*remove)(struct gunyah_device *ghdev);
> +};
> +
> +#define to_gunyah_driver(drv) container_of(drv, struct gunyah_driver, driver)
> +
> +int gunyah_register_driver(struct gunyah_driver *ghdrv);
> +void gunyah_unregister_driver(struct gunyah_driver *ghdrv);
> +
>   #endif


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Elliot Berman]

Gunyah message queues are unidirectional pipelines to communicate
between 2 virtual machines, but are typically paired to allow
bidirectional communication. The intended use case is for small control
messages between 2 VMs, as they support a maximum of 240 bytes.

Message queues can be discovered either by resource manager or on the
devicetree. To support discovery on the devicetree, client drivers can
use gh_msgq_platform_host_attach to allocate the tx and rx message
queues according to
Documentation/devicetree/bindings/gunyah/qcom,hypervisor.yml.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 arch/arm64/include/asm/gunyah.h      |   4 +
 drivers/virt/gunyah/Makefile         |   2 +-
 drivers/virt/gunyah/gunyah_private.h |   3 +
 drivers/virt/gunyah/msgq.c           | 223 +++++++++++++++++++++++++++
 drivers/virt/gunyah/sysfs.c          |   9 ++
 include/linux/gunyah.h               |  13 ++
 6 files changed, 253 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/gunyah/msgq.c

diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h
index 3aee35009910..ba7398bd851b 100644
--- a/arch/arm64/include/asm/gunyah.h
+++ b/arch/arm64/include/asm/gunyah.h
@@ -27,6 +27,10 @@
 							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
 
 #define GH_HYPERCALL_HYP_IDENTIFY		GH_HYPERCALL(0x0000)
+#define GH_HYPERCALL_MSGQ_SEND			GH_HYPERCALL(0x001B)
+#define GH_HYPERCALL_MSGQ_RECV			GH_HYPERCALL(0x001C)
+
+#define GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH	BIT(0)
 
 #define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
 
diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index 3869fb7371df..94dc8e738911 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
-gunyah-y += sysfs.o device.o
+gunyah-y += sysfs.o device.o msgq.o
 obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
index 5f3832608020..2ade32bd9bdf 100644
--- a/drivers/virt/gunyah/gunyah_private.h
+++ b/drivers/virt/gunyah/gunyah_private.h
@@ -9,4 +9,7 @@
 int __init gunyah_bus_init(void);
 void gunyah_bus_exit(void);
 
+int __init gh_msgq_init(void);
+void gh_msgq_exit(void);
+
 #endif
diff --git a/drivers/virt/gunyah/msgq.c b/drivers/virt/gunyah/msgq.c
new file mode 100644
index 000000000000..afc2572d3e7d
--- /dev/null
+++ b/drivers/virt/gunyah/msgq.c
@@ -0,0 +1,223 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#include <linux/interrupt.h>
+#include <linux/gunyah.h>
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/init.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+
+#include "gunyah_private.h"
+
+struct gh_msgq {
+	bool ready;
+	wait_queue_head_t wq;
+	spinlock_t lock;
+};
+
+static irqreturn_t gh_msgq_irq_handler(int irq, void *dev)
+{
+	struct gh_msgq *msgq = dev;
+
+	spin_lock(&msgq->lock);
+	msgq->ready = true;
+	spin_unlock(&msgq->lock);
+	wake_up_interruptible_all(&msgq->wq);
+
+	return IRQ_HANDLED;
+}
+
+static int __gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size, u64 tx_flags)
+{
+	unsigned long flags, gh_error;
+	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
+	ssize_t ret;
+	bool ready;
+
+	spin_lock_irqsave(&msgq->lock, flags);
+	arch_gh_hypercall(GH_HYPERCALL_MSGQ_SEND, 5,
+			  ghdev->capid, size, (uintptr_t)buff, tx_flags, 0,
+			  gh_error, ready);
+	switch (gh_error) {
+	case GH_ERROR_OK:
+		ret = 0;
+		msgq->ready = ready;
+		break;
+	case GH_ERROR_MSGQUEUE_FULL:
+		ret = -EAGAIN;
+		msgq->ready = false;
+		break;
+	default:
+		ret = gh_remap_error(gh_error);
+		break;
+	}
+	spin_unlock_irqrestore(&msgq->lock, flags);
+
+	return ret;
+}
+
+/**
+ * gh_msgq_send() - Send a message to the client running on a different VM
+ * @client: The client descriptor that was obtained via gh_msgq_register()
+ * @buff: Pointer to the buffer where the received data must be placed
+ * @buff_size: The size of the buffer space available
+ * @flags: Optional flags to pass to receive the data. For the list of flags,
+ *         see linux/gunyah/gh_msgq.h
+ *
+ * Returns: The number of bytes copied to buff. <0 if there was an error.
+ *
+ * Note: this function may sleep and should not be called from interrupt context
+ */
+ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size,
+		     const unsigned long flags)
+{
+	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
+	ssize_t ret;
+	u64 tx_flags = 0;
+
+	if (flags & GH_MSGQ_TX_PUSH)
+		tx_flags |= GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH;
+
+	do {
+		ret = __gh_msgq_send(ghdev, buff, size, tx_flags);
+
+		if (ret == -EAGAIN) {
+			if (flags & GH_MSGQ_NONBLOCK)
+				goto out;
+			if (wait_event_interruptible(msgq->wq, msgq->ready))
+				ret = -ERESTARTSYS;
+		}
+	} while (ret == -EAGAIN);
+
+out:
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_msgq_send);
+
+static ssize_t __gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size)
+{
+	unsigned long flags, gh_error;
+	size_t recv_size;
+	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
+	ssize_t ret;
+	bool ready;
+
+	spin_lock_irqsave(&msgq->lock, flags);
+
+	arch_gh_hypercall(GH_HYPERCALL_MSGQ_RECV, 4,
+			  ghdev->capid, (uintptr_t)buff, size, 0,
+			  gh_error, recv_size, ready);
+	switch (gh_error) {
+	case GH_ERROR_OK:
+		ret = recv_size;
+		msgq->ready = ready;
+		break;
+	case GH_ERROR_MSGQUEUE_EMPTY:
+		ret = -EAGAIN;
+		msgq->ready = false;
+		break;
+	default:
+		ret = gh_remap_error(gh_error);
+		break;
+	}
+	spin_unlock_irqrestore(&msgq->lock, flags);
+
+	return ret;
+}
+
+/**
+ * gh_msgq_recv() - Receive a message from the client running on a different VM
+ * @client: The client descriptor that was obtained via gh_msgq_register()
+ * @buff: Pointer to the buffer where the received data must be placed
+ * @buff_size: The size of the buffer space available
+ * @flags: Optional flags to pass to receive the data. For the list of flags,
+ *         see linux/gunyah/gh_msgq.h
+ *
+ * Returns: The number of bytes copied to buff. <0 if there was an error.
+ *
+ * Note: this function may sleep and should not be called from interrupt context
+ */
+ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size,
+		     const unsigned long flags)
+{
+	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
+	ssize_t ret;
+
+	do {
+		ret = __gh_msgq_recv(ghdev, buff, size);
+
+		if (ret == -EAGAIN) {
+			if (flags & GH_MSGQ_NONBLOCK)
+				goto out;
+			if (wait_event_interruptible(msgq->wq, msgq->ready))
+				ret = -ERESTARTSYS;
+		}
+	} while (ret == -EAGAIN);
+
+out:
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_msgq_recv);
+
+static int gh_msgq_probe(struct gunyah_device *ghdev)
+{
+	struct gh_msgq *msgq;
+
+	msgq = devm_kzalloc(&ghdev->dev, sizeof(*msgq), GFP_KERNEL);
+	if (!msgq)
+		return -ENOMEM;
+	ghdev_set_drvdata(ghdev, msgq);
+
+	msgq->ready = true; /* Assume we can use the message queue right away */
+	init_waitqueue_head(&msgq->wq);
+	spin_lock_init(&msgq->lock);
+
+	return devm_request_irq(&ghdev->dev, ghdev->irq, gh_msgq_irq_handler, 0,
+				dev_name(&ghdev->dev), msgq);
+}
+
+static struct gunyah_driver gh_msgq_tx_driver = {
+	.driver = {
+		.name = "gh_msgq_tx",
+		.owner = THIS_MODULE,
+	},
+	.type = GUNYAH_DEVICE_TYPE_MSGQ_TX,
+	.probe = gh_msgq_probe,
+};
+
+static struct gunyah_driver gh_msgq_rx_driver = {
+	.driver = {
+		.name = "gh_msgq_rx",
+		.owner = THIS_MODULE,
+	},
+	.type = GUNYAH_DEVICE_TYPE_MSGQ_RX,
+	.probe = gh_msgq_probe,
+};
+
+int __init gh_msgq_init(void)
+{
+	int ret;
+
+	ret = gunyah_register_driver(&gh_msgq_tx_driver);
+	if (ret)
+		return ret;
+
+	ret = gunyah_register_driver(&gh_msgq_rx_driver);
+	if (ret)
+		goto err_rx;
+
+	return ret;
+err_rx:
+	gunyah_unregister_driver(&gh_msgq_tx_driver);
+	return ret;
+}
+
+void gh_msgq_exit(void)
+{
+	gunyah_unregister_driver(&gh_msgq_rx_driver);
+	gunyah_unregister_driver(&gh_msgq_tx_driver);
+}
diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
index 220560cb3b1c..7589689e5e92 100644
--- a/drivers/virt/gunyah/sysfs.c
+++ b/drivers/virt/gunyah/sysfs.c
@@ -73,6 +73,8 @@ static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr,
 
 	if (GH_IDENTIFY_PARTITION_CSPACE(gunyah_api.flags))
 		len += sysfs_emit_at(buffer, len, "cspace ");
+	if (GH_IDENTIFY_MSGQUEUE(gunyah_api.flags))
+		len += sysfs_emit_at(buffer, len, "message-queue ");
 
 	len += sysfs_emit_at(buffer, len, "\n");
 	return len;
@@ -142,7 +144,13 @@ static int __init gunyah_init(void)
 	if (ret)
 		goto err_sysfs;
 
+	ret = gh_msgq_init();
+	if (ret)
+		goto err_bus;
+
 	return ret;
+err_bus:
+	gunyah_bus_exit();
 err_sysfs:
 	gh_sysfs_unregister();
 	return ret;
@@ -151,6 +159,7 @@ module_init(gunyah_init);
 
 static void __exit gunyah_exit(void)
 {
+	gh_msgq_exit();
 	gunyah_bus_exit();
 	gh_sysfs_unregister();
 }
diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
index ce35f4491773..099224f9d6d1 100644
--- a/include/linux/gunyah.h
+++ b/include/linux/gunyah.h
@@ -6,6 +6,7 @@
 #ifndef _GUNYAH_H
 #define _GUNYAH_H
 
+#include <linux/platform_device.h>
 #include <linux/device.h>
 #include <linux/types.h>
 #include <linux/errno.h>
@@ -117,4 +118,16 @@ struct gunyah_driver {
 int gunyah_register_driver(struct gunyah_driver *ghdrv);
 void gunyah_unregister_driver(struct gunyah_driver *ghdrv);
 
+#define GH_MSGQ_MAX_MSG_SIZE	1024
+
+/* Possible flags to pass for Tx or Rx */
+#define GH_MSGQ_TX_PUSH		BIT(0)
+#define GH_MSGQ_NONBLOCK	BIT(32)
+
+ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size,
+		     const unsigned long flags);
+ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size,
+		     const unsigned long flags);
+
+
 #endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Gunyah message queues are unidirectional pipelines to communicate
> between 2 virtual machines, but are typically paired to allow
> bidirectional communication. The intended use case is for small control
> messages between 2 VMs, as they support a maximum of 240 bytes.
> 
> Message queues can be discovered either by resource manager or on the
> devicetree. To support discovery on the devicetree, client drivers can

devicetree and discovery do not quite match to me. The device is delared 
in the DT, not discovered.

> use gh_msgq_platform_host_attach to allocate the tx and rx message
> queues according to
> Documentation/devicetree/bindings/gunyah/qcom,hypervisor.yml.

-ENOSUCHFILE

> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   arch/arm64/include/asm/gunyah.h      |   4 +
>   drivers/virt/gunyah/Makefile         |   2 +-
>   drivers/virt/gunyah/gunyah_private.h |   3 +
>   drivers/virt/gunyah/msgq.c           | 223 +++++++++++++++++++++++++++
>   drivers/virt/gunyah/sysfs.c          |   9 ++
>   include/linux/gunyah.h               |  13 ++
>   6 files changed, 253 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/virt/gunyah/msgq.c
> 
> diff --git a/arch/arm64/include/asm/gunyah.h b/arch/arm64/include/asm/gunyah.h
> index 3aee35009910..ba7398bd851b 100644
> --- a/arch/arm64/include/asm/gunyah.h
> +++ b/arch/arm64/include/asm/gunyah.h
> @@ -27,6 +27,10 @@
>   							| ((fn) & GH_CALL_FUNCTION_NUM_MASK))
>   
>   #define GH_HYPERCALL_HYP_IDENTIFY		GH_HYPERCALL(0x0000)
> +#define GH_HYPERCALL_MSGQ_SEND			GH_HYPERCALL(0x001B)
> +#define GH_HYPERCALL_MSGQ_RECV			GH_HYPERCALL(0x001C)
> +
> +#define GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH	BIT(0)
>   
>   #define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
>   
> diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
> index 3869fb7371df..94dc8e738911 100644
> --- a/drivers/virt/gunyah/Makefile
> +++ b/drivers/virt/gunyah/Makefile
> @@ -1,4 +1,4 @@
>   # SPDX-License-Identifier: GPL-2.0-only
>   
> -gunyah-y += sysfs.o device.o
> +gunyah-y += sysfs.o device.o msgq.o
>   obj-$(CONFIG_GUNYAH) += gunyah.o
> \ No newline at end of file

Newline

> diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
> index 5f3832608020..2ade32bd9bdf 100644
> --- a/drivers/virt/gunyah/gunyah_private.h
> +++ b/drivers/virt/gunyah/gunyah_private.h
> @@ -9,4 +9,7 @@
>   int __init gunyah_bus_init(void);
>   void gunyah_bus_exit(void);
>   
> +int __init gh_msgq_init(void);
> +void gh_msgq_exit(void);
> +
>   #endif
> diff --git a/drivers/virt/gunyah/msgq.c b/drivers/virt/gunyah/msgq.c
> new file mode 100644
> index 000000000000..afc2572d3e7d
> --- /dev/null
> +++ b/drivers/virt/gunyah/msgq.c
> @@ -0,0 +1,223 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
> + */
> +
> +#include <linux/interrupt.h>
> +#include <linux/gunyah.h>
> +#include <linux/module.h>
> +#include <linux/printk.h>
> +#include <linux/init.h>
> +#include <linux/slab.h>
> +#include <linux/wait.h>
> +
> +#include "gunyah_private.h"
> +
> +struct gh_msgq {
> +	bool ready;
> +	wait_queue_head_t wq;
> +	spinlock_t lock;
> +};
> +
> +static irqreturn_t gh_msgq_irq_handler(int irq, void *dev)
> +{
> +	struct gh_msgq *msgq = dev;
> +
> +	spin_lock(&msgq->lock);
> +	msgq->ready = true;
> +	spin_unlock(&msgq->lock);
> +	wake_up_interruptible_all(&msgq->wq);
> +
> +	return IRQ_HANDLED;
> +}
> +
> +static int __gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size, u64 tx_flags)
> +{
> +	unsigned long flags, gh_error;
> +	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
> +	ssize_t ret;
> +	bool ready;
> +
> +	spin_lock_irqsave(&msgq->lock, flags);
> +	arch_gh_hypercall(GH_HYPERCALL_MSGQ_SEND, 5,
> +			  ghdev->capid, size, (uintptr_t)buff, tx_flags, 0,
> +			  gh_error, ready);
> +	switch (gh_error) {
> +	case GH_ERROR_OK:
> +		ret = 0;
> +		msgq->ready = ready;
> +		break;
> +	case GH_ERROR_MSGQUEUE_FULL:
> +		ret = -EAGAIN;
> +		msgq->ready = false;
> +		break;
> +	default:
> +		ret = gh_remap_error(gh_error);
> +		break;
> +	}
> +	spin_unlock_irqrestore(&msgq->lock, flags);
> +
> +	return ret;
> +}
> +
> +/**
> + * gh_msgq_send() - Send a message to the client running on a different VM
> + * @client: The client descriptor that was obtained via gh_msgq_register()
> + * @buff: Pointer to the buffer where the received data must be placed
> + * @buff_size: The size of the buffer space available
> + * @flags: Optional flags to pass to receive the data. For the list of flags,
> + *         see linux/gunyah/gh_msgq.h
> + *
> + * Returns: The number of bytes copied to buff. <0 if there was an error.
> + *
> + * Note: this function may sleep and should not be called from interrupt context
> + */
> +ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size,
> +		     const unsigned long flags)
> +{
> +	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
> +	ssize_t ret;
> +	u64 tx_flags = 0;
> +
> +	if (flags & GH_MSGQ_TX_PUSH)
> +		tx_flags |= GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH;
> +
> +	do {
> +		ret = __gh_msgq_send(ghdev, buff, size, tx_flags);
> +
> +		if (ret == -EAGAIN) {
> +			if (flags & GH_MSGQ_NONBLOCK)
> +				goto out;
> +			if (wait_event_interruptible(msgq->wq, msgq->ready))
> +				ret = -ERESTARTSYS;
> +		}
> +	} while (ret == -EAGAIN);

Any limit on the amount of retries? Can the driver wait forever here?

> +
> +out:
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(gh_msgq_send);

Both _send and _recv functions are not well designed. Can you call 
gh_msgq_send() on any gunyah_device? Yes. Will it work? No.

Could you please check if mailbox API work for you? It seems that it is 
what you are trying to implement on your own.

> +
> +static ssize_t __gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size)
> +{
> +	unsigned long flags, gh_error;
> +	size_t recv_size;
> +	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
> +	ssize_t ret;
> +	bool ready;
> +
> +	spin_lock_irqsave(&msgq->lock, flags);
> +
> +	arch_gh_hypercall(GH_HYPERCALL_MSGQ_RECV, 4,
> +			  ghdev->capid, (uintptr_t)buff, size, 0,
> +			  gh_error, recv_size, ready);
> +	switch (gh_error) {
> +	case GH_ERROR_OK:
> +		ret = recv_size;
> +		msgq->ready = ready;
> +		break;
> +	case GH_ERROR_MSGQUEUE_EMPTY:
> +		ret = -EAGAIN;
> +		msgq->ready = false;
> +		break;
> +	default:
> +		ret = gh_remap_error(gh_error);
> +		break;
> +	}
> +	spin_unlock_irqrestore(&msgq->lock, flags);
> +
> +	return ret;
> +}
> +
> +/**
> + * gh_msgq_recv() - Receive a message from the client running on a different VM
> + * @client: The client descriptor that was obtained via gh_msgq_register()
> + * @buff: Pointer to the buffer where the received data must be placed
> + * @buff_size: The size of the buffer space available
> + * @flags: Optional flags to pass to receive the data. For the list of flags,
> + *         see linux/gunyah/gh_msgq.h
> + *
> + * Returns: The number of bytes copied to buff. <0 if there was an error.
> + *
> + * Note: this function may sleep and should not be called from interrupt context
> + */
> +ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size,
> +		     const unsigned long flags)
> +{
> +	struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
> +	ssize_t ret;
> +
> +	do {
> +		ret = __gh_msgq_recv(ghdev, buff, size);
> +
> +		if (ret == -EAGAIN) {
> +			if (flags & GH_MSGQ_NONBLOCK)
> +				goto out;
> +			if (wait_event_interruptible(msgq->wq, msgq->ready))
> +				ret = -ERESTARTSYS;
> +		}
> +	} while (ret == -EAGAIN);
> +
> +out:
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(gh_msgq_recv);
> +
> +static int gh_msgq_probe(struct gunyah_device *ghdev)
> +{
> +	struct gh_msgq *msgq;
> +
> +	msgq = devm_kzalloc(&ghdev->dev, sizeof(*msgq), GFP_KERNEL);
> +	if (!msgq)
> +		return -ENOMEM;
> +	ghdev_set_drvdata(ghdev, msgq);
> +
> +	msgq->ready = true; /* Assume we can use the message queue right away */
> +	init_waitqueue_head(&msgq->wq);
> +	spin_lock_init(&msgq->lock);
> +
> +	return devm_request_irq(&ghdev->dev, ghdev->irq, gh_msgq_irq_handler, 0,
> +				dev_name(&ghdev->dev), msgq);
> +}
> +
> +static struct gunyah_driver gh_msgq_tx_driver = {
> +	.driver = {
> +		.name = "gh_msgq_tx",
> +		.owner = THIS_MODULE,
> +	},
> +	.type = GUNYAH_DEVICE_TYPE_MSGQ_TX,
> +	.probe = gh_msgq_probe,
> +};
> +
> +static struct gunyah_driver gh_msgq_rx_driver = {
> +	.driver = {
> +		.name = "gh_msgq_rx",
> +		.owner = THIS_MODULE,
> +	},
> +	.type = GUNYAH_DEVICE_TYPE_MSGQ_RX,
> +	.probe = gh_msgq_probe,

If you have to duplicate the whole device structure just to bind to two 
difference devices, it looks like a bad abstraction. Please check how 
other busses have solved this issue. They did, believe me.

> +};

MODULE_DEVICE_TABLE() ?

> +
> +int __init gh_msgq_init(void)
> +{
> +	int ret;
> +
> +	ret = gunyah_register_driver(&gh_msgq_tx_driver);
> +	if (ret)
> +		return ret;
> +
> +	ret = gunyah_register_driver(&gh_msgq_rx_driver);
> +	if (ret)
> +		goto err_rx;
> +
> +	return ret;
> +err_rx:
> +	gunyah_unregister_driver(&gh_msgq_tx_driver);
> +	return ret;
> +}
> +
> +void gh_msgq_exit(void)
> +{
> +	gunyah_unregister_driver(&gh_msgq_rx_driver);
> +	gunyah_unregister_driver(&gh_msgq_tx_driver);
> +}
> diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
> index 220560cb3b1c..7589689e5e92 100644
> --- a/drivers/virt/gunyah/sysfs.c
> +++ b/drivers/virt/gunyah/sysfs.c
> @@ -73,6 +73,8 @@ static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr,
>   
>   	if (GH_IDENTIFY_PARTITION_CSPACE(gunyah_api.flags))
>   		len += sysfs_emit_at(buffer, len, "cspace ");
> +	if (GH_IDENTIFY_MSGQUEUE(gunyah_api.flags))
> +		len += sysfs_emit_at(buffer, len, "message-queue ");

Again, this should go to the sysfs patch.

>   
>   	len += sysfs_emit_at(buffer, len, "\n");
>   	return len;
> @@ -142,7 +144,13 @@ static int __init gunyah_init(void)
>   	if (ret)
>   		goto err_sysfs;
>   
> +	ret = gh_msgq_init();
> +	if (ret)
> +		goto err_bus;
> +

Please stop beating everything in a single module. Having a provider 
(bus) and a consumer (drivers for this bus) in a single module sounds 
like an overkill. Or, a wrong abstraction.

Please remind me, why do you need gunyah bus in the first place? I could 
not find any other calls to gunyah_device_add in this series. Which 
devices do you expect to be added in future? Would they require separate 
drivers?

>   	return ret;
> +err_bus:
> +	gunyah_bus_exit();
>   err_sysfs:
>   	gh_sysfs_unregister();
>   	return ret;
> @@ -151,6 +159,7 @@ module_init(gunyah_init);
>   
>   static void __exit gunyah_exit(void)
>   {
> +	gh_msgq_exit();
>   	gunyah_bus_exit();
>   	gh_sysfs_unregister();
>   }
> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
> index ce35f4491773..099224f9d6d1 100644
> --- a/include/linux/gunyah.h
> +++ b/include/linux/gunyah.h
> @@ -6,6 +6,7 @@
>   #ifndef _GUNYAH_H
>   #define _GUNYAH_H
>   
> +#include <linux/platform_device.h>
>   #include <linux/device.h>
>   #include <linux/types.h>
>   #include <linux/errno.h>
> @@ -117,4 +118,16 @@ struct gunyah_driver {
>   int gunyah_register_driver(struct gunyah_driver *ghdrv);
>   void gunyah_unregister_driver(struct gunyah_driver *ghdrv);
>   
> +#define GH_MSGQ_MAX_MSG_SIZE	1024
> +
> +/* Possible flags to pass for Tx or Rx */
> +#define GH_MSGQ_TX_PUSH		BIT(0)
> +#define GH_MSGQ_NONBLOCK	BIT(32)
> +
> +ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t size,
> +		     const unsigned long flags);
> +ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t size,
> +		     const unsigned long flags);
> +
> +
>   #endif


-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Elliot Berman]

On 8/2/2022 1:14 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Gunyah message queues are unidirectional pipelines to communicate
>> between 2 virtual machines, but are typically paired to allow
>> bidirectional communication. The intended use case is for small control
>> messages between 2 VMs, as they support a maximum of 240 bytes.
>>
>> Message queues can be discovered either by resource manager or on the
>> devicetree. To support discovery on the devicetree, client drivers can
> 
> devicetree and discovery do not quite match to me. The device is delared 
> in the DT, not discovered.
>  >> use gh_msgq_platform_host_attach to allocate the tx and rx message
>> queues according to
>> Documentation/devicetree/bindings/gunyah/qcom,hypervisor.yml.
> 
> -ENOSUCHFILE
> 

Should be Documentaton/devicetree/bindings/firmware/gunyah-hypervisor.yaml

>>
>> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
>> ---
>>   arch/arm64/include/asm/gunyah.h      |   4 +
>>   drivers/virt/gunyah/Makefile         |   2 +-
>>   drivers/virt/gunyah/gunyah_private.h |   3 +
>>   drivers/virt/gunyah/msgq.c           | 223 +++++++++++++++++++++++++++
>>   drivers/virt/gunyah/sysfs.c          |   9 ++
>>   include/linux/gunyah.h               |  13 ++
>>   6 files changed, 253 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/virt/gunyah/msgq.c
>>
>> diff --git a/arch/arm64/include/asm/gunyah.h 
>> b/arch/arm64/include/asm/gunyah.h
>> index 3aee35009910..ba7398bd851b 100644
>> --- a/arch/arm64/include/asm/gunyah.h
>> +++ b/arch/arm64/include/asm/gunyah.h
>> @@ -27,6 +27,10 @@
>>                               | ((fn) & GH_CALL_FUNCTION_NUM_MASK))
>>   #define GH_HYPERCALL_HYP_IDENTIFY        GH_HYPERCALL(0x0000)
>> +#define GH_HYPERCALL_MSGQ_SEND            GH_HYPERCALL(0x001B)
>> +#define GH_HYPERCALL_MSGQ_RECV            GH_HYPERCALL(0x001C)
>> +
>> +#define GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH    BIT(0)
>>   #define ___gh_count_args(_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x
>> diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
>> index 3869fb7371df..94dc8e738911 100644
>> --- a/drivers/virt/gunyah/Makefile
>> +++ b/drivers/virt/gunyah/Makefile
>> @@ -1,4 +1,4 @@
>>   # SPDX-License-Identifier: GPL-2.0-only
>> -gunyah-y += sysfs.o device.o
>> +gunyah-y += sysfs.o device.o msgq.o
>>   obj-$(CONFIG_GUNYAH) += gunyah.o
>> \ No newline at end of file
> 
> Newline
> 
>> diff --git a/drivers/virt/gunyah/gunyah_private.h 
>> b/drivers/virt/gunyah/gunyah_private.h
>> index 5f3832608020..2ade32bd9bdf 100644
>> --- a/drivers/virt/gunyah/gunyah_private.h
>> +++ b/drivers/virt/gunyah/gunyah_private.h
>> @@ -9,4 +9,7 @@
>>   int __init gunyah_bus_init(void);
>>   void gunyah_bus_exit(void);
>> +int __init gh_msgq_init(void);
>> +void gh_msgq_exit(void);
>> +
>>   #endif
>> diff --git a/drivers/virt/gunyah/msgq.c b/drivers/virt/gunyah/msgq.c
>> new file mode 100644
>> index 000000000000..afc2572d3e7d
>> --- /dev/null
>> +++ b/drivers/virt/gunyah/msgq.c
>> @@ -0,0 +1,223 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights 
>> reserved.
>> + */
>> +
>> +#include <linux/interrupt.h>
>> +#include <linux/gunyah.h>
>> +#include <linux/module.h>
>> +#include <linux/printk.h>
>> +#include <linux/init.h>
>> +#include <linux/slab.h>
>> +#include <linux/wait.h>
>> +
>> +#include "gunyah_private.h"
>> +
>> +struct gh_msgq {
>> +    bool ready;
>> +    wait_queue_head_t wq;
>> +    spinlock_t lock;
>> +};
>> +
>> +static irqreturn_t gh_msgq_irq_handler(int irq, void *dev)
>> +{
>> +    struct gh_msgq *msgq = dev;
>> +
>> +    spin_lock(&msgq->lock);
>> +    msgq->ready = true;
>> +    spin_unlock(&msgq->lock);
>> +    wake_up_interruptible_all(&msgq->wq);
>> +
>> +    return IRQ_HANDLED;
>> +}
>> +
>> +static int __gh_msgq_send(struct gunyah_device *ghdev, void *buff, 
>> size_t size, u64 tx_flags)
>> +{
>> +    unsigned long flags, gh_error;
>> +    struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
>> +    ssize_t ret;
>> +    bool ready;
>> +
>> +    spin_lock_irqsave(&msgq->lock, flags);
>> +    arch_gh_hypercall(GH_HYPERCALL_MSGQ_SEND, 5,
>> +              ghdev->capid, size, (uintptr_t)buff, tx_flags, 0,
>> +              gh_error, ready);
>> +    switch (gh_error) {
>> +    case GH_ERROR_OK:
>> +        ret = 0;
>> +        msgq->ready = ready;
>> +        break;
>> +    case GH_ERROR_MSGQUEUE_FULL:
>> +        ret = -EAGAIN;
>> +        msgq->ready = false;
>> +        break;
>> +    default:
>> +        ret = gh_remap_error(gh_error);
>> +        break;
>> +    }
>> +    spin_unlock_irqrestore(&msgq->lock, flags);
>> +
>> +    return ret;
>> +}
>> +
>> +/**
>> + * gh_msgq_send() - Send a message to the client running on a 
>> different VM
>> + * @client: The client descriptor that was obtained via 
>> gh_msgq_register()
>> + * @buff: Pointer to the buffer where the received data must be placed
>> + * @buff_size: The size of the buffer space available
>> + * @flags: Optional flags to pass to receive the data. For the list 
>> of flags,
>> + *         see linux/gunyah/gh_msgq.h
>> + *
>> + * Returns: The number of bytes copied to buff. <0 if there was an 
>> error.
>> + *
>> + * Note: this function may sleep and should not be called from 
>> interrupt context
>> + */
>> +ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t 
>> size,
>> +             const unsigned long flags)
>> +{
>> +    struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
>> +    ssize_t ret;
>> +    u64 tx_flags = 0;
>> +
>> +    if (flags & GH_MSGQ_TX_PUSH)
>> +        tx_flags |= GH_HYPERCALL_MSGQ_SEND_FLAGS_PUSH;
>> +
>> +    do {
>> +        ret = __gh_msgq_send(ghdev, buff, size, tx_flags);
>> +
>> +        if (ret == -EAGAIN) {
>> +            if (flags & GH_MSGQ_NONBLOCK)
>> +                goto out;
>> +            if (wait_event_interruptible(msgq->wq, msgq->ready))
>> +                ret = -ERESTARTSYS;
>> +        }
>> +    } while (ret == -EAGAIN);
> 
> Any limit on the amount of retries? Can the driver wait forever here?
> 
>> +
>> +out:
>> +    return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(gh_msgq_send);
> 
> Both _send and _recv functions are not well designed. Can you call 
> gh_msgq_send() on any gunyah_device? Yes. Will it work? No.
> 

My intention here is to rely on hypervisor to properly complain about 
it. I thought it better to not have redundant checks, but I can add it 
in the Linux layer as well.

> Could you please check if mailbox API work for you? It seems that it is 
> what you are trying to implement on your own.
> 

My understanding is that mailbox API was designed for queuing single 
register accesses. The mailbox APIs aren't intended to queue up 
arbitrary length messages like needed here. rpmsg is similar in the 
sense that it had variable length messages and doesn't use the mailbox 
framework for this reason.

>> +
>> +static ssize_t __gh_msgq_recv(struct gunyah_device *ghdev, void 
>> *buff, size_t size)
>> +{
>> +    unsigned long flags, gh_error;
>> +    size_t recv_size;
>> +    struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
>> +    ssize_t ret;
>> +    bool ready;
>> +
>> +    spin_lock_irqsave(&msgq->lock, flags);
>> +
>> +    arch_gh_hypercall(GH_HYPERCALL_MSGQ_RECV, 4,
>> +              ghdev->capid, (uintptr_t)buff, size, 0,
>> +              gh_error, recv_size, ready);
>> +    switch (gh_error) {
>> +    case GH_ERROR_OK:
>> +        ret = recv_size;
>> +        msgq->ready = ready;
>> +        break;
>> +    case GH_ERROR_MSGQUEUE_EMPTY:
>> +        ret = -EAGAIN;
>> +        msgq->ready = false;
>> +        break;
>> +    default:
>> +        ret = gh_remap_error(gh_error);
>> +        break;
>> +    }
>> +    spin_unlock_irqrestore(&msgq->lock, flags);
>> +
>> +    return ret;
>> +}
>> +
>> +/**
>> + * gh_msgq_recv() - Receive a message from the client running on a 
>> different VM
>> + * @client: The client descriptor that was obtained via 
>> gh_msgq_register()
>> + * @buff: Pointer to the buffer where the received data must be placed
>> + * @buff_size: The size of the buffer space available
>> + * @flags: Optional flags to pass to receive the data. For the list 
>> of flags,
>> + *         see linux/gunyah/gh_msgq.h
>> + *
>> + * Returns: The number of bytes copied to buff. <0 if there was an 
>> error.
>> + *
>> + * Note: this function may sleep and should not be called from 
>> interrupt context
>> + */
>> +ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t 
>> size,
>> +             const unsigned long flags)
>> +{
>> +    struct gh_msgq *msgq = ghdev_get_drvdata(ghdev);
>> +    ssize_t ret;
>> +
>> +    do {
>> +        ret = __gh_msgq_recv(ghdev, buff, size);
>> +
>> +        if (ret == -EAGAIN) {
>> +            if (flags & GH_MSGQ_NONBLOCK)
>> +                goto out;
>> +            if (wait_event_interruptible(msgq->wq, msgq->ready))
>> +                ret = -ERESTARTSYS;
>> +        }
>> +    } while (ret == -EAGAIN);
>> +
>> +out:
>> +    return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(gh_msgq_recv);
>> +
>> +static int gh_msgq_probe(struct gunyah_device *ghdev)
>> +{
>> +    struct gh_msgq *msgq;
>> +
>> +    msgq = devm_kzalloc(&ghdev->dev, sizeof(*msgq), GFP_KERNEL);
>> +    if (!msgq)
>> +        return -ENOMEM;
>> +    ghdev_set_drvdata(ghdev, msgq);
>> +
>> +    msgq->ready = true; /* Assume we can use the message queue right 
>> away */
>> +    init_waitqueue_head(&msgq->wq);
>> +    spin_lock_init(&msgq->lock);
>> +
>> +    return devm_request_irq(&ghdev->dev, ghdev->irq, 
>> gh_msgq_irq_handler, 0,
>> +                dev_name(&ghdev->dev), msgq);
>> +}
>> +
>> +static struct gunyah_driver gh_msgq_tx_driver = {
>> +    .driver = {
>> +        .name = "gh_msgq_tx",
>> +        .owner = THIS_MODULE,
>> +    },
>> +    .type = GUNYAH_DEVICE_TYPE_MSGQ_TX,
>> +    .probe = gh_msgq_probe,
>> +};
>> +
>> +static struct gunyah_driver gh_msgq_rx_driver = {
>> +    .driver = {
>> +        .name = "gh_msgq_rx",
>> +        .owner = THIS_MODULE,
>> +    },
>> +    .type = GUNYAH_DEVICE_TYPE_MSGQ_RX,
>> +    .probe = gh_msgq_probe,
> 
> If you have to duplicate the whole device structure just to bind to two 
> difference devices, it looks like a bad abstraction. Please check how 
> other busses have solved this issue. They did, believe me.
> 
>> +};
> 
> MODULE_DEVICE_TABLE() ?
> 
>> +
>> +int __init gh_msgq_init(void)
>> +{
>> +    int ret;
>> +
>> +    ret = gunyah_register_driver(&gh_msgq_tx_driver);
>> +    if (ret)
>> +        return ret;
>> +
>> +    ret = gunyah_register_driver(&gh_msgq_rx_driver);
>> +    if (ret)
>> +        goto err_rx;
>> +
>> +    return ret;
>> +err_rx:
>> +    gunyah_unregister_driver(&gh_msgq_tx_driver);
>> +    return ret;
>> +}
>> +
>> +void gh_msgq_exit(void)
>> +{
>> +    gunyah_unregister_driver(&gh_msgq_rx_driver);
>> +    gunyah_unregister_driver(&gh_msgq_tx_driver);
>> +}
>> diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
>> index 220560cb3b1c..7589689e5e92 100644
>> --- a/drivers/virt/gunyah/sysfs.c
>> +++ b/drivers/virt/gunyah/sysfs.c
>> @@ -73,6 +73,8 @@ static ssize_t features_show(struct kobject *kobj, 
>> struct kobj_attribute *attr,
>>       if (GH_IDENTIFY_PARTITION_CSPACE(gunyah_api.flags))
>>           len += sysfs_emit_at(buffer, len, "cspace ");
>> +    if (GH_IDENTIFY_MSGQUEUE(gunyah_api.flags))
>> +        len += sysfs_emit_at(buffer, len, "message-queue ");
> 
> Again, this should go to the sysfs patch.
> 
>>       len += sysfs_emit_at(buffer, len, "\n");
>>       return len;
>> @@ -142,7 +144,13 @@ static int __init gunyah_init(void)
>>       if (ret)
>>           goto err_sysfs;
>> +    ret = gh_msgq_init();
>> +    if (ret)
>> +        goto err_bus;
>> +
> 
> Please stop beating everything in a single module. Having a provider 
> (bus) and a consumer (drivers for this bus) in a single module sounds 
> like an overkill. Or, a wrong abstraction.
> 
> Please remind me, why do you need gunyah bus in the first place? I could 
> not find any other calls to gunyah_device_add in this series. Which 
> devices do you expect to be added in future? Would they require separate 
> drivers?
> 

In a future series, I'll add the support to load other virtual machines. 
When running other virtual machines, additional gunyah devices are 
needed for doorbells (e.g. to emulate interrupts for paravirtualized 
devices) and to represent the vCPUs of that other VM. Other gunyah 
devices are also possible, but those are the immediate devices coming 
over the horizon.

I had an offline discussion with Bjorn and he was recommending dropping 
the bus/device/driver design entirely.

>>       return ret;
>> +err_bus:
>> +    gunyah_bus_exit();
>>   err_sysfs:
>>       gh_sysfs_unregister();
>>       return ret;
>> @@ -151,6 +159,7 @@ module_init(gunyah_init);
>>   static void __exit gunyah_exit(void)
>>   {
>> +    gh_msgq_exit();
>>       gunyah_bus_exit();
>>       gh_sysfs_unregister();
>>   }
>> diff --git a/include/linux/gunyah.h b/include/linux/gunyah.h
>> index ce35f4491773..099224f9d6d1 100644
>> --- a/include/linux/gunyah.h
>> +++ b/include/linux/gunyah.h
>> @@ -6,6 +6,7 @@
>>   #ifndef _GUNYAH_H
>>   #define _GUNYAH_H
>> +#include <linux/platform_device.h>
>>   #include <linux/device.h>
>>   #include <linux/types.h>
>>   #include <linux/errno.h>
>> @@ -117,4 +118,16 @@ struct gunyah_driver {
>>   int gunyah_register_driver(struct gunyah_driver *ghdrv);
>>   void gunyah_unregister_driver(struct gunyah_driver *ghdrv);
>> +#define GH_MSGQ_MAX_MSG_SIZE    1024
>> +
>> +/* Possible flags to pass for Tx or Rx */
>> +#define GH_MSGQ_TX_PUSH        BIT(0)
>> +#define GH_MSGQ_NONBLOCK    BIT(32)
>> +
>> +ssize_t gh_msgq_send(struct gunyah_device *ghdev, void *buff, size_t 
>> size,
>> +             const unsigned long flags);
>> +ssize_t gh_msgq_recv(struct gunyah_device *ghdev, void *buff, size_t 
>> size,
>> +             const unsigned long flags);
>> +
>> +
>>   #endif
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Marc Zyngier]

On Mon, 08 Aug 2022 23:22:48 +0100,
Elliot Berman <quic_eberman@quicinc.com> wrote:
> 
> In a future series, I'll add the support to load other virtual
> machines. When running other virtual machines, additional gunyah
> devices are needed for doorbells (e.g. to emulate interrupts for
> paravirtualized devices) and to represent the vCPUs of that other
> VM. Other gunyah devices are also possible, but those are the
> immediate devices coming over the horizon.

Can you elaborate on this "doorbell" aspect? If you signal interrupts
to guests, they should be signalled as actual interrupts, not as some
hypervisor-specific events, as we rely on the interrupt semantics for
most things.

Or are you talking about injecting an interrupt from a guest into
another, the doorbell representing an interrupt source?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Elliot Berman]

On 8/9/2022 4:29 AM, Marc Zyngier wrote:
> On Mon, 08 Aug 2022 23:22:48 +0100,
> Elliot Berman <quic_eberman@quicinc.com> wrote:
>>
>> In a future series, I'll add the support to load other virtual
>> machines. When running other virtual machines, additional gunyah
>> devices are needed for doorbells (e.g. to emulate interrupts for
>> paravirtualized devices) and to represent the vCPUs of that other
>> VM. Other gunyah devices are also possible, but those are the
>> immediate devices coming over the horizon.
> 
> Can you elaborate on this "doorbell" aspect? If you signal interrupts
> to guests, they should be signalled as actual interrupts, not as some
> hypervisor-specific events, as we rely on the interrupt semantics for
> most things.
> 
> Or are you talking about injecting an interrupt from a guest into
> another, the doorbell representing an interrupt source?
> 

Doorbells can operate either of these modes:
  1. As simple interrupt sources. The doorbell sender makes a hypercall
     and an interrupt is raised on the receiver. The hypervisor can be
     configured to raise a specific SPI on the receiver VM and simply
     acknowledging the SPI is enough to clear the interrupt assert. No
     hypervisor-specific code is needed on the receiver to handle these
     interrupts. This is the mode one would expect to use for
     paravirtualized devices.
  2. As hypervisor-specific events which must be acknowledged using
     hypercalls. We aren't currently using this advanced use-case and no
     plans currently to post these. However, I can try to briefly
     explain: These doorbells can operate on a bitfield and the sender
     can assert flags on the bitmask; the receiver can decide which bits
     should trigger the interrupt and which SPI the doorbell "runs" on.
     The "user story" for this doorbell is to support multiple sender
     using the same doorbell object. Each sender has a few designated
     bits they should set. The receiver can choose which events it wants
     an interrupt to be raised for and then can process all the pending
     events. To re-iterate, we don't have an interesting use-case for
     this yet, so don't plan on post patches for this second mode of
     doorbell.


> Thanks,
> 
> 	M.
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 07/11] gunyah: msgq: Add Gunyah message queues

[Dmitry Baryshkov]

On 09/08/2022 19:50, Elliot Berman wrote:
> 
> 
> On 8/9/2022 4:29 AM, Marc Zyngier wrote:
>> On Mon, 08 Aug 2022 23:22:48 +0100,
>> Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>
>>> In a future series, I'll add the support to load other virtual
>>> machines. When running other virtual machines, additional gunyah
>>> devices are needed for doorbells (e.g. to emulate interrupts for
>>> paravirtualized devices) and to represent the vCPUs of that other
>>> VM. Other gunyah devices are also possible, but those are the
>>> immediate devices coming over the horizon.
>>
>> Can you elaborate on this "doorbell" aspect? If you signal interrupts
>> to guests, they should be signalled as actual interrupts, not as some
>> hypervisor-specific events, as we rely on the interrupt semantics for
>> most things.
>>
>> Or are you talking about injecting an interrupt from a guest into
>> another, the doorbell representing an interrupt source?
>>
> 
> Doorbells can operate either of these modes:
>   1. As simple interrupt sources. The doorbell sender makes a hypercall
>      and an interrupt is raised on the receiver. The hypervisor can be
>      configured to raise a specific SPI on the receiver VM and simply
>      acknowledging the SPI is enough to clear the interrupt assert. No
>      hypervisor-specific code is needed on the receiver to handle these
>      interrupts. This is the mode one would expect to use for
>      paravirtualized devices.

This sounds good.

>   2. As hypervisor-specific events which must be acknowledged using
>      hypercalls. We aren't currently using this advanced use-case and no
>      plans currently to post these. However, I can try to briefly
>      explain: These doorbells can operate on a bitfield and the sender
>      can assert flags on the bitmask; the receiver can decide which bits
>      should trigger the interrupt and which SPI the doorbell "runs" on.
>      The "user story" for this doorbell is to support multiple sender
>      using the same doorbell object. Each sender has a few designated
>      bits they should set. The receiver can choose which events it wants
>      an interrupt to be raised for and then can process all the pending
>      events. To re-iterate, we don't have an interesting use-case for
>      this yet, so don't plan on post patches for this second mode of
>      doorbell.

Well. For me this sounds like 'we have such capability, no real usecase, 
but we want to support it anyway' kind of story. As history has shown 
multiple times, the order should be the opposite one. First you have the 
use case, then you create the API for it. Otherwise it is very easy to 
end up with the abstraction that looks good on the API side, but is very 
hard to fit into the actual user code.

I would suggest to drop the second bullet for now and focus on getting 
the simple doorbells done and accepted into mainline.

-- 
With best wishes
Dmitry


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 08/11] gunyah: rsc_mgr: Add resource manager RPC core

[Elliot Berman]

The resource manager is a special virtual machine which is always
running on a Gunyah system. It provides APIs for creating and destroying
VMs, secure memory management, sharing/lending of memory between VMs,
and setup of inter-VM communication. Calls to the resource manager are
made via message queues.

This patch implements the basic probing and RPC mechanism to make those
API calls. Request/response calls can be made with gh_rm_call.
Drivers can also register to notifications pushed by RM via
gh_rm_register_notifier

Specific API calls that resource manager supports will be implemented in
subsequent patches.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 MAINTAINERS                          |   2 +-
 drivers/virt/gunyah/Kconfig          |   1 +
 drivers/virt/gunyah/Makefile         |   1 +
 drivers/virt/gunyah/gunyah_private.h |   3 +
 drivers/virt/gunyah/rsc_mgr.c        | 623 +++++++++++++++++++++++++++
 drivers/virt/gunyah/rsc_mgr.h        |  34 ++
 drivers/virt/gunyah/sysfs.c          |   7 +
 include/linux/gunyah_rsc_mgr.h       |  29 ++
 8 files changed, 699 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/gunyah/rsc_mgr.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.h
 create mode 100644 include/linux/gunyah_rsc_mgr.h

diff --git a/MAINTAINERS b/MAINTAINERS
index e63c51ee1a2a..dcd961d6623c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8746,7 +8746,7 @@ F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
 F:	arch/arm64/include/asm/gunyah.h
 F:	drivers/virt/gunyah/
-F:	include/linux/gunyah.h
+F:	include/linux/gunyah*.h
 
 HABANALABS PCI DRIVER
 M:	Oded Gabbay <ogabbay@kernel.org>
diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig
index 1e493017885e..0adb6efd4848 100644
--- a/drivers/virt/gunyah/Kconfig
+++ b/drivers/virt/gunyah/Kconfig
@@ -4,6 +4,7 @@ config GUNYAH
 	tristate "Gunyah Virtualization drivers"
 	depends on ARM64
 	select SYS_HYPERVISOR
+	select AUXILIARY_BUS
 	help
 	  The Gunyah drivers are the helper interfaces that runs in a guest VM
 	  such as basic inter-VM IPC and signaling mechanisms and higher level
diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index 94dc8e738911..86655bca8944 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 gunyah-y += sysfs.o device.o msgq.o
+gunyah-y += rsc_mgr.o
 obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
index 2ade32bd9bdf..6483ffa8c15d 100644
--- a/drivers/virt/gunyah/gunyah_private.h
+++ b/drivers/virt/gunyah/gunyah_private.h
@@ -12,4 +12,7 @@ void gunyah_bus_exit(void);
 int __init gh_msgq_init(void);
 void gh_msgq_exit(void);
 
+int __init gh_rsc_mgr_init(void);
+void gh_rsc_mgr_exit(void);
+
 #endif
diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c
new file mode 100644
index 000000000000..b8268ee02fab
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr.c
@@ -0,0 +1,623 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "gh_rsc_mgr: " fmt
+
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/gunyah.h>
+#include <linux/module.h>
+#include <linux/of_irq.h>
+#include <linux/kthread.h>
+#include <linux/notifier.h>
+#include <linux/irqdomain.h>
+#include <linux/workqueue.h>
+#include <linux/completion.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/gunyah_rsc_mgr.h>
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+
+#include "gunyah_private.h"
+#include "rsc_mgr.h"
+
+/* Resource Manager Header */
+struct gh_rm_rpc_hdr {
+	u8 version:4,
+		hdr_words:4;
+	u8 type:2,
+		fragments:6;
+	u16 seq;
+	u32 msg_id;
+} __packed;
+
+/* Standard reply header */
+struct gh_rm_rpc_reply_hdr {
+	struct gh_rm_rpc_hdr rpc_hdr;
+	u32 err_code;
+} __packed;
+
+/* RPC Header versions */
+#define GH_RM_RPC_HDR_VERSION_ONE	0x1
+
+/* RPC Header words */
+#define GH_RM_RPC_HDR_WORDS		0x2
+
+/* RPC Message types */
+#define GH_RM_RPC_TYPE_CONT		0x0
+#define GH_RM_RPC_TYPE_REQ		0x1
+#define GH_RM_RPC_TYPE_RPLY		0x2
+#define GH_RM_RPC_TYPE_NOTIF		0x3
+
+#define GH_RM_MAX_NUM_FRAGMENTS		62
+
+#define GH_RM_MAX_MSG_SIZE	(GH_MSGQ_MAX_MSG_SIZE - sizeof(struct gh_rm_rpc_hdr))
+
+/**
+ * struct gh_rm_connection - Represents a complete message from resource manager
+ * @payload: Combined payload of all the fragments (i.e. msg headers stripped off).
+ * @size: Size of the payload.
+ * @ret: Linux return code, set in case there was an error processing the connection.
+ * @msg_id: Message ID from the header.
+ * @type: GH_RM_RPC_TYPE_RPLY or GH_RM_RPC_TYPE_NOTIF.
+ * @num_fragments: total number of fragments expected to be received for this connection.
+ * @fragments_recieved: fragments received so far.
+ * @rm_error: For request/reply sequences with standard replies.
+ * @seq: Sequence ID for the main message.
+ */
+struct gh_rm_connection {
+	void *payload;
+	size_t size;
+	int ret;
+	u32 msg_id;
+	u8 type;
+
+	u8 num_fragments;
+	u8 fragments_received;
+
+	/* only for req/reply sequence */
+	u32 rm_error;
+	u16 seq;
+	struct completion seq_done;
+};
+
+struct gh_rm_notif_complete {
+	struct gh_rm_connection *conn;
+	struct work_struct work;
+};
+
+struct gh_rsc_mgr {
+	struct task_struct *recv_task;
+	struct gunyah_device *msgq_tx, *msgq_rx;
+
+	struct idr call_idr;
+	struct mutex call_idr_lock;
+
+	struct mutex send_lock;
+};
+
+static struct gh_rsc_mgr *__rsc_mgr;
+SRCU_NOTIFIER_HEAD_STATIC(gh_rm_notifier);
+
+static struct gh_rm_connection *gh_rm_alloc_connection(u32 msg_id, u8 type)
+{
+	struct gh_rm_connection *connection;
+
+	connection = kzalloc(sizeof(*connection), GFP_KERNEL);
+	if (!connection)
+		return NULL;
+
+	connection->type = type;
+	connection->msg_id = msg_id;
+
+	return connection;
+}
+
+/**
+ * gh_rm_init_connection_payload() - Fills the first message for a connection.
+ */
+static int gh_rm_init_connection_payload(struct gh_rm_connection *connection, void *msg,
+					size_t hdr_size, size_t payload_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	size_t max_buf_size;
+
+	connection->num_fragments = hdr->fragments;
+	connection->fragments_received = 0;
+	connection->type = hdr->type;
+
+	/* There's not going to be any payload, no need to allocate buffer. */
+	if (!payload_size && !connection->num_fragments)
+		return 0;
+
+	/*
+	 * maximum payload size is GH_MSGQ_MAX_MSG_SIZE - hdr_size
+	 * and can received (hdr->fragments + 1) of those
+	 */
+	max_buf_size = (GH_MSGQ_MAX_MSG_SIZE - hdr_size) * (hdr->fragments + 1);
+
+	connection->payload = kzalloc(max_buf_size, GFP_KERNEL);
+	if (!connection->payload)
+		return -ENOMEM;
+
+	memcpy(connection->payload, msg + hdr_size, payload_size);
+	connection->size = payload_size;
+	return 0;
+}
+
+static void gh_rm_notif_work(struct work_struct *work)
+{
+	struct gh_rm_notif_complete *notif = container_of(work, struct gh_rm_notif_complete, work);
+	struct gh_rm_connection *connection = notif->conn;
+	u32 notif_id = connection->msg_id;
+	struct gh_rm_notification notification = {
+		.buff = connection->payload,
+		.size = connection->size,
+	};
+
+	srcu_notifier_call_chain(&gh_rm_notifier, notif_id, &notification);
+
+	kfree(connection->payload);
+	kfree(connection);
+	kfree(notif);
+}
+
+static struct gh_rm_connection *gh_rm_process_notif(struct gh_rsc_mgr *rsc_mgr,
+						    void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	struct gh_rm_connection *connection;
+
+	connection = gh_rm_alloc_connection(hdr->msg_id, hdr->type);
+	if (!connection) {
+		pr_err("Failed to alloc connection for notification, dropping.\n");
+		return NULL;
+	}
+
+	if (gh_rm_init_connection_payload(connection, msg, sizeof(*hdr), msg_size - sizeof(*hdr))) {
+		pr_err("Failed to alloc connection buffer for notification, dropping.\n");
+		kfree(connection);
+		return NULL;
+	}
+
+	return connection;
+}
+
+static struct gh_rm_connection *gh_rm_process_rply(struct gh_rsc_mgr *rsc_mgr,
+						   void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_reply_hdr *reply_hdr = msg;
+	struct gh_rm_rpc_hdr *hdr = msg;
+	struct gh_rm_connection *connection;
+
+	if (mutex_lock_interruptible(&rsc_mgr->call_idr_lock))
+		return ERR_PTR(-ERESTARTSYS);
+
+	connection = idr_find(&rsc_mgr->call_idr, hdr->seq);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	if (!connection) {
+		pr_err("Failed to find connection for sequence %u\n", hdr->seq);
+		return NULL;
+	}
+	if (connection->msg_id != hdr->msg_id) {
+		pr_err("Reply for sequence %u expected msg_id: %x but got %x\n", hdr->seq,
+			connection->msg_id, hdr->msg_id);
+		/*
+		 * Don't complete connection and error the client, maybe resource manager will
+		 * send us the expected reply sequence soon.
+		 */
+		return NULL;
+	}
+
+	if (gh_rm_init_connection_payload(connection, msg, sizeof(*reply_hdr),
+					msg_size - sizeof(*reply_hdr))) {
+		pr_err("Failed to alloc connection buffer for sequence %d\n", hdr->seq);
+		/* Send connection complete and error the client. */
+		connection->ret = -ENOMEM;
+		complete(&connection->seq_done);
+		return NULL;
+	}
+
+	connection->rm_error = reply_hdr->err_code;
+	return connection;
+}
+
+static void gh_rm_process_cont(struct gh_rm_connection *connection, void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	size_t payload_size = msg_size - sizeof(*hdr);
+
+	/*
+	 * hdr->fragments and hdr->msg_id preserves the value from first reply or notif message.
+	 * For sake of sanity, check if it's still intact.
+	 */
+	if (connection->msg_id != hdr->msg_id)
+		pr_warn("Appending mismatched continuation with id %d to connection with id %d\n",
+			hdr->msg_id, connection->msg_id);
+	if (connection->num_fragments != hdr->fragments)
+		pr_warn("Number of fragments mismatch for seq: %d\n", hdr->seq);
+
+	memcpy(connection->payload + connection->size, msg + sizeof(*hdr), payload_size);
+	connection->size += payload_size;
+	connection->fragments_received++;
+}
+
+static bool gh_rm_complete_connection(struct gh_rm_connection *connection)
+{
+	struct gh_rm_notif_complete *notif_work;
+
+	if (!connection)
+		return false;
+
+	if (connection->fragments_received != connection->num_fragments)
+		return false;
+
+	switch (connection->type) {
+	case GH_RM_RPC_TYPE_RPLY:
+		complete(&connection->seq_done);
+		break;
+	case GH_RM_RPC_TYPE_NOTIF:
+		notif_work = kzalloc(sizeof(*notif_work), GFP_KERNEL);
+		if (notif_work == NULL)
+			break;
+
+		notif_work->conn = connection;
+		INIT_WORK(&notif_work->work, gh_rm_notif_work);
+
+		schedule_work(&notif_work->work);
+		break;
+	default:
+		pr_err("Invalid message type (%d) received\n", connection->type);
+		break;
+	}
+
+	return true;
+}
+
+static void gh_rm_abort_connection(struct gh_rm_connection *connection)
+{
+	switch (connection->type) {
+	case GH_RM_RPC_TYPE_RPLY:
+		connection->ret = -EIO;
+		complete(&connection->seq_done);
+		break;
+	case GH_RM_RPC_TYPE_NOTIF:
+		fallthrough;
+	default:
+		kfree(connection->payload);
+		kfree(connection);
+	}
+}
+
+static int gh_rm_recv_task_fn(void *data)
+{
+	struct gh_rsc_mgr *rsc_mgr = data;
+	struct gh_rm_connection *connection = NULL;
+	struct gh_rm_rpc_hdr *hdr = NULL;
+	ssize_t msg_size;
+	void *msg;
+
+	msg = kzalloc(GH_MSGQ_MAX_MSG_SIZE, GFP_KERNEL);
+	if (!msg)
+		return -ENOMEM;
+
+	while (!kthread_should_stop()) {
+		/* Block until a new message is received */
+		msg_size = gh_msgq_recv(rsc_mgr->msgq_rx, msg, GH_MSGQ_MAX_MSG_SIZE, 0);
+		if (msg_size < 0) {
+			pr_err("Failed to receive the message: %ld\n", msg_size);
+			continue;
+		} else if (msg_size <= sizeof(struct gh_rm_rpc_hdr)) {
+			pr_err("Invalid message size received: %ld is too small\n", msg_size);
+			continue;
+		}
+
+		hdr = msg;
+		switch (hdr->type) {
+		case GH_RM_RPC_TYPE_NOTIF:
+			if (connection) {
+				/* Not possible per protocol. Do something better than BUG_ON */
+				pr_warn("Received start of new notification without finishing existing message series.\n");
+				gh_rm_abort_connection(connection);
+			}
+			connection = gh_rm_process_notif(rsc_mgr, msg, msg_size);
+			break;
+		case GH_RM_RPC_TYPE_RPLY:
+			if (connection) {
+				/* Not possible per protocol. Do something better than BUG_ON */
+				pr_warn("Received start of new reply without finishing existing message series.\n");
+				gh_rm_abort_connection(connection);
+			}
+			connection = gh_rm_process_rply(rsc_mgr, msg, msg_size);
+			break;
+		case GH_RM_RPC_TYPE_CONT:
+			if (!connection) {
+				pr_warn("Received a continuation message without receiving initial message\n");
+				break;
+			}
+			gh_rm_process_cont(connection, msg, msg_size);
+			break;
+		default:
+			pr_err("Invalid message type (%d) received\n", hdr->type);
+			continue;
+		}
+
+		if (gh_rm_complete_connection(connection))
+			connection = NULL;
+	}
+
+	return 0;
+}
+
+static int gh_rm_send_request(struct gh_rsc_mgr *rsc_mgr, u32 message_id,
+				const void *req_buff, size_t req_buff_size,
+				struct gh_rm_connection *connection)
+{
+	size_t buff_size_remaining = req_buff_size;
+	const void *req_buff_curr = req_buff;
+	struct gh_rm_rpc_hdr *hdr;
+	unsigned long tx_flags;
+	u32 num_fragments = 0;
+	size_t payload_size;
+	void *msg;
+	int i, ret = 0;
+
+	if (req_buff_size > GH_RM_MAX_MSG_SIZE)
+		num_fragments = req_buff_size / GH_RM_MAX_MSG_SIZE;
+
+	if (WARN(num_fragments > GH_RM_MAX_NUM_FRAGMENTS,
+		 "Limit exceeded for the number of fragments: %u\n", num_fragments))
+		return -E2BIG;
+
+	/*
+	 * The above calculation also includes the count for the 'request' packet.
+	 * Exclude it as the header needs to fill the num. of fragments to follow.
+	 */
+	if (num_fragments)
+		num_fragments--;
+
+	if (mutex_lock_interruptible(&rsc_mgr->send_lock))
+		return -ERESTARTSYS;
+
+	msg = kzalloc(GH_MSGQ_MAX_MSG_SIZE, GFP_KERNEL);
+	if (!msg) {
+		mutex_unlock(&rsc_mgr->send_lock);
+		return -ENOMEM;
+	}
+
+	/* Consider also the 'request' packet for the loop count */
+	for (i = 0; i <= num_fragments; i++) {
+		if (buff_size_remaining > GH_RM_MAX_MSG_SIZE) {
+			payload_size = GH_RM_MAX_MSG_SIZE;
+			buff_size_remaining -= payload_size;
+		} else {
+			payload_size = buff_size_remaining;
+		}
+
+		memset(msg, 0, GH_MSGQ_MAX_MSG_SIZE);
+
+		/* Fill header */
+		hdr = msg;
+		hdr->version = GH_RM_RPC_HDR_VERSION_ONE;
+		hdr->hdr_words = GH_RM_RPC_HDR_WORDS;
+		hdr->type = i == 0 ? GH_RM_RPC_TYPE_REQ : GH_RM_RPC_TYPE_CONT;
+		hdr->fragments = num_fragments;
+		hdr->seq = connection->seq;
+		hdr->msg_id = message_id;
+
+		/* Copy payload */
+		memcpy(msg + sizeof(*hdr), req_buff_curr, payload_size);
+		req_buff_curr += payload_size;
+
+		/* Force the last fragment to be sent immediately to the receiver */
+		tx_flags = (i == num_fragments) ? GH_MSGQ_TX_PUSH : 0;
+
+		ret = gh_msgq_send(rsc_mgr->msgq_tx, msg, sizeof(*hdr) + payload_size, tx_flags);
+
+		if (ret < 0)
+			break;
+	}
+
+	mutex_unlock(&rsc_mgr->send_lock);
+	return ret < 0 ? ret : 0;
+}
+
+/**
+ * gh_rm_call: Achieve request-response type communication with RPC
+ * @message_id: The RM RPC message-id
+ * @req_buff: Request buffer that contains the payload
+ * @req_buff_size: Total size of the payload
+ * @resp_buf: Pointer to a response buffer
+ * @resp_buff_size: Size of the response buffer
+ * @reply_err_code: Returns Gunyah standard error code for the response
+ *
+ * Make a request to the RM-VM and wait for reply back. For a successful
+ * response, the function returns the payload. The size of the payload is set in resp_buff_size.
+ * The resp_buf should be freed by the caller.
+ *
+ * Context: Process context. Will sleep waiting for reply.
+ * Return: >0 is standard reply error from RM. <0 on internal error.
+ */
+int gh_rm_call(u32 message_id, void *req_buff, size_t req_buff_size,
+		void **resp_buf, size_t *resp_buff_size)
+{
+	struct gh_rm_connection *connection;
+	int ret;
+	struct gh_rsc_mgr *rsc_mgr = __rsc_mgr;
+
+	/* messaged_id 0 is reserved */
+	if (!message_id)
+		return -EINVAL;
+
+	if (!rsc_mgr)
+		return -EPROBE_DEFER;
+
+	connection = gh_rm_alloc_connection(message_id, GH_RM_RPC_TYPE_RPLY);
+	if (!connection)
+		return -ENOMEM;
+
+	init_completion(&connection->seq_done);
+
+	/* Allocate a new seq number for this connection */
+	if (mutex_lock_interruptible(&rsc_mgr->call_idr_lock)) {
+		kfree(connection);
+		return -ERESTARTSYS;
+	}
+	connection->seq = idr_alloc_cyclic(&rsc_mgr->call_idr, connection, 0, U16_MAX, GFP_KERNEL);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	/* Send the request to the Resource Manager */
+	ret = gh_rm_send_request(rsc_mgr, message_id, req_buff, req_buff_size, connection);
+	if (ret < 0)
+		goto out;
+
+	/* Wait for response */
+	wait_for_completion(&connection->seq_done);
+
+	if (connection->ret) {
+		ret = connection->ret;
+		kfree(connection->payload);
+		goto out;
+	}
+
+	if (connection->rm_error) {
+		ret = connection->rm_error;
+		kfree(connection->payload);
+		goto out;
+	}
+
+	*resp_buf = connection->payload;
+	*resp_buff_size = connection->size;
+
+out:
+	mutex_lock(&rsc_mgr->call_idr_lock);
+	idr_remove(&rsc_mgr->call_idr, connection->seq);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	kfree(connection);
+	return ret;
+}
+
+int gh_rm_register_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_register(&gh_rm_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(gh_rm_register_notifier);
+
+int gh_rm_unregister_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_unregister(&gh_rm_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(gh_rm_unregister_notifier);
+
+static struct gunyah_device *gh_msgq_platform_probe_direction(struct platform_device *pdev,
+				u8 gh_type, int idx)
+{
+	int irq, ret;
+	u64 capid;
+	struct device_node *node = pdev->dev.of_node;
+	struct gunyah_device *ghdev;
+
+	irq = platform_get_irq(pdev, idx);
+	if (irq < 0) {
+		dev_err(&pdev->dev, "Failed to get irq%d: %d\n", idx, irq);
+		return ERR_PTR(irq);
+	}
+
+	ret = of_property_read_u64_index(node, "reg", idx, &capid);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to get capid%d: %d\n", idx, ret);
+		return ERR_PTR(ret);
+	}
+
+	ghdev = gunyah_device_alloc(&pdev->dev, capid, gh_type);
+	ghdev->irq = irq;
+	ret = gunyah_device_add(ghdev);
+	if (ret) {
+		kfree(ghdev);
+		return ERR_PTR(ret);
+	}
+
+	return ghdev;
+}
+
+static int gh_rm_drv_probe(struct platform_device *pdev)
+{
+	struct gh_rsc_mgr *rsc_mgr;
+	int ret;
+
+	rsc_mgr = devm_kzalloc(&pdev->dev, sizeof(*rsc_mgr), GFP_KERNEL);
+	if (!rsc_mgr)
+		return -ENOMEM;
+	platform_set_drvdata(pdev, rsc_mgr);
+
+	mutex_init(&rsc_mgr->call_idr_lock);
+	idr_init(&rsc_mgr->call_idr);
+	mutex_init(&rsc_mgr->send_lock);
+
+	rsc_mgr->msgq_tx = gh_msgq_platform_probe_direction(pdev, GUNYAH_DEVICE_TYPE_MSGQ_TX, 0);
+	if (IS_ERR(rsc_mgr->msgq_tx))
+		return PTR_ERR(rsc_mgr->msgq_tx);
+	rsc_mgr->msgq_rx = gh_msgq_platform_probe_direction(pdev, GUNYAH_DEVICE_TYPE_MSGQ_RX, 1);
+	if (IS_ERR(rsc_mgr->msgq_rx)) {
+		ret = PTR_ERR(rsc_mgr->msgq_rx);
+		goto err_msgq_tx;
+	}
+
+	rsc_mgr->recv_task = kthread_run(gh_rm_recv_task_fn, rsc_mgr, "gh_rm_recv_task");
+	if (IS_ERR_OR_NULL(rsc_mgr->recv_task)) {
+		ret = PTR_ERR(rsc_mgr->recv_task);
+		goto err_msgq;
+	}
+
+	__rsc_mgr = rsc_mgr;
+
+	return 0;
+
+err_msgq:
+	gunyah_device_remove(rsc_mgr->msgq_rx);
+err_msgq_tx:
+	gunyah_device_remove(rsc_mgr->msgq_tx);
+	return ret;
+}
+
+static int gh_rm_drv_remove(struct platform_device *pdev)
+{
+	struct gh_rsc_mgr *rsc_mgr = platform_get_drvdata(pdev);
+
+	gunyah_device_remove(rsc_mgr->msgq_tx);
+	gunyah_device_remove(rsc_mgr->msgq_rx);
+
+	return 0;
+}
+
+static const struct of_device_id gh_rm_of_match[] = {
+	{ .compatible = "gunyah-resource-manager" },
+	{ }
+};
+MODULE_DEVICE_TABLE(of, gh_rm_of_match);
+
+static struct platform_driver gh_rsc_mgr_driver = {
+	.probe = gh_rm_drv_probe,
+	.remove = gh_rm_drv_remove,
+	.driver = {
+		.name = "gh_rsc_mgr",
+		.of_match_table = gh_rm_of_match,
+	},
+};
+
+int __init gh_rsc_mgr_init(void)
+{
+	return platform_driver_register(&gh_rsc_mgr_driver);
+}
+
+void gh_rsc_mgr_exit(void)
+{
+	platform_driver_unregister(&gh_rsc_mgr_driver);
+}
diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h
new file mode 100644
index 000000000000..e4f2499267bf
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+#ifndef __GH_RSC_MGR_PRIV_H
+#define __GH_RSC_MGR_PRIV_H
+
+#include <linux/gunyah.h>
+
+/* RM Error codes */
+#define GH_RM_ERROR_OK			0x0
+#define GH_RM_ERROR_UNIMPLEMENTED	0xFFFFFFFF
+#define GH_RM_ERROR_NOMEM		0x1
+#define GH_RM_ERROR_NORESOURCE		0x2
+#define GH_RM_ERROR_DENIED		0x3
+#define GH_RM_ERROR_INVALID		0x4
+#define GH_RM_ERROR_BUSY		0x5
+#define GH_RM_ERROR_ARGUMENT_INVALID	0x6
+#define GH_RM_ERROR_HANDLE_INVALID	0x7
+#define GH_RM_ERROR_VALIDATE_FAILED	0x8
+#define GH_RM_ERROR_MAP_FAILED		0x9
+#define GH_RM_ERROR_MEM_INVALID		0xA
+#define GH_RM_ERROR_MEM_INUSE		0xB
+#define GH_RM_ERROR_MEM_RELEASED	0xC
+#define GH_RM_ERROR_VMID_INVALID	0xD
+#define GH_RM_ERROR_LOOKUP_FAILED	0xE
+#define GH_RM_ERROR_IRQ_INVALID		0xF
+#define GH_RM_ERROR_IRQ_INUSE		0x10
+#define GH_RM_ERROR_IRQ_RELEASED	0x11
+
+int gh_rm_call(u32 message_id, void *req_buff, size_t req_buff_size,
+		void **resp_buf, size_t *resp_buff_size);
+
+#endif
diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
index 7589689e5e92..7c0efc80f85e 100644
--- a/drivers/virt/gunyah/sysfs.c
+++ b/drivers/virt/gunyah/sysfs.c
@@ -148,7 +148,13 @@ static int __init gunyah_init(void)
 	if (ret)
 		goto err_bus;
 
+	ret = gh_rsc_mgr_init();
+	if (ret)
+		goto err_msgq;
+
 	return ret;
+err_msgq:
+	gh_msgq_exit();
 err_bus:
 	gunyah_bus_exit();
 err_sysfs:
@@ -159,6 +165,7 @@ module_init(gunyah_init);
 
 static void __exit gunyah_exit(void)
 {
+	gh_rsc_mgr_exit();
 	gh_msgq_exit();
 	gunyah_bus_exit();
 	gh_sysfs_unregister();
diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h
new file mode 100644
index 000000000000..015bd851e1a3
--- /dev/null
+++ b/include/linux/gunyah_rsc_mgr.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#ifndef _GUNYAH_RSC_MGR_H
+#define _GUNYAH_RSC_MGR_H
+
+#include <linux/list.h>
+#include <linux/notifier.h>
+#include <linux/gunyah.h>
+
+typedef u16 gh_vmid_t;
+typedef u32 gh_virq_handle_t;
+
+#define GH_VMID_INVAL	U16_MAX
+
+/* Gunyah recognizes VMID0 as an alias to the current VM's ID */
+#define GH_VMID_SELF			0
+
+struct gh_rm_notification {
+	const void *buff;
+	const size_t size;
+};
+
+int gh_rm_register_notifier(struct notifier_block *nb);
+int gh_rm_unregister_notifier(struct notifier_block *nb);
+
+#endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 09/11] gunyah: rsc_mgr: Add auxiliary devices for console

[Elliot Berman]

Gunyah resource manager exposes a concrete functionalities which
complicate a single resource manager driver. Use auxiliary bus
to help split high level functions for the resource manager and keep the
primary resource manager driver focused on the RPC with RM itself.
Delegate Resource Manager's console functionality to the auxiliary bus.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 drivers/virt/gunyah/rsc_mgr.c | 61 ++++++++++++++++++++++++++++++++++-
 1 file changed, 60 insertions(+), 1 deletion(-)

diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c
index b8268ee02fab..44b22cef7d44 100644
--- a/drivers/virt/gunyah/rsc_mgr.c
+++ b/drivers/virt/gunyah/rsc_mgr.c
@@ -91,6 +91,11 @@ struct gh_rm_notif_complete {
 	struct work_struct work;
 };
 
+struct gh_rsc_mgr_adev {
+	struct auxiliary_device adev;
+	struct list_head list;
+};
+
 struct gh_rsc_mgr {
 	struct task_struct *recv_task;
 	struct gunyah_device *msgq_tx, *msgq_rx;
@@ -99,6 +104,13 @@ struct gh_rsc_mgr {
 	struct mutex call_idr_lock;
 
 	struct mutex send_lock;
+
+	struct list_head adevs;
+};
+
+/* List of auxiliary devices which resource manager creates */
+static const char * const adev_names[] = {
+	"console",
 };
 
 static struct gh_rsc_mgr *__rsc_mgr;
@@ -516,6 +528,14 @@ int gh_rm_unregister_notifier(struct notifier_block *nb)
 }
 EXPORT_SYMBOL_GPL(gh_rm_unregister_notifier);
 
+static void gh_rm_adev_release(struct device *dev)
+{
+	struct gh_rsc_mgr_adev *rm_adev = container_of(dev, struct gh_rsc_mgr_adev, adev.dev);
+
+	list_del(&rm_adev->list);
+	kfree(rm_adev);
+}
+
 static struct gunyah_device *gh_msgq_platform_probe_direction(struct platform_device *pdev,
 				u8 gh_type, int idx)
 {
@@ -550,7 +570,9 @@ static struct gunyah_device *gh_msgq_platform_probe_direction(struct platform_de
 static int gh_rm_drv_probe(struct platform_device *pdev)
 {
 	struct gh_rsc_mgr *rsc_mgr;
-	int ret;
+	struct gh_rsc_mgr_adev *rm_adev;
+	struct list_head *l, *n;
+	int ret, i;
 
 	rsc_mgr = devm_kzalloc(&pdev->dev, sizeof(*rsc_mgr), GFP_KERNEL);
 	if (!rsc_mgr)
@@ -560,6 +582,7 @@ static int gh_rm_drv_probe(struct platform_device *pdev)
 	mutex_init(&rsc_mgr->call_idr_lock);
 	idr_init(&rsc_mgr->call_idr);
 	mutex_init(&rsc_mgr->send_lock);
+	INIT_LIST_HEAD(&rsc_mgr->adevs);
 
 	rsc_mgr->msgq_tx = gh_msgq_platform_probe_direction(pdev, GUNYAH_DEVICE_TYPE_MSGQ_TX, 0);
 	if (IS_ERR(rsc_mgr->msgq_tx))
@@ -576,10 +599,38 @@ static int gh_rm_drv_probe(struct platform_device *pdev)
 		goto err_msgq;
 	}
 
+	for (i = 0; i < ARRAY_SIZE(adev_names); i++) {
+		rm_adev = kzalloc(sizeof(*rm_adev), GFP_KERNEL);
+
+		rm_adev->adev.dev.parent = &pdev->dev;
+		rm_adev->adev.dev.release = gh_rm_adev_release;
+		rm_adev->adev.name = adev_names[i];
+		ret = auxiliary_device_init(&rm_adev->adev);
+		if (ret) {
+			kfree(rm_adev);
+			goto err_adevs;
+		}
+
+		list_add(&rm_adev->list, &rsc_mgr->adevs);
+
+		ret = auxiliary_device_add(&rm_adev->adev);
+		if (ret) {
+			auxiliary_device_uninit(&rm_adev->adev);
+			goto err_adevs;
+		}
+	}
+
 	__rsc_mgr = rsc_mgr;
 
 	return 0;
 
+err_adevs:
+	list_for_each_safe(l, n, &rsc_mgr->adevs) {
+		rm_adev = container_of(l, struct gh_rsc_mgr_adev, list);
+		auxiliary_device_delete(&rm_adev->adev);
+		auxiliary_device_uninit(&rm_adev->adev);
+	}
+
 err_msgq:
 	gunyah_device_remove(rsc_mgr->msgq_rx);
 err_msgq_tx:
@@ -590,6 +641,14 @@ static int gh_rm_drv_probe(struct platform_device *pdev)
 static int gh_rm_drv_remove(struct platform_device *pdev)
 {
 	struct gh_rsc_mgr *rsc_mgr = platform_get_drvdata(pdev);
+	struct gh_rsc_mgr_adev *rm_adev;
+	struct list_head *l, *n;
+
+	list_for_each_safe(l, n, &rsc_mgr->adevs) {
+		rm_adev = container_of(l, struct gh_rsc_mgr_adev, list);
+		auxiliary_device_delete(&rm_adev->adev);
+		auxiliary_device_uninit(&rm_adev->adev);
+	}
 
 	gunyah_device_remove(rsc_mgr->msgq_tx);
 	gunyah_device_remove(rsc_mgr->msgq_rx);
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 09/11] gunyah: rsc_mgr: Add auxiliary devices for console

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Gunyah resource manager exposes a concrete functionalities which
> complicate a single resource manager driver. Use auxiliary bus
> to help split high level functions for the resource manager and keep the
> primary resource manager driver focused on the RPC with RM itself.
> Delegate Resource Manager's console functionality to the auxiliary bus.
> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   drivers/virt/gunyah/rsc_mgr.c | 61 ++++++++++++++++++++++++++++++++++-
>   1 file changed, 60 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c
> index b8268ee02fab..44b22cef7d44 100644
> --- a/drivers/virt/gunyah/rsc_mgr.c
> +++ b/drivers/virt/gunyah/rsc_mgr.c
> @@ -91,6 +91,11 @@ struct gh_rm_notif_complete {
>   	struct work_struct work;
>   };
>   
> +struct gh_rsc_mgr_adev {
> +	struct auxiliary_device adev;
> +	struct list_head list;
> +};
> +
>   struct gh_rsc_mgr {
>   	struct task_struct *recv_task;
>   	struct gunyah_device *msgq_tx, *msgq_rx;
> @@ -99,6 +104,13 @@ struct gh_rsc_mgr {
>   	struct mutex call_idr_lock;
>   
>   	struct mutex send_lock;
> +
> +	struct list_head adevs;
> +};
> +
> +/* List of auxiliary devices which resource manager creates */
> +static const char * const adev_names[] = {
> +	"console",
>   };

Which other auxilliary devices do you expect at this moment?

>   
>   static struct gh_rsc_mgr *__rsc_mgr;
> @@ -516,6 +528,14 @@ int gh_rm_unregister_notifier(struct notifier_block *nb)
>   }
>   EXPORT_SYMBOL_GPL(gh_rm_unregister_notifier);
>   
> +static void gh_rm_adev_release(struct device *dev)
> +{
> +	struct gh_rsc_mgr_adev *rm_adev = container_of(dev, struct gh_rsc_mgr_adev, adev.dev);
> +
> +	list_del(&rm_adev->list);

is there a race for the rsc_mgr->list? Can multiple release functions be 
called in parallel?
Rather than having a list, it would be easier to have an array of 
devices. Less race conditions, simpler code.

Or just add gh_rsc_msg->console_adev and use it directly without any 
additional bells and whistles.

> +	kfree(rm_adev);
> +}
> +
>   static struct gunyah_device *gh_msgq_platform_probe_direction(struct platform_device *pdev,
>   				u8 gh_type, int idx)
>   {
-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 09/11] gunyah: rsc_mgr: Add auxiliary devices for console

[Elliot Berman]

On 8/2/2022 1:38 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Gunyah resource manager exposes a concrete functionalities which
>> complicate a single resource manager driver. Use auxiliary bus
>> to help split high level functions for the resource manager and keep the
>> primary resource manager driver focused on the RPC with RM itself.
>> Delegate Resource Manager's console functionality to the auxiliary bus.
>>
>> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
>> ---
>>   drivers/virt/gunyah/rsc_mgr.c | 61 ++++++++++++++++++++++++++++++++++-
>>   1 file changed, 60 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/virt/gunyah/rsc_mgr.c 
>> b/drivers/virt/gunyah/rsc_mgr.c
>> index b8268ee02fab..44b22cef7d44 100644
>> --- a/drivers/virt/gunyah/rsc_mgr.c
>> +++ b/drivers/virt/gunyah/rsc_mgr.c
>> @@ -91,6 +91,11 @@ struct gh_rm_notif_complete {
>>       struct work_struct work;
>>   };
>> +struct gh_rsc_mgr_adev {
>> +    struct auxiliary_device adev;
>> +    struct list_head list;
>> +};
>> +
>>   struct gh_rsc_mgr {
>>       struct task_struct *recv_task;
>>       struct gunyah_device *msgq_tx, *msgq_rx;
>> @@ -99,6 +104,13 @@ struct gh_rsc_mgr {
>>       struct mutex call_idr_lock;
>>       struct mutex send_lock;
>> +
>> +    struct list_head adevs;
>> +};
>> +
>> +/* List of auxiliary devices which resource manager creates */
>> +static const char * const adev_names[] = {
>> +    "console",
>>   };
> 
> Which other auxilliary devices do you expect at this moment?
> 

Only foresee a VM loader auxiliary device.

>>   static struct gh_rsc_mgr *__rsc_mgr;
>> @@ -516,6 +528,14 @@ int gh_rm_unregister_notifier(struct 
>> notifier_block *nb)
>>   }
>>   EXPORT_SYMBOL_GPL(gh_rm_unregister_notifier);
>> +static void gh_rm_adev_release(struct device *dev)
>> +{
>> +    struct gh_rsc_mgr_adev *rm_adev = container_of(dev, struct 
>> gh_rsc_mgr_adev, adev.dev);
>> +
>> +    list_del(&rm_adev->list);
> 
> is there a race for the rsc_mgr->list? Can multiple release functions be 
> called in parallel?
> Rather than having a list, it would be easier to have an array of 
> devices. Less race conditions, simpler code.
> 
> Or just add gh_rsc_msg->console_adev and use it directly without any 
> additional bells and whistles.
> 

I like this approach, it is much simpler.

>> +    kfree(rm_adev);
>> +}
>> +
>>   static struct gunyah_device *gh_msgq_platform_probe_direction(struct 
>> platform_device *pdev,
>>                   u8 gh_type, int idx)
>>   {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 10/11] gunyah: rsc_mgr: Add RPC for console services

[Elliot Berman]

Gunyah resource manager defines a simple API for virtual machine log
sharing with the console service. A VM's own log can be opened by using
GH_VMID_SELF. Another VM's log can be accessed via its VMID. Once
opened, characters can be written to the log with a write command.
Characters are received with resource manager notifications (using ID
GH_RM_NOTIF_VM_CONSOLE_CHARS).

These high level rpc calls are kept in
drivers/virt/gunyah/rsc_mgr_rpc.c. Future RPC calls, e.g. to launch a VM
will also be maintained in this file.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 drivers/virt/gunyah/Makefile      |   4 +-
 drivers/virt/gunyah/rsc_mgr.h     |  22 +++++
 drivers/virt/gunyah/rsc_mgr_rpc.c | 151 ++++++++++++++++++++++++++++++
 include/linux/gunyah_rsc_mgr.h    |  16 ++++
 4 files changed, 191 insertions(+), 2 deletions(-)
 create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c

diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index 86655bca8944..b3f15c052297 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 gunyah-y += sysfs.o device.o msgq.o
-gunyah-y += rsc_mgr.o
-obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
+gunyah-y += rsc_mgr.o rsc_mgr_rpc.o
+obj-$(CONFIG_GUNYAH) += gunyah.o
diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h
index e4f2499267bf..c82a2e97ad49 100644
--- a/drivers/virt/gunyah/rsc_mgr.h
+++ b/drivers/virt/gunyah/rsc_mgr.h
@@ -28,6 +28,28 @@
 #define GH_RM_ERROR_IRQ_INUSE		0x10
 #define GH_RM_ERROR_IRQ_RELEASED	0x11
 
+/* Message IDs: VM Management */
+#define GH_RM_RPC_VM_GET_VMID			0x56000024
+
+/* Message IDs: VM Services */
+#define GH_RM_RPC_VM_CONSOLE_OPEN_ID		0x56000081
+#define GH_RM_RPC_VM_CONSOLE_CLOSE_ID		0x56000082
+#define GH_RM_RPC_VM_CONSOLE_WRITE_ID		0x56000083
+#define GH_RM_RPC_VM_CONSOLE_FLUSH_ID		0x56000084
+
+/* Call: CONSOLE_OPEN, CONSOLE_CLOSE, CONSOLE_FLUSH */
+struct gh_vm_console_common_req {
+	gh_vmid_t vmid;
+	u16 reserved0;
+} __packed;
+
+/* Call: CONSOLE_WRITE */
+struct gh_vm_console_write_req {
+	gh_vmid_t vmid;
+	u16 num_bytes;
+	u8 data[0];
+} __packed;
+
 int gh_rm_call(u32 message_id, void *req_buff, size_t req_buff_size,
 		void **resp_buf, size_t *resp_buff_size);
 
diff --git a/drivers/virt/gunyah/rsc_mgr_rpc.c b/drivers/virt/gunyah/rsc_mgr_rpc.c
new file mode 100644
index 000000000000..be9317968537
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr_rpc.c
@@ -0,0 +1,151 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "gh_rsc_mgr: " fmt
+
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/printk.h>
+#include <linux/gunyah_rsc_mgr.h>
+
+#include "rsc_mgr.h"
+
+/**
+ * gh_rm_get_vmid: Retrieve VMID of this virtual machine
+ * @vmid: Filled with the VMID of this VM
+ */
+int gh_rm_get_vmid(gh_vmid_t *vmid)
+{
+	void *resp;
+	size_t resp_size;
+	int ret;
+	int payload = 0;
+
+	ret = gh_rm_call(GH_RM_RPC_VM_GET_VMID, &payload, sizeof(payload), &resp, &resp_size);
+	if (ret)
+		return ret;
+
+	if (resp_size != sizeof(*vmid))
+		return -EIO;
+	*vmid = *(gh_vmid_t *)resp;
+	kfree(resp);
+
+	return ret;
+}
+
+/**
+ * gh_rm_console_open: Open a console with a VM
+ * @vmid: VMID of the other vmid whose console to open. If VMID is GH_VMID_SELF, the
+ *        console associated with this VM is opened.
+ */
+int gh_rm_console_open(gh_vmid_t vmid)
+{
+	void *resp;
+	struct gh_vm_console_common_req req_payload = {0};
+	size_t resp_size;
+	int ret;
+
+	req_payload.vmid = vmid;
+
+	ret = gh_rm_call(GH_RM_RPC_VM_CONSOLE_OPEN_ID,
+			  &req_payload, sizeof(req_payload),
+			  &resp, &resp_size);
+	kfree(resp);
+
+	if (!ret && resp_size)
+		pr_warn("Received unexpected payload for CONSOLE_OPEN: %lu\n", resp_size);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_rm_console_open);
+
+/**
+ * gh_rm_console_close: Close a console with a VM
+ * @vmid: The vmid of the vm whose console to close.
+ */
+int gh_rm_console_close(gh_vmid_t vmid)
+{
+	void *resp;
+	struct gh_vm_console_common_req req_payload = {0};
+	size_t resp_size;
+	int ret;
+
+	req_payload.vmid = vmid;
+
+	ret = gh_rm_call(GH_RM_RPC_VM_CONSOLE_CLOSE_ID,
+			  &req_payload, sizeof(req_payload),
+			  &resp, &resp_size);
+	kfree(resp);
+
+	if (!ret && resp_size)
+		pr_warn("Received unexpected payload for CONSOLE_CLOSE: %lu\n", resp_size);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_rm_console_close);
+
+/**
+ * gh_rm_console_write: Write to a VM's console
+ * @vmid: The vmid of the vm whose console to write to.
+ * @buf: Buffer to write to the VM's console
+ * @size: Size of the buffer
+ */
+int gh_rm_console_write(gh_vmid_t vmid, const char *buf, size_t size)
+{
+	void *resp;
+	struct gh_vm_console_write_req *req_payload;
+	size_t resp_size;
+	int ret = 0;
+	size_t req_payload_size = sizeof(*req_payload) + size;
+
+	if (size < 1 || size > (U32_MAX - sizeof(*req_payload)))
+		return -EINVAL;
+
+	req_payload = kzalloc(req_payload_size, GFP_KERNEL);
+
+	if (!req_payload)
+		return -ENOMEM;
+
+	req_payload->vmid = vmid;
+	req_payload->num_bytes = size;
+	memcpy(req_payload->data, buf, size);
+
+	ret = gh_rm_call(GH_RM_RPC_VM_CONSOLE_WRITE_ID,
+		   req_payload, req_payload_size,
+		   &resp, &resp_size);
+	kfree(req_payload);
+	kfree(resp);
+
+	if (!ret && resp_size)
+		pr_warn("Received unexpected payload for CONSOLE_WRITE: %lu\n", resp_size);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_rm_console_write);
+
+/**
+ * gh_rm_console_flush: Flush a console with a VM
+ * @vmid: The vmid of the vm whose console to flush
+ */
+int gh_rm_console_flush(gh_vmid_t vmid)
+{
+	void *resp;
+	struct gh_vm_console_common_req req_payload = {0};
+	size_t resp_size;
+	int ret;
+
+	req_payload.vmid = vmid;
+
+	ret = gh_rm_call(GH_RM_RPC_VM_CONSOLE_FLUSH_ID,
+			  &req_payload, sizeof(req_payload),
+			  &resp, &resp_size);
+	kfree(resp);
+
+	if (!ret && resp_size)
+		pr_warn("Received unexpected payload for CONSOLE_FLUSH: %lu\n", resp_size);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(gh_rm_console_flush);
diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h
index 015bd851e1a3..4211ac9a219c 100644
--- a/include/linux/gunyah_rsc_mgr.h
+++ b/include/linux/gunyah_rsc_mgr.h
@@ -26,4 +26,20 @@ struct gh_rm_notification {
 int gh_rm_register_notifier(struct notifier_block *nb);
 int gh_rm_unregister_notifier(struct notifier_block *nb);
 
+/* Notification type Message IDs */
+#define GH_RM_NOTIF_VM_CONSOLE_CHARS	0x56100080
+
+struct gh_rm_notif_vm_console_chars {
+	gh_vmid_t vmid;
+	u16 num_bytes;
+	u8 bytes[0];
+} __packed;
+
+/* RPC Calls */
+int gh_rm_get_vmid(gh_vmid_t *vmid);
+int gh_rm_console_open(gh_vmid_t vmid);
+int gh_rm_console_close(gh_vmid_t vmid);
+int gh_rm_console_write(gh_vmid_t vmid, const char *buf, size_t size);
+int gh_rm_console_flush(gh_vmid_t vmid);
+
 #endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 11/11] gunyah: Add tty console driver for RM Console Serivces

[Elliot Berman]

Gunyah provides a console for each VM using the VM console resource
manager APIs. This driver allows console data from other
VMs to be accessed via a TTY device and exports a console device to dump
Linux's own logs to our console.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 Documentation/virt/gunyah/index.rst   |   7 +
 drivers/virt/gunyah/Kconfig           |  10 +
 drivers/virt/gunyah/Makefile          |   3 +
 drivers/virt/gunyah/rsc_mgr_console.c | 405 ++++++++++++++++++++++++++
 4 files changed, 425 insertions(+)
 create mode 100644 drivers/virt/gunyah/rsc_mgr_console.c

diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst
index e7bb2b14543e..95ba9b71ab30 100644
--- a/Documentation/virt/gunyah/index.rst
+++ b/Documentation/virt/gunyah/index.rst
@@ -90,3 +90,10 @@ When booting a virtual machine which uses a devicetree, resource manager overlay
 how to communicate with resource manager, and basic description and capabilities of
 this VM. See Documentation/devicetree/bindings/gunyah/qcom,hypervisor.yml for a description
 of this node.
+
+Resource Manager Consoles
+-------------------------
+RM provides infrastructure for virtual machines to share an interactive console. This can be used to
+interact with a VM which may not have access to a serial port. Linux will register a printk console:
+ttyGH0. That console and other VM's consoles can be accessed via ttyGHX.
+/sys/class/tty/ttyGHX/vmid will print the VM which is associated with that TTY.
diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig
index 0adb6efd4848..ba7bd058e959 100644
--- a/drivers/virt/gunyah/Kconfig
+++ b/drivers/virt/gunyah/Kconfig
@@ -12,3 +12,13 @@ config GUNYAH
 
 	  Say Y here to enable the drivers needed to interact in a Gunyah
 	  virtual environment.
+
+if GUNYAH
+config GUNYAH_RESOURCE_MANAGER_CONSOLE
+	tristate "Gunyah Resource Manager Consoles"
+	depends on TTY
+	help
+	  This enables support for console output using Gunyah's Resource Manager RPC.
+	  This is normally used when a secondary VM which does not have exclusive access
+	  to a real serial device and virtio-console is unavailable.
+endif
diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index b3f15c052297..001cf1630c03 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -3,3 +3,6 @@
 gunyah-y += sysfs.o device.o msgq.o
 gunyah-y += rsc_mgr.o rsc_mgr_rpc.o
 obj-$(CONFIG_GUNYAH) += gunyah.o
+
+gunyah_console-y += rsc_mgr_console.o
+obj-$(CONFIG_GUNYAH_RESOURCE_MANAGER_CONSOLE) += gunyah_console.o
diff --git a/drivers/virt/gunyah/rsc_mgr_console.c b/drivers/virt/gunyah/rsc_mgr_console.c
new file mode 100644
index 000000000000..eda25100314f
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr_console.c
@@ -0,0 +1,405 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "gh_rsc_mgr_console: " fmt
+
+#include <linux/gunyah_rsc_mgr.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/workqueue.h>
+#include <linux/spinlock.h>
+#include <linux/tty_flip.h>
+#include <linux/console.h>
+#include <linux/module.h>
+#include <linux/kfifo.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <linux/tty.h>
+#include <linux/of.h>
+
+/*
+ * The Linux TTY code does not support dynamic addition of tty derived devices so we need to know
+ * how many tty devices we might need when space is allocated for the tty device. Since VMs might be
+ * added/removed dynamically, we need to make sure we have enough allocated.
+ */
+#define RSC_MGR_TTY_ADAPTERS		16
+
+/* # of payload bytes that can fit in a 1-fragment CONSOLE_WRITE message */
+#define RM_CONS_WRITE_MSG_SIZE	((1 * (GH_MSGQ_MAX_MSG_SIZE - 8)) - 4)
+
+struct rm_cons_port {
+	struct tty_port port;
+	gh_vmid_t vmid;
+	bool open;
+	unsigned int index;
+
+	DECLARE_KFIFO(put_fifo, char, 1024);
+	spinlock_t fifo_lock;
+	struct work_struct put_work;
+
+	struct rm_cons_data *cons_data;
+};
+
+struct rm_cons_data {
+	struct tty_driver *tty_driver;
+	struct device *dev;
+
+	spinlock_t ports_lock;
+	struct rm_cons_port *ports[RSC_MGR_TTY_ADAPTERS];
+
+	struct notifier_block rsc_mgr_notif;
+	struct console console;
+};
+
+static void put_work_fn(struct work_struct *ws)
+{
+	char buf[RM_CONS_WRITE_MSG_SIZE];
+	int count, ret;
+	struct rm_cons_port *port = container_of(ws, struct rm_cons_port, put_work);
+
+	while (!kfifo_is_empty(&port->put_fifo)) {
+		count = kfifo_out_spinlocked(&port->put_fifo, buf, sizeof(buf), &port->fifo_lock);
+		if (count <= 0)
+			continue;
+
+		ret = gh_rm_console_write(port->vmid, buf, count);
+		if (ret) {
+			pr_warn_once("failed to send characters: %d\n", ret);
+			break;
+		}
+	}
+}
+
+static int rsc_mgr_console_notif(struct notifier_block *nb, unsigned long cmd, void *data)
+{
+	int count, i;
+	struct rm_cons_port *rm_port;
+	struct tty_port *tty_port = NULL;
+	struct rm_cons_data *cons_data = container_of(nb, struct rm_cons_data, rsc_mgr_notif);
+	const struct gh_rm_notification *notif = data;
+	struct gh_rm_notif_vm_console_chars const * const msg = notif->buff;
+
+	if (cmd != GH_RM_NOTIF_VM_CONSOLE_CHARS ||
+		notif->size < sizeof(*msg))
+		return NOTIFY_DONE;
+
+	spin_lock(&cons_data->ports_lock);
+	for (i = 0; i < RSC_MGR_TTY_ADAPTERS; i++) {
+		if (!cons_data->ports[i])
+			continue;
+		if (cons_data->ports[i]->vmid == msg->vmid) {
+			rm_port = cons_data->ports[i];
+			break;
+		}
+	}
+	if (rm_port)
+		tty_port = tty_port_get(&rm_port->port);
+	spin_unlock(&cons_data->ports_lock);
+
+	if (!rm_port)
+		pr_warn("Received unexpected console characters for VMID %u\n", msg->vmid);
+	if (!tty_port)
+		return NOTIFY_DONE;
+
+	count = tty_buffer_request_room(tty_port, msg->num_bytes);
+	tty_insert_flip_string(tty_port, msg->bytes, count);
+	tty_flip_buffer_push(tty_port);
+
+	tty_port_put(tty_port);
+	return NOTIFY_OK;
+}
+
+static ssize_t vmid_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct rm_cons_port *rm_port = dev_get_drvdata(dev);
+
+	if (rm_port->vmid == GH_VMID_SELF)
+		return sysfs_emit(buf, "self\n");
+
+	return sysfs_emit(buf, "%u\n", rm_port->vmid);
+}
+
+static DEVICE_ATTR_RO(vmid);
+
+static struct attribute *rsc_mgr_tty_dev_attrs[] = {
+	&dev_attr_vmid.attr,
+	NULL
+};
+
+static const struct attribute_group rsc_mgr_tty_dev_attr_group = {
+	.attrs = rsc_mgr_tty_dev_attrs,
+};
+
+static const struct attribute_group *rsc_mgr_tty_dev_attr_groups[] = {
+	&rsc_mgr_tty_dev_attr_group,
+	NULL
+};
+
+static int rsc_mgr_tty_open(struct tty_struct *tty, struct file *filp)
+{
+	int ret;
+	struct rm_cons_port *rm_port = dev_get_drvdata(tty->dev);
+
+	if (!rm_port->open) {
+		ret = gh_rm_console_open(rm_port->vmid);
+		if (ret) {
+			pr_err("Failed to open RM console for vmid %x: %d\n", rm_port->vmid, ret);
+			return ret;
+		}
+		rm_port->open = true;
+	}
+
+	return tty_port_open(&rm_port->port, tty, filp);
+}
+
+static void rsc_mgr_tty_close(struct tty_struct *tty, struct file *filp)
+{
+	int ret;
+	struct rm_cons_port *rm_port = dev_get_drvdata(tty->dev);
+
+	if (rm_port->open) {
+		if (rm_port->vmid != GH_VMID_SELF) {
+			ret = gh_rm_console_close(rm_port->vmid);
+			if (ret)
+				pr_warn("Failed to close RM console for vmid %d: %d\n",
+					rm_port->vmid, ret);
+		}
+		rm_port->open = false;
+
+		tty_port_close(&rm_port->port, tty, filp);
+	}
+
+}
+
+static int rsc_mgr_tty_write(struct tty_struct *tty, const unsigned char *buf, int count)
+{
+	struct rm_cons_port *rm_port = dev_get_drvdata(tty->dev);
+	int ret;
+
+	ret = kfifo_in_spinlocked(&rm_port->put_fifo, buf, count, &rm_port->fifo_lock);
+	if (ret > 0)
+		schedule_work(&rm_port->put_work);
+
+	return ret;
+}
+
+static unsigned int rsc_mgr_mgr_tty_write_room(struct tty_struct *tty)
+{
+	struct rm_cons_port *rm_port = dev_get_drvdata(tty->dev);
+
+	return kfifo_avail(&rm_port->put_fifo);
+}
+
+static void rsc_mgr_console_write(struct console *co, const char *buf, unsigned count)
+{
+	struct rm_cons_port *rm_port = co->data;
+	int ret;
+
+	ret = kfifo_in_spinlocked(&rm_port->put_fifo, buf, count, &rm_port->fifo_lock);
+	if (ret > 0)
+		schedule_work(&rm_port->put_work);
+}
+
+static struct tty_driver *rsc_mgr_console_device(struct console *co, int *index)
+{
+	struct rm_cons_port *rm_port = co->data;
+
+	*index = rm_port->index;
+	return rm_port->port.tty->driver;
+}
+
+static int rsc_mgr_console_setup(struct console *co, char *unused)
+{
+	int ret;
+	struct rm_cons_port *rm_port = co->data;
+
+	if (!rm_port->open) {
+		ret = gh_rm_console_open(rm_port->vmid);
+		if (ret) {
+			pr_err("Failed to open RM console for vmid %x: %d\n", rm_port->vmid, ret);
+			return ret;
+		}
+		rm_port->open = true;
+	}
+
+	return 0;
+}
+
+static int rsc_mgr_console_exit(struct console *co)
+{
+	int ret;
+	struct rm_cons_port *rm_port = co->data;
+
+	if (rm_port->open) {
+		ret = gh_rm_console_close(rm_port->vmid);
+		if (ret) {
+			pr_err("Failed to close RM console for vmid %x: %d\n", rm_port->vmid, ret);
+			return ret;
+		}
+		rm_port->open = false;
+	}
+
+	return 0;
+}
+
+static const struct tty_operations rsc_mgr_tty_ops = {
+	.open = rsc_mgr_tty_open,
+	.close = rsc_mgr_tty_close,
+	.write = rsc_mgr_tty_write,
+	.write_room = rsc_mgr_mgr_tty_write_room,
+};
+
+static void rsc_mgr_port_destruct(struct tty_port *port)
+{
+	struct rm_cons_port *rm_port = container_of(port, struct rm_cons_port, port);
+	struct rm_cons_data *cons_data = rm_port->cons_data;
+
+	spin_lock(&cons_data->ports_lock);
+	WARN_ON(cons_data->ports[rm_port->index] != rm_port);
+	cons_data->ports[rm_port->index] = NULL;
+	spin_unlock(&cons_data->ports_lock);
+	kfree(rm_port);
+}
+
+static const struct tty_port_operations rsc_mgr_port_ops = {
+	.destruct = rsc_mgr_port_destruct,
+};
+
+static struct rm_cons_port *rsc_mgr_port_create(struct rm_cons_data *cons_data, gh_vmid_t vmid)
+{
+	struct rm_cons_port *rm_port;
+	struct device *ttydev;
+	unsigned int index;
+	int ret;
+
+	rm_port = kzalloc(sizeof(*rm_port), GFP_KERNEL);
+	rm_port->vmid = vmid;
+	INIT_KFIFO(rm_port->put_fifo);
+	spin_lock_init(&rm_port->fifo_lock);
+	INIT_WORK(&rm_port->put_work, put_work_fn);
+	tty_port_init(&rm_port->port);
+	rm_port->port.ops = &rsc_mgr_port_ops;
+
+	spin_lock(&cons_data->ports_lock);
+	for (index = 0; index < RSC_MGR_TTY_ADAPTERS; index++) {
+		if (!cons_data->ports[index]) {
+			cons_data->ports[index] = rm_port;
+			rm_port->index = index;
+			break;
+		}
+	}
+	spin_unlock(&cons_data->ports_lock);
+	if (index >= RSC_MGR_TTY_ADAPTERS) {
+		ret = -ENOSPC;
+		goto err_put_port;
+	}
+
+	ttydev = tty_port_register_device_attr(&rm_port->port, cons_data->tty_driver, index,
+					      cons_data->dev, rm_port, rsc_mgr_tty_dev_attr_groups);
+	if (IS_ERR(ttydev)) {
+		ret = PTR_ERR(ttydev);
+		goto err_put_port;
+	}
+
+	return rm_port;
+err_put_port:
+	tty_port_put(&rm_port->port);
+	return ERR_PTR(ret);
+}
+
+static int rsc_mgr_console_probe(struct auxiliary_device *auxdev,
+	const struct auxiliary_device_id *aux_dev_id)
+{
+	struct rm_cons_data *cons_data;
+	struct rm_cons_port *rm_port;
+	int ret;
+	gh_vmid_t vmid;
+
+	cons_data = devm_kzalloc(&auxdev->dev, sizeof(*cons_data), GFP_KERNEL);
+	if (!cons_data)
+		return -ENOMEM;
+	dev_set_drvdata(&auxdev->dev, cons_data);
+	cons_data->dev = &auxdev->dev;
+
+	cons_data->tty_driver = tty_alloc_driver(RSC_MGR_TTY_ADAPTERS,
+						 TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV);
+	if (IS_ERR(cons_data->tty_driver))
+		return PTR_ERR(cons_data->tty_driver);
+
+	cons_data->tty_driver->driver_name = "gh";
+	cons_data->tty_driver->name = "ttyGH";
+	cons_data->tty_driver->type = TTY_DRIVER_TYPE_SYSTEM;
+	cons_data->tty_driver->init_termios = tty_std_termios;
+	tty_set_operations(cons_data->tty_driver, &rsc_mgr_tty_ops);
+
+	ret = tty_register_driver(cons_data->tty_driver);
+	if (ret) {
+		dev_err(&auxdev->dev, "Could not register tty driver: %d\n", ret);
+		goto err_put_tty;
+	}
+
+	spin_lock_init(&cons_data->ports_lock);
+
+	cons_data->rsc_mgr_notif.notifier_call = rsc_mgr_console_notif;
+	ret = gh_rm_register_notifier(&cons_data->rsc_mgr_notif);
+	if (ret) {
+		dev_err(&auxdev->dev, "Could not register for resource manager notifications: %d\n",
+			ret);
+		goto err_put_tty;
+	}
+
+	rm_port = rsc_mgr_port_create(cons_data, GH_VMID_SELF);
+	if (IS_ERR(rm_port)) {
+		ret = PTR_ERR(rm_port);
+		dev_err(&auxdev->dev, "Could not create own console: %d\n", ret);
+		goto err_unreg_notif;
+	}
+
+	strncpy(cons_data->console.name, "ttyGH", sizeof(cons_data->console.name));
+	cons_data->console.write = rsc_mgr_console_write;
+	cons_data->console.device = rsc_mgr_console_device;
+	cons_data->console.setup = rsc_mgr_console_setup;
+	cons_data->console.exit = rsc_mgr_console_exit;
+	cons_data->console.index = rm_port->index;
+	cons_data->console.data = rm_port;
+	register_console(&cons_data->console);
+
+	ret = gh_rm_get_vmid(&vmid);
+	if (!ret)
+		rsc_mgr_port_create(cons_data, vmid);
+	else
+		pr_warn("Failed to get this VM's VMID: %d. Not creating loop-back console\n", ret);
+
+	return 0;
+err_unreg_notif:
+	gh_rm_unregister_notifier(&cons_data->rsc_mgr_notif);
+err_put_tty:
+	tty_driver_kref_put(cons_data->tty_driver);
+	return ret;
+}
+
+static void rsc_mgr_console_remove(struct auxiliary_device *auxdev)
+{
+	struct rm_cons_data *cons_data = dev_get_drvdata(&auxdev->dev);
+
+	unregister_console(&cons_data->console);
+	gh_rm_unregister_notifier(&cons_data->rsc_mgr_notif);
+	tty_driver_kref_put(cons_data->tty_driver);
+}
+
+static struct auxiliary_device_id rsc_mgr_console_ids[] = {
+	{ .name = "gunyah.console" },
+	{}
+};
+MODULE_DEVICE_TABLE(auxiliary, rsc_mgr_console_ids);
+
+static struct auxiliary_driver rsc_mgr_console_drv = {
+	.probe = rsc_mgr_console_probe,
+	.remove = rsc_mgr_console_remove,
+	.id_table = rsc_mgr_console_ids,
+};
+module_auxiliary_driver(rsc_mgr_console_drv);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Gunyah Console");
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 11/11] gunyah: Add tty console driver for RM Console Serivces

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Gunyah provides a console for each VM using the VM console resource
> manager APIs. This driver allows console data from other
> VMs to be accessed via a TTY device and exports a console device to dump
> Linux's own logs to our console.

Is such 'console' an unidirectional stream of chars or bidirection one? 
IOW, can I have getty on such console?

> 
> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
> ---
>   Documentation/virt/gunyah/index.rst   |   7 +
>   drivers/virt/gunyah/Kconfig           |  10 +
>   drivers/virt/gunyah/Makefile          |   3 +
>   drivers/virt/gunyah/rsc_mgr_console.c | 405 ++++++++++++++++++++++++++
>   4 files changed, 425 insertions(+)
>   create mode 100644 drivers/virt/gunyah/rsc_mgr_console.c
>

-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 11/11] gunyah: Add tty console driver for RM Console Serivces

[Elliot Berman]

On 8/2/2022 1:31 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Gunyah provides a console for each VM using the VM console resource
>> manager APIs. This driver allows console data from other
>> VMs to be accessed via a TTY device and exports a console device to dump
>> Linux's own logs to our console.
> 
> Is such 'console' an unidirectional stream of chars or bidirection one? 
> IOW, can I have getty on such console?
> 

Consoles are bidrectional.

>>
>> Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
>> ---
>>   Documentation/virt/gunyah/index.rst   |   7 +
>>   drivers/virt/gunyah/Kconfig           |  10 +
>>   drivers/virt/gunyah/Makefile          |   3 +
>>   drivers/virt/gunyah/rsc_mgr_console.c | 405 ++++++++++++++++++++++++++
>>   4 files changed, 425 insertions(+)
>>   create mode 100644 drivers/virt/gunyah/rsc_mgr_console.c
>>
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Jeffrey Hugo]

On Mon, Aug 1, 2022 at 3:16 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>
> Gunyah is a Type-1 hypervisor independent of any
> high-level OS kernel, and runs in a higher CPU privilege level. It does
> not depend on any lower-privileged OS kernel/code for its core
> functionality. This increases its security and can support a much smaller
> trusted computing base than a Type-2 hypervisor.
>
> Gunyah is an open source hypervisor. The source repo is available at
> https://github.com/quic/gunyah-hypervisor.

Nowhere in this series do I see a change log, yet this is marked as
v2.  How is anyone supposed to identify what is the difference between
v1 and v2?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

Hi Jeffrey,

On 8/1/2022 2:27 PM, Jeffrey Hugo wrote:
> On Mon, Aug 1, 2022 at 3:16 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>
>> Gunyah is a Type-1 hypervisor independent of any
>> high-level OS kernel, and runs in a higher CPU privilege level. It does
>> not depend on any lower-privileged OS kernel/code for its core
>> functionality. This increases its security and can support a much smaller
>> trusted computing base than a Type-2 hypervisor.
>>
>> Gunyah is an open source hypervisor. The source repo is available at
>> https://github.com/quic/gunyah-hypervisor.
> 
> Nowhere in this series do I see a change log, yet this is marked as
> v2.  How is anyone supposed to identify what is the difference between
> v1 and v2?
I dropped the message when copying cover letter:

Changes in v2:
  - DT bindings clean up
  - Switch hypercalls to follow SMCCC

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Dmitry Baryshkov]

On 02/08/2022 00:12, Elliot Berman wrote:
> Gunyah is a Type-1 hypervisor independent of any
> high-level OS kernel, and runs in a higher CPU privilege level. It does
> not depend on any lower-privileged OS kernel/code for its core
> functionality. This increases its security and can support a much smaller
> trusted computing base than a Type-2 hypervisor.
> 
> Gunyah is an open source hypervisor. The source repo is available at
> https://github.com/quic/gunyah-hypervisor.
> 
> The diagram below shows the architecture.
> 
> ::
> 
>          Primary VM           Secondary VMs

Is there any significant difference between Primary VM and other VMs?

>       +-----+ +-----+  | +-----+ +-----+ +-----+
>       |     | |     |  | |     | |     | |     |
>   EL0 | APP | | APP |  | | APP | | APP | | APP |
>       |     | |     |  | |     | |     | |     |
>       +-----+ +-----+  | +-----+ +-----+ +-----+
>   ---------------------|-------------------------
>       +--------------+ | +----------------------+
>       |              | | |                      |
>   EL1 | Linux Kernel | | |Linux kernel/Other OS |   ...
>       |              | | |                      |
>       +--------------+ | +----------------------+
>   --------hvc/smc------|------hvc/smc------------
>       +----------------------------------------+
>       |                                        |
>   EL2 |            Gunyah Hypervisor           |
>       |                                        |
>       +----------------------------------------+
> 
> Gunyah provides these following features.
> 
> - Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on
> physical CPUs and enables time-sharing of the CPUs.

Is the scheduling provided behind the back of the OS or does it require 
cooperation?

> - Memory Management: Gunyah tracks memory ownership and use of all memory
> under its control. Memory partitioning between VMs is a fundamental
> security feature.
> - Interrupt Virtualization: All interrupts are handled in the hypervisor
> and routed to the assigned VM.
> - Inter-VM Communication: There are several different mechanisms provided
> for communicating between VMs.
> - Device Virtualization: Para-virtualization of devices is supported using
> inter-VM communication. Low level system features and devices such as
> interrupt controllers are supported with emulation where required.

After reviewing some of the patches from the series, I'd like to 
understand, what does it provide (and can be provided) to the VMs.

I'd like to understand it first, before going deep into the API issues.

1) The hypervisor provides message queues, doorbells and vCPUs

Each of resources has it's own capability ID.
Why is it called capability? Is it just a misname for the resource ID, 
or has it any other meaning behind? If it is a capability, who is 
capable of what?

At this moment you create allocate two message queues with fixed IDs for 
communication with resource manager. Then you use these message queues 
to organize a console and a pack of tty devices.

What other kinds of services does RM provide to the guest OS?
Do you expect any other drivers to be calling into the RM?

What is the usecase for the doorbells? Who provides doorbells?

You mentioned that the RM generates DT overlays. What kind of 
information goes to the overlay?

My current impression of this series is that you have misused the 
concept of devices. Rather than exporting MSGQs and BELLs as 
gunyah_devices and then using them from other drivers, I'd suggest 
turning them into resources provided by the gunyah driver core. I 
mentioned using the mailbox API for this. Another subsystem that might 
ring the bell for you is the remoteproc, especially the rproc_subdev.

I might be completely wrong about this, but if my in-mind picture of 
Gunyah is correct, I'd have implemented the gunyah core subsytem as 
mailbox provider, RM as a separate platform driver consuming these 
mailboxes and in turn being a remoteproc driver, and consoles as 
remoteproc subdevices.

I can assume that at some point you would like to use Gunyah to boot 
secondary VMs from the primary VM by calling into RM, etc.
Most probably at this moment a VM would be allocated other bells, 
message queues, etc. If this assumption is correct, them the VM can 
become a separate device (remoteproc?) in the Linux device tree.

I might be wrong in any of the assumptions above. Please feel free to 
correct me. We can then think about a better API for your usecase.

-- 
With best wishes
Dmitry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

On 8/2/2022 2:24 AM, Dmitry Baryshkov wrote:
> On 02/08/2022 00:12, Elliot Berman wrote:
>> Gunyah is a Type-1 hypervisor independent of any
>> high-level OS kernel, and runs in a higher CPU privilege level. It does
>> not depend on any lower-privileged OS kernel/code for its core
>> functionality. This increases its security and can support a much smaller
>> trusted computing base than a Type-2 hypervisor.
>>
>> Gunyah is an open source hypervisor. The source repo is available at
>> https://github.com/quic/gunyah-hypervisor.
>>
>> The diagram below shows the architecture.
>>
>> ::
>>
>>          Primary VM           Secondary VMs
> 
> Is there any significant difference between Primary VM and other VMs?
> 

The primary VM is started by RM. Secondary VMs are not otherwise special 
except that they are (usually) launched by the primary VM.

>>       +-----+ +-----+  | +-----+ +-----+ +-----+
>>       |     | |     |  | |     | |     | |     |
>>   EL0 | APP | | APP |  | | APP | | APP | | APP |
>>       |     | |     |  | |     | |     | |     |
>>       +-----+ +-----+  | +-----+ +-----+ +-----+
>>   ---------------------|-------------------------
>>       +--------------+ | +----------------------+
>>       |              | | |                      |
>>   EL1 | Linux Kernel | | |Linux kernel/Other OS |   ...
>>       |              | | |                      |
>>       +--------------+ | +----------------------+
>>   --------hvc/smc------|------hvc/smc------------
>>       +----------------------------------------+
>>       |                                        |
>>   EL2 |            Gunyah Hypervisor           |
>>       |                                        |
>>       +----------------------------------------+
>>
>> Gunyah provides these following features.
>>
>> - Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on
>> physical CPUs and enables time-sharing of the CPUs.
> 
> Is the scheduling provided behind the back of the OS or does it require 
> cooperation?
> 

Gunyah supports both of these scheduling models. For instance, 
scheduling of resource manager and the primary VM are done by Gunyah 
itself. A VM that the primary VM launches could be scheduled by the 
primary VM itself (by making a hypercall requesting a vCPU be switched 
in), or by Gunyah itself. We've been calling the former "proxy 
scheduling" and this would be the default behavior of VMs.

>> - Memory Management: Gunyah tracks memory ownership and use of all memory
>> under its control. Memory partitioning between VMs is a fundamental
>> security feature.
>> - Interrupt Virtualization: All interrupts are handled in the hypervisor
>> and routed to the assigned VM.
>> - Inter-VM Communication: There are several different mechanisms provided
>> for communicating between VMs.
>> - Device Virtualization: Para-virtualization of devices is supported 
>> using
>> inter-VM communication. Low level system features and devices such as
>> interrupt controllers are supported with emulation where required.
> 
> After reviewing some of the patches from the series, I'd like to 
> understand, what does it provide (and can be provided) to the VMs.
> > I'd like to understand it first, before going deep into the API issues.
> 
> 1) The hypervisor provides message queues, doorbells and vCPUs
> 
> Each of resources has it's own capability ID.
> Why is it called capability? Is it just a misname for the resource ID, 
> or has it any other meaning behind? If it is a capability, who is 
> capable of what?
> 

We are following Gunyah's naming convention here. For each virtual 
machine, Gunyah maintains a table of resources which can be accessed by 
that VM. An entry in this table is called a "capability" and VMs can 
only access resources via this capability table. Hence, they get called 
"capability IDs" and not "resource IDs". A VM can have multiple 
capability IDs mapping to the same resource. If 2 VMs have access to the 
same resource, they may not be using the same capability ID to access 
that resource since the tables are independent per VM.

> At this moment you create allocate two message queues with fixed IDs for 
> communication with resource manager. Then you use these message queues 
> to organize a console and a pack of tty devices.
> 
> What other kinds of services does RM provide to the guest OS?
> Do you expect any other drivers to be calling into the RM?
> 

I want to establish the framework to build a VM loader for Gunyah. 
Internally, we are working with a prototype of a "generic VM loader" 
which works with crosvm [1]. In this generic VM loader, memory sharing, 
memory lending, cooperative scheduling, and raising virtual interrupts 
are all supported. Emulating virtio devices in userspace is supported in 
a way which feels very similar to KVM. Our internal VM loader uses an 
IOCTL interface which is similar to KVM's.

> What is the usecase for the doorbells? Who provides doorbells? >

The basic use case I'll start with is for userspace to create an IRQFD. 
Userspace can use the IRQFD to raise a doorbell (interrupt) on the other VM.

> You mentioned that the RM generates DT overlays. What kind of 
> information goes to the overlay?
> 

The info is described in 
Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml.

> My current impression of this series is that you have misused the 
> concept of devices. Rather than exporting MSGQs and BELLs as 
> gunyah_devices and then using them from other drivers, I'd suggest 
> turning them into resources provided by the gunyah driver core. I 
> mentioned using the mailbox API for this. Another subsystem that might 
> ring the bell for you is the remoteproc, especially the rproc_subdev. >

I had an offline discussion with Bjorn and he agreed with this approach 
here. He suggested avoiding using the device bus model and will go with 
smaller approach in v3.

> I might be completely wrong about this, but if my in-mind picture of 
> Gunyah is correct, I'd have implemented the gunyah core subsytem as 
> mailbox provider, RM as a separate platform driver consuming these 
> mailboxes and in turn being a remoteproc driver, and consoles as 
> remoteproc subdevices. >

The mailbox framework can only fit with message queues and not doorbells 
or vCPUs. The mailbox framework also relies on the mailbox being defined 
in the devicetree. RM is an exceptional case in that it is described in 
the devicetree. Message queues for other VMs would be dynamically 
created at runtime as/when that VM is created. Thus, the client of the 
message queue would need to "own" both the controller and client ends of 
the mailbox.

RM is not loaded or managed by Linux, so I don't think remoteproc 
framework provides us any code re-use except for the subdevices code. 
Remoteproc is much larger framework than just the subdevices code, so I 
don't think it fits well overall.

> I can assume that at some point you would like to use Gunyah to boot 
> secondary VMs from the primary VM by calling into RM, etc.
> Most probably at this moment a VM would be allocated other bells, 
> message queues, etc. If this assumption is correct, them the VM can 
> become a separate device (remoteproc?) in the Linux device tree.
> 
> I might be wrong in any of the assumptions above. Please feel free to 
> correct me. We can then think about a better API for your usecase.
> 

We don't want to limit VM configuration to the devicetree as this limits 
the number and kinds of VMs that can be launched to build time. I'm not 
sure if you might have seen an early presentation of Gunyah at Linaro? 
In the early days of Gunyah, we had static configuration of VMs and many 
properties of the VMs were described in the devicetree. We are moving 
away from static configuration of VMs as much as possible.

[1]: https://chromium.googlesource.com/chromiumos/platform/crosvm


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Robin Murphy]

[drive-by observation since one thing caught my interest...]

On 2022-08-09 00:38, Elliot Berman wrote:
>> I might be completely wrong about this, but if my in-mind picture of 
>> Gunyah is correct, I'd have implemented the gunyah core subsytem as 
>> mailbox provider, RM as a separate platform driver consuming these 
>> mailboxes and in turn being a remoteproc driver, and consoles as 
>> remoteproc subdevices. >
> 
> The mailbox framework can only fit with message queues and not doorbells 
> or vCPUs.

Is that so? There was a whole long drawn-out saga around the SCMI 
protocol using the Arm MHU mailbox as a set of doorbells for 
shared-memory payloads, but it did eventually get merged as the separate 
arm_mhu_db.c driver, so unless we're talking about some completely 
different notion of "doorbell"... :/

> The mailbox framework also relies on the mailbox being defined 
> in the devicetree. RM is an exceptional case in that it is described in 
> the devicetree. Message queues for other VMs would be dynamically 
> created at runtime as/when that VM is created. Thus, the client of the 
> message queue would need to "own" both the controller and client ends of 
> the mailbox.

FWIW, if the mailbox API does fit conceptually then it looks like it 
shouldn't be *too* hard to better abstract the DT details in the 
framework itself and allow providers to offer additional means to 
validate channel requests, which might be more productive than inventing 
a whole new thing.

Thanks,
Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

On 8/9/2022 6:13 AM, Robin Murphy wrote:
> [drive-by observation since one thing caught my interest...] >

Appreciate all the comments.

Jassi,

I understood you have talked with some of our folks (Trilok and Carl) a
few years ago about using the mailbox APIs. We were steered away from
using mailboxes then. Is that still the recommendation today?

> On 2022-08-09 00:38, Elliot Berman wrote:
>>> I might be completely wrong about this, but if my in-mind picture of 
>>> Gunyah is correct, I'd have implemented the gunyah core subsytem as 
>>> mailbox provider, RM as a separate platform driver consuming these 
>>> mailboxes and in turn being a remoteproc driver, and consoles as 
>>> remoteproc subdevices. >
>>
>> The mailbox framework can only fit with message queues and not 
>> doorbells or vCPUs.
> 
> Is that so? There was a whole long drawn-out saga around the SCMI 
> protocol using the Arm MHU mailbox as a set of doorbells for 
> shared-memory payloads, but it did eventually get merged as the separate 
> arm_mhu_db.c driver, so unless we're talking about some completely 
> different notion of "doorbell"... :/
> 

Doorbells will be harder to fit into mailbox API framework.

  - Simple doorbells don't have any TX done acknowledgement model at
    the doorbell layer (see bullet 1 from 
https://lore.kernel.org/all/68e241fd-16f0-96b4-eab8-369628292e03@quicinc.com/).
    Doorbell clients might have a doorbell acknowledgement flow, but the
    only client I have for doorbells doesn't. IRQFDs would send an
    empty message to the mailbox and immediately do a client-triggered
    TX_DONE.

  - Using mailboxes for the more advanced use-case doorbell forces client
    to use doorbells a certain way because each channel could be a bit on
    the bitmask, or the client could have complete control of the entire
    bitmask. I think implementing the mailbox API would force the
    otherwise-generic doorbell code to make that decision for clients.

Further, I wanted to highlight one other challenge with fitting Gunyah
message queues into mailbox API:

  - Message queues track a flag which indicates whether there is space
    available in the queue. The flag is returned on msgq_send. When the
    message queue is full, an interrupt is raised when there is more
    space available. This could be used as a TX_DONE indicator, but
    mailbox framework's API prevents us from doing mbox_chan_txdone
    inside the send_data channel op.

I think this might be solvable by adding a new txdone mechanism.

>> The mailbox framework also relies on the mailbox being defined in the 
>> devicetree. RM is an exceptional case in that it is described in the 
>> devicetree. Message queues for other VMs would be dynamically created 
>> at runtime as/when that VM is created. Thus, the client of the message 
>> queue would need to "own" both the controller and client ends of the 
>> mailbox.
> 
> FWIW, if the mailbox API does fit conceptually then it looks like it 
> shouldn't be *too* hard to better abstract the DT details in the 
> framework itself and allow providers to offer additional means to 
> validate channel requests, which might be more productive than inventing 
> a whole new thing. >
Some notes about fitting mailboxes into Gunyah IPC:

  - A single mailbox controller can't cover all the gunyah devices. The
    number of gunyah devices is not fixed and varies per VM launched.
    Mailbox controller would need to be per-VM or per-device, where each
    channel represents a capability.

  - The other device types (like vCPU) don't fit into message-based
    style framework. I'd like to have a consistent way of binding a
    device's function with the device. If we use mailbox API, some
    devices will use mailbox and others will use some other mechanism.
    I'd prefer to consistently use "some other mechanism" throughout.

  - TX and RX message queues are independent and "combining" a TX and RX
    message queue happens at client layer by the client requesting access
    to two otherwise unassociated message queues. A mailbox channel would
    either be associated with a TX message queue capability or an RX
    message queue capability. This isn't a major hurdle per se, but it
    decreases how cleanly we can use the mailbox APIs IMO.
      - A VM might only have a TX message queue and no RX message queue,
        or vice versa. We won't be able to require coupling a TX and RX
        message queue for the mailbox.

  - TX done acknowledgement doesn't fit Gunyah IPC (see above) and a new
    TX_DONE mode would need to be implemented.

  - Need to make it possible for a client to binding a mailbox channel
    without DT.

I'm getting a bit apprehensive about the tweaks needed to make mailbox
framework usable for Gunyah. Will there be enough code re-use and help
with abstracting the direct-to-Gunyah APIs? IMO, there isn't, but
opinions are welcome :)

Thanks,
Elliot

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Jassi Brar]

On Tue, Aug 9, 2022 at 7:07 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>
> On 8/9/2022 6:13 AM, Robin Murphy wrote:
> > [drive-by observation since one thing caught my interest...] >
>
> Appreciate all the comments.
>
> Jassi,
>
> I understood you have talked with some of our folks (Trilok and Carl) a
> few years ago about using the mailbox APIs. We were steered away from
> using mailboxes then. Is that still the recommendation today?
>
Neither I nor Google remember any such conversation.

Doorbell had always been supported by the api. It was the
doorbell-mode of _mhu_ controller that had some contention.

I haven't read the complete history of Gunyah yet, but from a quick
look it uses the hvc/smc instruction as the "physical link" between
entities (?).    zynqmp-ipi-mailbox.c is one driver that uses smc in
such a manner. And I know there are some platforms that don't call
hvc/smc under mailbox api and I don't blame them.

Let me educate myself with the background and get back.... unless you
want to summarize a usecase that you doubt is supported.

Thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

On 8/9/2022 9:12 PM, Jassi Brar wrote:
> On Tue, Aug 9, 2022 at 7:07 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
> 
> I haven't read the complete history of Gunyah yet, but from a quick
> look it uses the hvc/smc instruction as the "physical link" between
> entities (?).    zynqmp-ipi-mailbox.c is one driver that uses smc in
> such a manner. And I know there are some platforms that don't call
> hvc/smc under mailbox api and I don't blame them.
> 
> Let me educate myself with the background and get back.... unless you
> want to summarize a usecase that you doubt is supported.
> 

Hi Jassi,

Did you have chance to evaluate? I have given a summary in this mail, 
especially the last paragraph:

https://lore.kernel.org/all/36303c20-5d30-2edd-0863-0cad804e3f8f@quicinc.com/


Thanks,
Elliot

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Dmitry Baryshkov]

On 09/08/2022 02:38, Elliot Berman wrote:
> 
> 
> On 8/2/2022 2:24 AM, Dmitry Baryshkov wrote:
>> I might be completely wrong about this, but if my in-mind picture of 
>> Gunyah is correct, I'd have implemented the gunyah core subsytem as 
>> mailbox provider, RM as a separate platform driver consuming these 
>> mailboxes and in turn being a remoteproc driver, and consoles as 
>> remoteproc subdevices. >
> 
> The mailbox framework can only fit with message queues and not doorbells 
> or vCPUs. The mailbox framework also relies on the mailbox being defined 
> in the devicetree. RM is an exceptional case in that it is described in 
> the devicetree. Message queues for other VMs would be dynamically 
> created at runtime as/when that VM is created. Thus, the client of the 
> message queue would need to "own" both the controller and client ends of 
> the mailbox.

I'd still suggest using the mailbox API for the doorbells. You do not 
have to implement the txdone, if I'm not mistaken.

> 
> RM is not loaded or managed by Linux, so I don't think remoteproc 
> framework provides us any code re-use except for the subdevices code. 
> Remoteproc is much larger framework than just the subdevices code, so I 
> don't think it fits well overall.
> 
>> I can assume that at some point you would like to use Gunyah to boot 
>> secondary VMs from the primary VM by calling into RM, etc.
>> Most probably at this moment a VM would be allocated other bells, 
>> message queues, etc. If this assumption is correct, them the VM can 
>> become a separate device (remoteproc?) in the Linux device tree.
>>
>> I might be wrong in any of the assumptions above. Please feel free to 
>> correct me. We can then think about a better API for your usecase.
>>
> 
> We don't want to limit VM configuration to the devicetree as this limits 
> the number and kinds of VMs that can be launched to build time. I'm not 
> sure if you might have seen an early presentation of Gunyah at Linaro? 
> In the early days of Gunyah, we had static configuration of VMs and many 
> properties of the VMs were described in the devicetree. We are moving 
> away from static configuration of VMs as much as possible.

ack, this is correct.

> 
> [1]: https://chromium.googlesource.com/chromiumos/platform/crosvm
> 

-- 
With best wishes
Dmitry


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Bagas Sanjaya]

On Mon, Aug 01, 2022 at 02:12:29PM -0700, Elliot Berman wrote:
> Gunyah is a Type-1 hypervisor independent of any
> high-level OS kernel, and runs in a higher CPU privilege level. It does
> not depend on any lower-privileged OS kernel/code for its core
> functionality. This increases its security and can support a much smaller
> trusted computing base than a Type-2 hypervisor.
> 
> Gunyah is an open source hypervisor. The source repo is available at
> https://github.com/quic/gunyah-hypervisor.
> 
> The diagram below shows the architecture.
> 
> ::
> 
>         Primary VM           Secondary VMs
>      +-----+ +-----+  | +-----+ +-----+ +-----+
>      |     | |     |  | |     | |     | |     |
>  EL0 | APP | | APP |  | | APP | | APP | | APP |
>      |     | |     |  | |     | |     | |     |
>      +-----+ +-----+  | +-----+ +-----+ +-----+
>  ---------------------|-------------------------
>      +--------------+ | +----------------------+
>      |              | | |                      |
>  EL1 | Linux Kernel | | |Linux kernel/Other OS |   ...
>      |              | | |                      |
>      +--------------+ | +----------------------+
>  --------hvc/smc------|------hvc/smc------------
>      +----------------------------------------+
>      |                                        |
>  EL2 |            Gunyah Hypervisor           |
>      |                                        |
>      +----------------------------------------+
> 
> Gunyah provides these following features.
> 
> - Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on
> physical CPUs and enables time-sharing of the CPUs.
> - Memory Management: Gunyah tracks memory ownership and use of all memory
> under its control. Memory partitioning between VMs is a fundamental
> security feature.
> - Interrupt Virtualization: All interrupts are handled in the hypervisor
> and routed to the assigned VM.
> - Inter-VM Communication: There are several different mechanisms provided
> for communicating between VMs.
> - Device Virtualization: Para-virtualization of devices is supported using
> inter-VM communication. Low level system features and devices such as
> interrupt controllers are supported with emulation where required.
> 

Hi,

I can't apply this series on top of mainline or linux-next. On what tree
(and what commit) this series is based on? I'd like to do htmldocs test.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Elliot Berman]

On 8/4/2022 1:26 AM, Bagas Sanjaya wrote:
> On Mon, Aug 01, 2022 at 02:12:29PM -0700, Elliot Berman wrote:
>> Gunyah is a Type-1 hypervisor independent of any
>> high-level OS kernel, and runs in a higher CPU privilege level. It does
>> not depend on any lower-privileged OS kernel/code for its core
>> functionality. This increases its security and can support a much smaller
>> trusted computing base than a Type-2 hypervisor.
>>
>> Gunyah is an open source hypervisor. The source repo is available at
>> https://github.com/quic/gunyah-hypervisor.
>>
>> The diagram below shows the architecture.
>>
>> ::
>>
>>          Primary VM           Secondary VMs
>>       +-----+ +-----+  | +-----+ +-----+ +-----+
>>       |     | |     |  | |     | |     | |     |
>>   EL0 | APP | | APP |  | | APP | | APP | | APP |
>>       |     | |     |  | |     | |     | |     |
>>       +-----+ +-----+  | +-----+ +-----+ +-----+
>>   ---------------------|-------------------------
>>       +--------------+ | +----------------------+
>>       |              | | |                      |
>>   EL1 | Linux Kernel | | |Linux kernel/Other OS |   ...
>>       |              | | |                      |
>>       +--------------+ | +----------------------+
>>   --------hvc/smc------|------hvc/smc------------
>>       +----------------------------------------+
>>       |                                        |
>>   EL2 |            Gunyah Hypervisor           |
>>       |                                        |
>>       +----------------------------------------+
>>
>> Gunyah provides these following features.
>>
>> - Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on
>> physical CPUs and enables time-sharing of the CPUs.
>> - Memory Management: Gunyah tracks memory ownership and use of all memory
>> under its control. Memory partitioning between VMs is a fundamental
>> security feature.
>> - Interrupt Virtualization: All interrupts are handled in the hypervisor
>> and routed to the assigned VM.
>> - Inter-VM Communication: There are several different mechanisms provided
>> for communicating between VMs.
>> - Device Virtualization: Para-virtualization of devices is supported using
>> inter-VM communication. Low level system features and devices such as
>> interrupt controllers are supported with emulation where required.
>>
> 
> Hi,
> 
> I can't apply this series on top of mainline or linux-next. On what tree
> (and what commit) this series is based on? I'd like to do htmldocs test.
> 

The series should apply cleanly on commit 4a57a8400075 ("vf/remap: 
return the amount of bytes actually deduplicated") from Linus's tree.

> Thanks.
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Bagas Sanjaya]

On Thu, Aug 04, 2022 at 02:48:58PM -0700, Elliot Berman wrote:
> > 
> > Hi,
> > 
> > I can't apply this series on top of mainline or linux-next. On what tree
> > (and what commit) this series is based on? I'd like to do htmldocs test.
> > 
> 
> The series should apply cleanly on commit 4a57a8400075 ("vf/remap: return
> the amount of bytes actually deduplicated") from Linus's tree.
> 

Applied, thanks.

Next time, don't forget to specify --base when using git-format-patch.

-- 
An old man doll... just what I always wanted! - Clara

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2 00/11] Drivers for gunyah hypervisor

[Marc Zyngier]

On Fri, 05 Aug 2022 03:15:24 +0100,
Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> 
> On Thu, Aug 04, 2022 at 02:48:58PM -0700, Elliot Berman wrote:
> > > 
> > > Hi,
> > > 
> > > I can't apply this series on top of mainline or linux-next. On what tree
> > > (and what commit) this series is based on? I'd like to do htmldocs test.
> > > 
> > 
> > The series should apply cleanly on commit 4a57a8400075 ("vf/remap: return
> > the amount of bytes actually deduplicated") from Linus's tree.
> > 
> 
> Applied, thanks.
> 
> Next time, don't forget to specify --base when using git-format-patch.

Or even better, use a tagged release as the base (an early -rc would
do), and not some random commit.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 00/11] Gunyah Hypervisor drivers

[Elliot Berman]

Gunyah is a Type-1 hypervisor independent of any
high-level OS kernel, and runs in a higher CPU privilege level. It does
not depend on any lower-privileged OS kernel/code for its core
functionality. This increases its security and can support a much smaller
trusted computing base than Type-2 hypervisors. This series adds the initial
support for Gunyah hypercalls, IPC via message queues, communication with the
Gunyah Resource Manager to enable Gunyah's paravirtualized console.

Gunyah is an open source hypervisor. The source repo is available at
https://github.com/quic/gunyah-hypervisor.

This series enables guest awareness of Gunyah and establishes the basic
architecture of Gunyah capabilities in the Linux kernel. As an end-to-end
use case, a TTY driver for the Gunyah-based console is added which can demonstrate
communication with the resource manager. In a future series, we intend to add
support for loading secondary VMs.

To self-test the console driver without other VMs, reads and writes to
/dev/ttyGH0 will be sent to /dev/ttyGH1 via the hypervisor. In a system with
multiple VMs, a secondary VM could use ttyGH0 as its boot console.
The primary VM would be able to access that secondary VM's console via a ttyGHx.

Changes in v2:
 - DT bindings clean up
 - Switch hypercalls to follow SMCCC

Elliot Berman (11):
  docs: gunyah: Introduce Gunyah Hypervisor
  dt-bindings: Add binding for gunyah hypervisor
  arm64: gunyah: Add Gunyah hypercalls ABI
  gunyah: Common types and error codes for Gunyah hypercalls
  virt: gunyah: Add sysfs nodes
  virt: gunyah: Add capabilities bus and devices
  gunyah: msgq: Add Gunyah message queues
  gunyah: rsc_mgr: Add resource manager RPC core
  gunyah: rsc_mgr: Add auxiliary devices for console
  gunyah: rsc_mgr: Add RPC for console services
  gunyah: Add tty console driver for RM Console Serivces

 .../ABI/testing/sysfs-hypervisor-gunyah       |  37 +
 .../bindings/firmware/gunyah-hypervisor.yaml  |  84 +++
 Documentation/virt/gunyah/index.rst           |  99 +++
 Documentation/virt/gunyah/message-queue.rst   |  52 ++
 Documentation/virt/index.rst                  |   1 +
 MAINTAINERS                                   |  12 +
 arch/arm64/include/asm/gunyah.h               | 142 ++++
 drivers/virt/Kconfig                          |   1 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/gunyah/Kconfig                   |  24 +
 drivers/virt/gunyah/Makefile                  |   8 +
 drivers/virt/gunyah/device.c                  | 108 +++
 drivers/virt/gunyah/gunyah_private.h          |  18 +
 drivers/virt/gunyah/msgq.c                    | 223 ++++++
 drivers/virt/gunyah/rsc_mgr.c                 | 682 ++++++++++++++++++
 drivers/virt/gunyah/rsc_mgr.h                 |  56 ++
 drivers/virt/gunyah/rsc_mgr_console.c         | 405 +++++++++++
 drivers/virt/gunyah/rsc_mgr_rpc.c             | 151 ++++
 drivers/virt/gunyah/sysfs.c                   | 176 +++++
 include/linux/gunyah.h                        | 133 ++++
 include/linux/gunyah_rsc_mgr.h                |  45 ++
 21 files changed, 2458 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-hypervisor-gunyah
 create mode 100644 Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 create mode 100644 Documentation/virt/gunyah/index.rst
 create mode 100644 Documentation/virt/gunyah/message-queue.rst
 create mode 100644 arch/arm64/include/asm/gunyah.h
 create mode 100644 drivers/virt/gunyah/Kconfig
 create mode 100644 drivers/virt/gunyah/Makefile
 create mode 100644 drivers/virt/gunyah/device.c
 create mode 100644 drivers/virt/gunyah/gunyah_private.h
 create mode 100644 drivers/virt/gunyah/msgq.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.h
 create mode 100644 drivers/virt/gunyah/rsc_mgr_console.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr_rpc.c
 create mode 100644 drivers/virt/gunyah/sysfs.c
 create mode 100644 include/linux/gunyah.h
 create mode 100644 include/linux/gunyah_rsc_mgr.h

-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 08/11] gunyah: rsc_mgr: Add resource manager RPC core

[Elliot Berman]

The resource manager is a special virtual machine which is always
running on a Gunyah system. It provides APIs for creating and destroying
VMs, secure memory management, sharing/lending of memory between VMs,
and setup of inter-VM communication. Calls to the resource manager are
made via message queues.

This patch implements the basic probing and RPC mechanism to make those
API calls. Request/response calls can be made with gh_rm_call.
Drivers can also register to notifications pushed by RM via
gh_rm_register_notifier

Specific API calls that resource manager supports will be implemented in
subsequent patches.

Signed-off-by: Elliot Berman <quic_eberman@quicinc.com>
---
 MAINTAINERS                          |   2 +-
 drivers/virt/gunyah/Kconfig          |   1 +
 drivers/virt/gunyah/Makefile         |   1 +
 drivers/virt/gunyah/gunyah_private.h |   3 +
 drivers/virt/gunyah/rsc_mgr.c        | 623 +++++++++++++++++++++++++++
 drivers/virt/gunyah/rsc_mgr.h        |  34 ++
 drivers/virt/gunyah/sysfs.c          |   7 +
 include/linux/gunyah_rsc_mgr.h       |  29 ++
 8 files changed, 699 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/gunyah/rsc_mgr.c
 create mode 100644 drivers/virt/gunyah/rsc_mgr.h
 create mode 100644 include/linux/gunyah_rsc_mgr.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 5d46c1f91b92..4bbec2ae4a6c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8745,7 +8745,7 @@ F:	Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml
 F:	Documentation/virt/gunyah/
 F:	arch/arm64/include/asm/gunyah.h
 F:	drivers/virt/gunyah/
-F:	include/linux/gunyah.h
+F:	include/linux/gunyah*.h
 
 HABANALABS PCI DRIVER
 M:	Oded Gabbay <ogabbay@kernel.org>
diff --git a/drivers/virt/gunyah/Kconfig b/drivers/virt/gunyah/Kconfig
index e88289963518..2ef4887e280d 100644
--- a/drivers/virt/gunyah/Kconfig
+++ b/drivers/virt/gunyah/Kconfig
@@ -4,6 +4,7 @@ config GUNYAH
 	tristate "Gunyah Virtualization drivers"
 	depends on ARM64
 	select SYS_HYPERVISOR
+	select AUXILIARY_BUS
 	help
 	  The Gunyah drivers are the helper interfaces that runs in a guest VM
 	  such as basic inter-VM IPC and signaling mechanism,s and higher level
diff --git a/drivers/virt/gunyah/Makefile b/drivers/virt/gunyah/Makefile
index 94dc8e738911..86655bca8944 100644
--- a/drivers/virt/gunyah/Makefile
+++ b/drivers/virt/gunyah/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 gunyah-y += sysfs.o device.o msgq.o
+gunyah-y += rsc_mgr.o
 obj-$(CONFIG_GUNYAH) += gunyah.o
\ No newline at end of file
diff --git a/drivers/virt/gunyah/gunyah_private.h b/drivers/virt/gunyah/gunyah_private.h
index 2ade32bd9bdf..6483ffa8c15d 100644
--- a/drivers/virt/gunyah/gunyah_private.h
+++ b/drivers/virt/gunyah/gunyah_private.h
@@ -12,4 +12,7 @@ void gunyah_bus_exit(void);
 int __init gh_msgq_init(void);
 void gh_msgq_exit(void);
 
+int __init gh_rsc_mgr_init(void);
+void gh_rsc_mgr_exit(void);
+
 #endif
diff --git a/drivers/virt/gunyah/rsc_mgr.c b/drivers/virt/gunyah/rsc_mgr.c
new file mode 100644
index 000000000000..b8268ee02fab
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr.c
@@ -0,0 +1,623 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#define pr_fmt(fmt) "gh_rsc_mgr: " fmt
+
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/gunyah.h>
+#include <linux/module.h>
+#include <linux/of_irq.h>
+#include <linux/kthread.h>
+#include <linux/notifier.h>
+#include <linux/irqdomain.h>
+#include <linux/workqueue.h>
+#include <linux/completion.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/gunyah_rsc_mgr.h>
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+
+#include "gunyah_private.h"
+#include "rsc_mgr.h"
+
+/* Resource Manager Header */
+struct gh_rm_rpc_hdr {
+	u8 version:4,
+		hdr_words:4;
+	u8 type:2,
+		fragments:6;
+	u16 seq;
+	u32 msg_id;
+} __packed;
+
+/* Standard reply header */
+struct gh_rm_rpc_reply_hdr {
+	struct gh_rm_rpc_hdr rpc_hdr;
+	u32 err_code;
+} __packed;
+
+/* RPC Header versions */
+#define GH_RM_RPC_HDR_VERSION_ONE	0x1
+
+/* RPC Header words */
+#define GH_RM_RPC_HDR_WORDS		0x2
+
+/* RPC Message types */
+#define GH_RM_RPC_TYPE_CONT		0x0
+#define GH_RM_RPC_TYPE_REQ		0x1
+#define GH_RM_RPC_TYPE_RPLY		0x2
+#define GH_RM_RPC_TYPE_NOTIF		0x3
+
+#define GH_RM_MAX_NUM_FRAGMENTS		62
+
+#define GH_RM_MAX_MSG_SIZE	(GH_MSGQ_MAX_MSG_SIZE - sizeof(struct gh_rm_rpc_hdr))
+
+/**
+ * struct gh_rm_connection - Represents a complete message from resource manager
+ * @payload: Combined payload of all the fragments (i.e. msg headers stripped off).
+ * @size: Size of the payload.
+ * @ret: Linux return code, set in case there was an error processing the connection.
+ * @msg_id: Message ID from the header.
+ * @type: GH_RM_RPC_TYPE_RPLY or GH_RM_RPC_TYPE_NOTIF.
+ * @num_fragments: total number of fragments expected to be received for this connection.
+ * @fragments_recieved: fragments received so far.
+ * @rm_error: For request/reply sequences with standard replies.
+ * @seq: Sequence ID for the main message.
+ */
+struct gh_rm_connection {
+	void *payload;
+	size_t size;
+	int ret;
+	u32 msg_id;
+	u8 type;
+
+	u8 num_fragments;
+	u8 fragments_received;
+
+	/* only for req/reply sequence */
+	u32 rm_error;
+	u16 seq;
+	struct completion seq_done;
+};
+
+struct gh_rm_notif_complete {
+	struct gh_rm_connection *conn;
+	struct work_struct work;
+};
+
+struct gh_rsc_mgr {
+	struct task_struct *recv_task;
+	struct gunyah_device *msgq_tx, *msgq_rx;
+
+	struct idr call_idr;
+	struct mutex call_idr_lock;
+
+	struct mutex send_lock;
+};
+
+static struct gh_rsc_mgr *__rsc_mgr;
+SRCU_NOTIFIER_HEAD_STATIC(gh_rm_notifier);
+
+static struct gh_rm_connection *gh_rm_alloc_connection(u32 msg_id, u8 type)
+{
+	struct gh_rm_connection *connection;
+
+	connection = kzalloc(sizeof(*connection), GFP_KERNEL);
+	if (!connection)
+		return NULL;
+
+	connection->type = type;
+	connection->msg_id = msg_id;
+
+	return connection;
+}
+
+/**
+ * gh_rm_init_connection_payload() - Fills the first message for a connection.
+ */
+static int gh_rm_init_connection_payload(struct gh_rm_connection *connection, void *msg,
+					size_t hdr_size, size_t payload_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	size_t max_buf_size;
+
+	connection->num_fragments = hdr->fragments;
+	connection->fragments_received = 0;
+	connection->type = hdr->type;
+
+	/* There's not going to be any payload, no need to allocate buffer. */
+	if (!payload_size && !connection->num_fragments)
+		return 0;
+
+	/*
+	 * maximum payload size is GH_MSGQ_MAX_MSG_SIZE - hdr_size
+	 * and can received (hdr->fragments + 1) of those
+	 */
+	max_buf_size = (GH_MSGQ_MAX_MSG_SIZE - hdr_size) * (hdr->fragments + 1);
+
+	connection->payload = kzalloc(max_buf_size, GFP_KERNEL);
+	if (!connection->payload)
+		return -ENOMEM;
+
+	memcpy(connection->payload, msg + hdr_size, payload_size);
+	connection->size = payload_size;
+	return 0;
+}
+
+static void gh_rm_notif_work(struct work_struct *work)
+{
+	struct gh_rm_notif_complete *notif = container_of(work, struct gh_rm_notif_complete, work);
+	struct gh_rm_connection *connection = notif->conn;
+	u32 notif_id = connection->msg_id;
+	struct gh_rm_notification notification = {
+		.buff = connection->payload,
+		.size = connection->size,
+	};
+
+	srcu_notifier_call_chain(&gh_rm_notifier, notif_id, &notification);
+
+	kfree(connection->payload);
+	kfree(connection);
+	kfree(notif);
+}
+
+static struct gh_rm_connection *gh_rm_process_notif(struct gh_rsc_mgr *rsc_mgr,
+						    void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	struct gh_rm_connection *connection;
+
+	connection = gh_rm_alloc_connection(hdr->msg_id, hdr->type);
+	if (!connection) {
+		pr_err("Failed to alloc connection for notification, dropping.\n");
+		return NULL;
+	}
+
+	if (gh_rm_init_connection_payload(connection, msg, sizeof(*hdr), msg_size - sizeof(*hdr))) {
+		pr_err("Failed to alloc connection buffer for notification, dropping.\n");
+		kfree(connection);
+		return NULL;
+	}
+
+	return connection;
+}
+
+static struct gh_rm_connection *gh_rm_process_rply(struct gh_rsc_mgr *rsc_mgr,
+						   void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_reply_hdr *reply_hdr = msg;
+	struct gh_rm_rpc_hdr *hdr = msg;
+	struct gh_rm_connection *connection;
+
+	if (mutex_lock_interruptible(&rsc_mgr->call_idr_lock))
+		return ERR_PTR(-ERESTARTSYS);
+
+	connection = idr_find(&rsc_mgr->call_idr, hdr->seq);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	if (!connection) {
+		pr_err("Failed to find connection for sequence %u\n", hdr->seq);
+		return NULL;
+	}
+	if (connection->msg_id != hdr->msg_id) {
+		pr_err("Reply for sequence %u expected msg_id: %x but got %x\n", hdr->seq,
+			connection->msg_id, hdr->msg_id);
+		/*
+		 * Don't complete connection and error the client, maybe resource manager will
+		 * send us the expected reply sequence soon.
+		 */
+		return NULL;
+	}
+
+	if (gh_rm_init_connection_payload(connection, msg, sizeof(*reply_hdr),
+					msg_size - sizeof(*reply_hdr))) {
+		pr_err("Failed to alloc connection buffer for sequence %d\n", hdr->seq);
+		/* Send connection complete and error the client. */
+		connection->ret = -ENOMEM;
+		complete(&connection->seq_done);
+		return NULL;
+	}
+
+	connection->rm_error = reply_hdr->err_code;
+	return connection;
+}
+
+static void gh_rm_process_cont(struct gh_rm_connection *connection, void *msg, size_t msg_size)
+{
+	struct gh_rm_rpc_hdr *hdr = msg;
+	size_t payload_size = msg_size - sizeof(*hdr);
+
+	/*
+	 * hdr->fragments and hdr->msg_id preserves the value from first reply or notif message.
+	 * For sake of sanity, check if it's still intact.
+	 */
+	if (connection->msg_id != hdr->msg_id)
+		pr_warn("Appending mismatched continuation with id %d to connection with id %d\n",
+			hdr->msg_id, connection->msg_id);
+	if (connection->num_fragments != hdr->fragments)
+		pr_warn("Number of fragments mismatch for seq: %d\n", hdr->seq);
+
+	memcpy(connection->payload + connection->size, msg + sizeof(*hdr), payload_size);
+	connection->size += payload_size;
+	connection->fragments_received++;
+}
+
+static bool gh_rm_complete_connection(struct gh_rm_connection *connection)
+{
+	struct gh_rm_notif_complete *notif_work;
+
+	if (!connection)
+		return false;
+
+	if (connection->fragments_received != connection->num_fragments)
+		return false;
+
+	switch (connection->type) {
+	case GH_RM_RPC_TYPE_RPLY:
+		complete(&connection->seq_done);
+		break;
+	case GH_RM_RPC_TYPE_NOTIF:
+		notif_work = kzalloc(sizeof(*notif_work), GFP_KERNEL);
+		if (notif_work == NULL)
+			break;
+
+		notif_work->conn = connection;
+		INIT_WORK(&notif_work->work, gh_rm_notif_work);
+
+		schedule_work(&notif_work->work);
+		break;
+	default:
+		pr_err("Invalid message type (%d) received\n", connection->type);
+		break;
+	}
+
+	return true;
+}
+
+static void gh_rm_abort_connection(struct gh_rm_connection *connection)
+{
+	switch (connection->type) {
+	case GH_RM_RPC_TYPE_RPLY:
+		connection->ret = -EIO;
+		complete(&connection->seq_done);
+		break;
+	case GH_RM_RPC_TYPE_NOTIF:
+		fallthrough;
+	default:
+		kfree(connection->payload);
+		kfree(connection);
+	}
+}
+
+static int gh_rm_recv_task_fn(void *data)
+{
+	struct gh_rsc_mgr *rsc_mgr = data;
+	struct gh_rm_connection *connection = NULL;
+	struct gh_rm_rpc_hdr *hdr = NULL;
+	ssize_t msg_size;
+	void *msg;
+
+	msg = kzalloc(GH_MSGQ_MAX_MSG_SIZE, GFP_KERNEL);
+	if (!msg)
+		return -ENOMEM;
+
+	while (!kthread_should_stop()) {
+		/* Block until a new message is received */
+		msg_size = gh_msgq_recv(rsc_mgr->msgq_rx, msg, GH_MSGQ_MAX_MSG_SIZE, 0);
+		if (msg_size < 0) {
+			pr_err("Failed to receive the message: %ld\n", msg_size);
+			continue;
+		} else if (msg_size <= sizeof(struct gh_rm_rpc_hdr)) {
+			pr_err("Invalid message size received: %ld is too small\n", msg_size);
+			continue;
+		}
+
+		hdr = msg;
+		switch (hdr->type) {
+		case GH_RM_RPC_TYPE_NOTIF:
+			if (connection) {
+				/* Not possible per protocol. Do something better than BUG_ON */
+				pr_warn("Received start of new notification without finishing existing message series.\n");
+				gh_rm_abort_connection(connection);
+			}
+			connection = gh_rm_process_notif(rsc_mgr, msg, msg_size);
+			break;
+		case GH_RM_RPC_TYPE_RPLY:
+			if (connection) {
+				/* Not possible per protocol. Do something better than BUG_ON */
+				pr_warn("Received start of new reply without finishing existing message series.\n");
+				gh_rm_abort_connection(connection);
+			}
+			connection = gh_rm_process_rply(rsc_mgr, msg, msg_size);
+			break;
+		case GH_RM_RPC_TYPE_CONT:
+			if (!connection) {
+				pr_warn("Received a continuation message without receiving initial message\n");
+				break;
+			}
+			gh_rm_process_cont(connection, msg, msg_size);
+			break;
+		default:
+			pr_err("Invalid message type (%d) received\n", hdr->type);
+			continue;
+		}
+
+		if (gh_rm_complete_connection(connection))
+			connection = NULL;
+	}
+
+	return 0;
+}
+
+static int gh_rm_send_request(struct gh_rsc_mgr *rsc_mgr, u32 message_id,
+				const void *req_buff, size_t req_buff_size,
+				struct gh_rm_connection *connection)
+{
+	size_t buff_size_remaining = req_buff_size;
+	const void *req_buff_curr = req_buff;
+	struct gh_rm_rpc_hdr *hdr;
+	unsigned long tx_flags;
+	u32 num_fragments = 0;
+	size_t payload_size;
+	void *msg;
+	int i, ret = 0;
+
+	if (req_buff_size > GH_RM_MAX_MSG_SIZE)
+		num_fragments = req_buff_size / GH_RM_MAX_MSG_SIZE;
+
+	if (WARN(num_fragments > GH_RM_MAX_NUM_FRAGMENTS,
+		 "Limit exceeded for the number of fragments: %u\n", num_fragments))
+		return -E2BIG;
+
+	/*
+	 * The above calculation also includes the count for the 'request' packet.
+	 * Exclude it as the header needs to fill the num. of fragments to follow.
+	 */
+	if (num_fragments)
+		num_fragments--;
+
+	if (mutex_lock_interruptible(&rsc_mgr->send_lock))
+		return -ERESTARTSYS;
+
+	msg = kzalloc(GH_MSGQ_MAX_MSG_SIZE, GFP_KERNEL);
+	if (!msg) {
+		mutex_unlock(&rsc_mgr->send_lock);
+		return -ENOMEM;
+	}
+
+	/* Consider also the 'request' packet for the loop count */
+	for (i = 0; i <= num_fragments; i++) {
+		if (buff_size_remaining > GH_RM_MAX_MSG_SIZE) {
+			payload_size = GH_RM_MAX_MSG_SIZE;
+			buff_size_remaining -= payload_size;
+		} else {
+			payload_size = buff_size_remaining;
+		}
+
+		memset(msg, 0, GH_MSGQ_MAX_MSG_SIZE);
+
+		/* Fill header */
+		hdr = msg;
+		hdr->version = GH_RM_RPC_HDR_VERSION_ONE;
+		hdr->hdr_words = GH_RM_RPC_HDR_WORDS;
+		hdr->type = i == 0 ? GH_RM_RPC_TYPE_REQ : GH_RM_RPC_TYPE_CONT;
+		hdr->fragments = num_fragments;
+		hdr->seq = connection->seq;
+		hdr->msg_id = message_id;
+
+		/* Copy payload */
+		memcpy(msg + sizeof(*hdr), req_buff_curr, payload_size);
+		req_buff_curr += payload_size;
+
+		/* Force the last fragment to be sent immediately to the receiver */
+		tx_flags = (i == num_fragments) ? GH_MSGQ_TX_PUSH : 0;
+
+		ret = gh_msgq_send(rsc_mgr->msgq_tx, msg, sizeof(*hdr) + payload_size, tx_flags);
+
+		if (ret < 0)
+			break;
+	}
+
+	mutex_unlock(&rsc_mgr->send_lock);
+	return ret < 0 ? ret : 0;
+}
+
+/**
+ * gh_rm_call: Achieve request-response type communication with RPC
+ * @message_id: The RM RPC message-id
+ * @req_buff: Request buffer that contains the payload
+ * @req_buff_size: Total size of the payload
+ * @resp_buf: Pointer to a response buffer
+ * @resp_buff_size: Size of the response buffer
+ * @reply_err_code: Returns Gunyah standard error code for the response
+ *
+ * Make a request to the RM-VM and wait for reply back. For a successful
+ * response, the function returns the payload. The size of the payload is set in resp_buff_size.
+ * The resp_buf should be freed by the caller.
+ *
+ * Context: Process context. Will sleep waiting for reply.
+ * Return: >0 is standard reply error from RM. <0 on internal error.
+ */
+int gh_rm_call(u32 message_id, void *req_buff, size_t req_buff_size,
+		void **resp_buf, size_t *resp_buff_size)
+{
+	struct gh_rm_connection *connection;
+	int ret;
+	struct gh_rsc_mgr *rsc_mgr = __rsc_mgr;
+
+	/* messaged_id 0 is reserved */
+	if (!message_id)
+		return -EINVAL;
+
+	if (!rsc_mgr)
+		return -EPROBE_DEFER;
+
+	connection = gh_rm_alloc_connection(message_id, GH_RM_RPC_TYPE_RPLY);
+	if (!connection)
+		return -ENOMEM;
+
+	init_completion(&connection->seq_done);
+
+	/* Allocate a new seq number for this connection */
+	if (mutex_lock_interruptible(&rsc_mgr->call_idr_lock)) {
+		kfree(connection);
+		return -ERESTARTSYS;
+	}
+	connection->seq = idr_alloc_cyclic(&rsc_mgr->call_idr, connection, 0, U16_MAX, GFP_KERNEL);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	/* Send the request to the Resource Manager */
+	ret = gh_rm_send_request(rsc_mgr, message_id, req_buff, req_buff_size, connection);
+	if (ret < 0)
+		goto out;
+
+	/* Wait for response */
+	wait_for_completion(&connection->seq_done);
+
+	if (connection->ret) {
+		ret = connection->ret;
+		kfree(connection->payload);
+		goto out;
+	}
+
+	if (connection->rm_error) {
+		ret = connection->rm_error;
+		kfree(connection->payload);
+		goto out;
+	}
+
+	*resp_buf = connection->payload;
+	*resp_buff_size = connection->size;
+
+out:
+	mutex_lock(&rsc_mgr->call_idr_lock);
+	idr_remove(&rsc_mgr->call_idr, connection->seq);
+	mutex_unlock(&rsc_mgr->call_idr_lock);
+
+	kfree(connection);
+	return ret;
+}
+
+int gh_rm_register_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_register(&gh_rm_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(gh_rm_register_notifier);
+
+int gh_rm_unregister_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_unregister(&gh_rm_notifier, nb);
+}
+EXPORT_SYMBOL_GPL(gh_rm_unregister_notifier);
+
+static struct gunyah_device *gh_msgq_platform_probe_direction(struct platform_device *pdev,
+				u8 gh_type, int idx)
+{
+	int irq, ret;
+	u64 capid;
+	struct device_node *node = pdev->dev.of_node;
+	struct gunyah_device *ghdev;
+
+	irq = platform_get_irq(pdev, idx);
+	if (irq < 0) {
+		dev_err(&pdev->dev, "Failed to get irq%d: %d\n", idx, irq);
+		return ERR_PTR(irq);
+	}
+
+	ret = of_property_read_u64_index(node, "reg", idx, &capid);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to get capid%d: %d\n", idx, ret);
+		return ERR_PTR(ret);
+	}
+
+	ghdev = gunyah_device_alloc(&pdev->dev, capid, gh_type);
+	ghdev->irq = irq;
+	ret = gunyah_device_add(ghdev);
+	if (ret) {
+		kfree(ghdev);
+		return ERR_PTR(ret);
+	}
+
+	return ghdev;
+}
+
+static int gh_rm_drv_probe(struct platform_device *pdev)
+{
+	struct gh_rsc_mgr *rsc_mgr;
+	int ret;
+
+	rsc_mgr = devm_kzalloc(&pdev->dev, sizeof(*rsc_mgr), GFP_KERNEL);
+	if (!rsc_mgr)
+		return -ENOMEM;
+	platform_set_drvdata(pdev, rsc_mgr);
+
+	mutex_init(&rsc_mgr->call_idr_lock);
+	idr_init(&rsc_mgr->call_idr);
+	mutex_init(&rsc_mgr->send_lock);
+
+	rsc_mgr->msgq_tx = gh_msgq_platform_probe_direction(pdev, GUNYAH_DEVICE_TYPE_MSGQ_TX, 0);
+	if (IS_ERR(rsc_mgr->msgq_tx))
+		return PTR_ERR(rsc_mgr->msgq_tx);
+	rsc_mgr->msgq_rx = gh_msgq_platform_probe_direction(pdev, GUNYAH_DEVICE_TYPE_MSGQ_RX, 1);
+	if (IS_ERR(rsc_mgr->msgq_rx)) {
+		ret = PTR_ERR(rsc_mgr->msgq_rx);
+		goto err_msgq_tx;
+	}
+
+	rsc_mgr->recv_task = kthread_run(gh_rm_recv_task_fn, rsc_mgr, "gh_rm_recv_task");
+	if (IS_ERR_OR_NULL(rsc_mgr->recv_task)) {
+		ret = PTR_ERR(rsc_mgr->recv_task);
+		goto err_msgq;
+	}
+
+	__rsc_mgr = rsc_mgr;
+
+	return 0;
+
+err_msgq:
+	gunyah_device_remove(rsc_mgr->msgq_rx);
+err_msgq_tx:
+	gunyah_device_remove(rsc_mgr->msgq_tx);
+	return ret;
+}
+
+static int gh_rm_drv_remove(struct platform_device *pdev)
+{
+	struct gh_rsc_mgr *rsc_mgr = platform_get_drvdata(pdev);
+
+	gunyah_device_remove(rsc_mgr->msgq_tx);
+	gunyah_device_remove(rsc_mgr->msgq_rx);
+
+	return 0;
+}
+
+static const struct of_device_id gh_rm_of_match[] = {
+	{ .compatible = "gunyah-resource-manager" },
+	{ }
+};
+MODULE_DEVICE_TABLE(of, gh_rm_of_match);
+
+static struct platform_driver gh_rsc_mgr_driver = {
+	.probe = gh_rm_drv_probe,
+	.remove = gh_rm_drv_remove,
+	.driver = {
+		.name = "gh_rsc_mgr",
+		.of_match_table = gh_rm_of_match,
+	},
+};
+
+int __init gh_rsc_mgr_init(void)
+{
+	return platform_driver_register(&gh_rsc_mgr_driver);
+}
+
+void gh_rsc_mgr_exit(void)
+{
+	platform_driver_unregister(&gh_rsc_mgr_driver);
+}
diff --git a/drivers/virt/gunyah/rsc_mgr.h b/drivers/virt/gunyah/rsc_mgr.h
new file mode 100644
index 000000000000..e4f2499267bf
--- /dev/null
+++ b/drivers/virt/gunyah/rsc_mgr.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+#ifndef __GH_RSC_MGR_PRIV_H
+#define __GH_RSC_MGR_PRIV_H
+
+#include <linux/gunyah.h>
+
+/* RM Error codes */
+#define GH_RM_ERROR_OK			0x0
+#define GH_RM_ERROR_UNIMPLEMENTED	0xFFFFFFFF
+#define GH_RM_ERROR_NOMEM		0x1
+#define GH_RM_ERROR_NORESOURCE		0x2
+#define GH_RM_ERROR_DENIED		0x3
+#define GH_RM_ERROR_INVALID		0x4
+#define GH_RM_ERROR_BUSY		0x5
+#define GH_RM_ERROR_ARGUMENT_INVALID	0x6
+#define GH_RM_ERROR_HANDLE_INVALID	0x7
+#define GH_RM_ERROR_VALIDATE_FAILED	0x8
+#define GH_RM_ERROR_MAP_FAILED		0x9
+#define GH_RM_ERROR_MEM_INVALID		0xA
+#define GH_RM_ERROR_MEM_INUSE		0xB
+#define GH_RM_ERROR_MEM_RELEASED	0xC
+#define GH_RM_ERROR_VMID_INVALID	0xD
+#define GH_RM_ERROR_LOOKUP_FAILED	0xE
+#define GH_RM_ERROR_IRQ_INVALID		0xF
+#define GH_RM_ERROR_IRQ_INUSE		0x10
+#define GH_RM_ERROR_IRQ_RELEASED	0x11
+
+int gh_rm_call(u32 message_id, void *req_buff, size_t req_buff_size,
+		void **resp_buf, size_t *resp_buff_size);
+
+#endif
diff --git a/drivers/virt/gunyah/sysfs.c b/drivers/virt/gunyah/sysfs.c
index 7589689e5e92..7c0efc80f85e 100644
--- a/drivers/virt/gunyah/sysfs.c
+++ b/drivers/virt/gunyah/sysfs.c
@@ -148,7 +148,13 @@ static int __init gunyah_init(void)
 	if (ret)
 		goto err_bus;
 
+	ret = gh_rsc_mgr_init();
+	if (ret)
+		goto err_msgq;
+
 	return ret;
+err_msgq:
+	gh_msgq_exit();
 err_bus:
 	gunyah_bus_exit();
 err_sysfs:
@@ -159,6 +165,7 @@ module_init(gunyah_init);
 
 static void __exit gunyah_exit(void)
 {
+	gh_rsc_mgr_exit();
 	gh_msgq_exit();
 	gunyah_bus_exit();
 	gh_sysfs_unregister();
diff --git a/include/linux/gunyah_rsc_mgr.h b/include/linux/gunyah_rsc_mgr.h
new file mode 100644
index 000000000000..015bd851e1a3
--- /dev/null
+++ b/include/linux/gunyah_rsc_mgr.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2022 Qualcomm Innovation Center, Inc. All rights reserved.
+ */
+
+#ifndef _GUNYAH_RSC_MGR_H
+#define _GUNYAH_RSC_MGR_H
+
+#include <linux/list.h>
+#include <linux/notifier.h>
+#include <linux/gunyah.h>
+
+typedef u16 gh_vmid_t;
+typedef u32 gh_virq_handle_t;
+
+#define GH_VMID_INVAL	U16_MAX
+
+/* Gunyah recognizes VMID0 as an alias to the current VM's ID */
+#define GH_VMID_SELF			0
+
+struct gh_rm_notification {
+	const void *buff;
+	const size_t size;
+};
+
+int gh_rm_register_notifier(struct notifier_block *nb);
+int gh_rm_unregister_notifier(struct notifier_block *nb);
+
+#endif
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel