[QUERY] Flushing cache from userspace using VFIO

[QUERY] Flushing cache from userspace using VFIO

[Kishon Vijay Abraham I]

Hi Alex, Cornelia,

I'm trying to see if I can use VFIO (Versatile Framework for userspace I/O
[1]) for communication between two cores within the same SoC. I've tried to put
down a picture like below which tries to communicate between ARM64 (running
Linux) and CORTEX R5 (running firmware). It uses rpmsg/remoteproc for the
control messages and the actual data buffers are directly accessed from the
userspace. The location of the data buffers can be informed to the userspace via
rpmsg_vfio (which has to be built as a rpmsg endpoint).

My question is after the userspace application in ARM64 writes to a buffer in
the SYSTEM MEMORY, can it flush it (through a VFIO IOCTL) before handing the
buffer to the CORTEX R5.

If it's implemented within kernel either we use dma_alloc_coherent() for
allocating coherent memory or streaming DMA APIs like
dma_map_single()/dma_unmap_single() for flushing/invalidate the cache.

Trying to see if that is already supported in VFIO or if not, would it be
acceptable to implement it.

Please let me know your thoughts.

┌───────────────────────────────────────────────────────────────────────────┐
│                                                                           │
│ ┌────────────────────┐                                                    │
│ │                    │                                                    │
│ │ ┌──────────────┐   │                                                    │
│ │ │ userspace    │   │  Data Buffers                                      │
│ │ │ Application  ├───┼──────────────┐                                     │
│ │ │              │   │              │                                     │
│ │ └──────▲──┬────┘   │              │                                     │
│ │        │  │        │              │                                     │
│ │        │  │ user   │              │                 ┌─────────────────┐ │
│ │  ──────┼──┼─────── │              │                 │                 │ │
│ │        │  │ kernel │              │                 │                 │ │
│ │  ┌─────┴──▼────┐   │    ┌─────────┼────────────┐    │                 │ │
│ │  │             │   │    │         │            │    │  Data           │ │
│ │  │  rpmsg_vfio │   │    │  ┌──────▼─────────┐  │    │  Buffers        │ │
│ │  │             │   │    │  │ Reserved Region◄──┼────┼────────┐        │ │
│ │  └─────▲──┬────┘   │    │  │                │  │    │        │        │ │
│ │        │  │        │    │  └────────────────┘  │    │        │        │ │
│ │  ┌─────┴──▼────┐   │    │                      │    │        │        │ │
│ │  │             │   │    │                      │    │ ┌──────┴──────┐ │ │
│ │  │  rpmsg      │   │    │     SYSTEM MEMORY    │    │ │ Application │ │ │
│ │  │             │   │    │       (DDR)          │    │ │   Logic     │ │ │
│ │  └─────▲──┬────┘   │    └──────────────────────┘    │ └───▲────┬────┘ │ │
│ │        │  │        │                                │     │    │      │ │
│ │  ┌─────┴──▼────┐   │Notify Firmware/Control Message │ ┌───┴────▼────┐ │ │
│ │  │             ├───┼────────────────────────────────┼─►             │ │ │
│ │  │  remoteproc │   │Interrupt ARM/Control Message   │ │   Firmware  │ │ │
│ │  │             ◄───┼────────────────────────────────┼─┤             │ │ │
│ │  └─────────────┘   │                                │ └─────────────┘ │ │
│ │      ARM64(Linux)  │                                │   ARM CORTEX R5 │ │
│ └────────────────────┘                                └─────────────────┘ │
│                                                                           │
│                                                                       SoC │
└───────────────────────────────────────────────────────────────────────────┘

Thank You,
Kishon

[1] -> https://youtu.be/WFkdTFTOTpA

Re: [QUERY] Flushing cache from userspace using VFIO

[Alex Williamson]

Hi Kishon,

On Mon, 6 Sep 2021 21:22:15 +0530
Kishon Vijay Abraham I <kishon@ti.com> wrote:

> Hi Alex, Cornelia,
> 
> I'm trying to see if I can use VFIO (Versatile Framework for userspace I/O
> [1]) for communication between two cores within the same SoC. I've tried to put
> down a picture like below which tries to communicate between ARM64 (running
> Linux) and CORTEX R5 (running firmware). It uses rpmsg/remoteproc for the
> control messages and the actual data buffers are directly accessed from the
> userspace. The location of the data buffers can be informed to the userspace via
> rpmsg_vfio (which has to be built as a rpmsg endpoint).

In the vfio model, the user gets access to a device that's a member of
an IOMMU isolation group whose IOMMU context is managed by a vfio
container.  What "device" is the user getting access to here and is an
IOMMU involved?

> My question is after the userspace application in ARM64 writes to a buffer in
> the SYSTEM MEMORY, can it flush it (through a VFIO IOCTL) before handing the
> buffer to the CORTEX R5.

No such vfio ioctl currently exists.  Now you're starting to get into
KVM space if userspace requires elevated privileges to flush memory.
See for example the handling of wbinvd (write-back-invalidate) in x86
KVM based on an assigned device and coherency model supported by the
IOMMU.  vfio is only facilitating isolated access to the device. 
 
> If it's implemented within kernel either we use dma_alloc_coherent() for
> allocating coherent memory or streaming DMA APIs like
> dma_map_single()/dma_unmap_single() for flushing/invalidate the cache.

In vfio, DMA is mapped to userspace buffers.  The user allocates a
buffer and maps it for device access.  The IOMMU restricts the device to
only allow access to those buffers.  Accessing device memory in vfio is
done via regions on the device file descriptor, a device specific
region could allow a user to mmap that buffer, but the fact that this
buffer actually lives in host memory per your model and requires DMA
programming for the cortex core makes that really troubling.

For a vfio model to work, I think userspace would need to allocate the
buffers and the cortex core would need to be represented as a device
that supports isolation via an IOMMU.  Otherwise I'm not sure what
benefit you're getting from vfio.
 
> Trying to see if that is already supported in VFIO or if not, would it be
> acceptable to implement it.

vfio provides a user with privilege to access an isolated device, what
you're proposing could possibly be mangled to fit that model, but it
seems pretty awkward and there are existing solutions such as KVM for
processor virtualization if userspace needs elevated privileges to
handle CPU/RAM coherency.  Thanks,

Alex