ENQCMD

ENQCMD

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

Hi,
The "Scalable Work Submission in Device Virtualization" talk at KVM
Forum 2020 was interesting and I have some beginner questions about
ENQCMD:
https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Submission_In_Device_Virtualization.pdf

Security
--------
If the ENQCMD instruction is allowed for userspace applications, how can
they be prevented from writing to the MMIO address directly (without the
ENQCMD instruction) and faking the 64-byte enqueue register data format?
For example, they could set the PRIV bit or an arbitrary PASID.

Work Queue Design
-----------------
Have you looked at extending existing hardware interfaces like NVMe or
VIRTIO to support enqueue registers?

Have you benchmarked NVMe or VIRTIO devices using ENQCMD instead of
the traditional submission queuing mechanism?

Is ENQCMD faster than traditional I/O request submission? If not, then I
guess it's only intended for shared queues where the PASID translation
is needed?

A few thoughts come to mind:

 * Traditional submission queues are no longer needed and can be
   replaced by an enqueue register. NVMe sqs and VIRTIO avail rings
   aren't needed anymore, although the sqes and vring descriptors are
   still needed to represent commands and buffers.

   Or the enqueue register can be used simply as a doorbell to start DMA
   reading requests from a traditional submission queue. In this case
   the advantage is that a single shared hardware unit (ADI) can emulate
   multiple queues at the same time.

 * In order to support submitting multiple requests in a single enqueue
   register access there needs to be some kind of chaining mechanism.
   For example, the Device Specific Command field contains a num_reqs
   field telling the device how many requests to DMA.

I don't know much about ENQCMD and am trying to figure out where it fits
in. Please let me know if this matches how this feature is intended to
be used.

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

ENQCMD

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

Hi,
The "Scalable Work Submission in Device Virtualization" talk at KVM
Forum 2020 was interesting and I have some beginner questions about
ENQCMD:
https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Submission_In_Device_Virtualization.pdf

Security
--------
If the ENQCMD instruction is allowed for userspace applications, how can
they be prevented from writing to the MMIO address directly (without the
ENQCMD instruction) and faking the 64-byte enqueue register data format?
For example, they could set the PRIV bit or an arbitrary PASID.

Work Queue Design
-----------------
Have you looked at extending existing hardware interfaces like NVMe or
VIRTIO to support enqueue registers?

Have you benchmarked NVMe or VIRTIO devices using ENQCMD instead of
the traditional submission queuing mechanism?

Is ENQCMD faster than traditional I/O request submission? If not, then I
guess it's only intended for shared queues where the PASID translation
is needed?

A few thoughts come to mind:

 * Traditional submission queues are no longer needed and can be
   replaced by an enqueue register. NVMe sqs and VIRTIO avail rings
   aren't needed anymore, although the sqes and vring descriptors are
   still needed to represent commands and buffers.

   Or the enqueue register can be used simply as a doorbell to start DMA
   reading requests from a traditional submission queue. In this case
   the advantage is that a single shared hardware unit (ADI) can emulate
   multiple queues at the same time.

 * In order to support submitting multiple requests in a single enqueue
   register access there needs to be some kind of chaining mechanism.
   For example, the Device Specific Command field contains a num_reqs
   field telling the device how many requests to DMA.

I don't know much about ENQCMD and am trying to figure out where it fits
in. Please let me know if this matches how this feature is intended to
be used.

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: ENQCMD

[Tian, Kevin]

> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: Friday, October 30, 2020 3:51 PM
> 
> Hi,
> The "Scalable Work Submission in Device Virtualization" talk at KVM
> Forum 2020 was interesting and I have some beginner questions about
> ENQCMD:
> https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Su
> bmission_In_Device_Virtualization.pdf
> 
> Security
> --------
> If the ENQCMD instruction is allowed for userspace applications, how can
> they be prevented from writing to the MMIO address directly (without the
> ENQCMD instruction) and faking the 64-byte enqueue register data format?
> For example, they could set the PRIV bit or an arbitrary PASID.

ENQCMD payload is transmitted through DMWr transactions (slide 10), which
cannot be triggered through other memory instructions. The device portal
only handles DMWr transactions.

PRIV cannot be set by ENQCMD. Only by ENQCMDS (in ring0).

PASID is copied from a MSR when ENQCMD is executed. the MSR is managed
by the kernel. Userspace cannot specify it directly.

> 
> Work Queue Design
> -----------------
> Have you looked at extending existing hardware interfaces like NVMe or
> VIRTIO to support enqueue registers?

The first devices supporting ENQCMD are DSA/IAX. The native support 
just landed in the kernel. In concept any device which wants to support
shared work queue can implement this capability.

> 
> Have you benchmarked NVMe or VIRTIO devices using ENQCMD instead of
> the traditional submission queuing mechanism?

No. We don't have such devices yet.

> 
> Is ENQCMD faster than traditional I/O request submission? If not, then I
> guess it's only intended for shared queues where the PASID translation
> is needed?

ENQCMD is non-posted transaction. So it's a little bit slower than normal
writes. The point is to carry 'retry' signal back in case of resource contention.

and 64bytes payload is also an advantage. e.g. for DSA/IAX the whole
work descriptor can be included in the ENQCMD payload.

> 
> A few thoughts come to mind:
> 
>  * Traditional submission queues are no longer needed and can be
>    replaced by an enqueue register. NVMe sqs and VIRTIO avail rings
>    aren't needed anymore, although the sqes and vring descriptors are
>    still needed to represent commands and buffers.

yes, this is one possible enhancement.

> 
>    Or the enqueue register can be used simply as a doorbell to start DMA
>    reading requests from a traditional submission queue. In this case
>    the advantage is that a single shared hardware unit (ADI) can emulate
>    multiple queues at the same time.
> 
>  * In order to support submitting multiple requests in a single enqueue
>    register access there needs to be some kind of chaining mechanism.
>    For example, the Device Specific Command field contains a num_reqs
>    field telling the device how many requests to DMA.

yes, this leaves to device specific format. e.g. DSA defines a chaining
descript type to chain multiple requests together.

> 
> I don't know much about ENQCMD and am trying to figure out where it fits
> in. Please let me know if this matches how this feature is intended to
> be used.
> 

I think you captured most of the intention of  ENQCMD. 😊

Thanks
Kevin

RE: ENQCMD

[Tian, Kevin]

> From: Stefan Hajnoczi <stefanha@redhat.com>
> Sent: Friday, October 30, 2020 3:51 PM
> 
> Hi,
> The "Scalable Work Submission in Device Virtualization" talk at KVM
> Forum 2020 was interesting and I have some beginner questions about
> ENQCMD:
> https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Su
> bmission_In_Device_Virtualization.pdf
> 
> Security
> --------
> If the ENQCMD instruction is allowed for userspace applications, how can
> they be prevented from writing to the MMIO address directly (without the
> ENQCMD instruction) and faking the 64-byte enqueue register data format?
> For example, they could set the PRIV bit or an arbitrary PASID.

ENQCMD payload is transmitted through DMWr transactions (slide 10), which
cannot be triggered through other memory instructions. The device portal
only handles DMWr transactions.

PRIV cannot be set by ENQCMD. Only by ENQCMDS (in ring0).

PASID is copied from a MSR when ENQCMD is executed. the MSR is managed
by the kernel. Userspace cannot specify it directly.

> 
> Work Queue Design
> -----------------
> Have you looked at extending existing hardware interfaces like NVMe or
> VIRTIO to support enqueue registers?

The first devices supporting ENQCMD are DSA/IAX. The native support 
just landed in the kernel. In concept any device which wants to support
shared work queue can implement this capability.

> 
> Have you benchmarked NVMe or VIRTIO devices using ENQCMD instead of
> the traditional submission queuing mechanism?

No. We don't have such devices yet.

> 
> Is ENQCMD faster than traditional I/O request submission? If not, then I
> guess it's only intended for shared queues where the PASID translation
> is needed?

ENQCMD is non-posted transaction. So it's a little bit slower than normal
writes. The point is to carry 'retry' signal back in case of resource contention.

and 64bytes payload is also an advantage. e.g. for DSA/IAX the whole
work descriptor can be included in the ENQCMD payload.

> 
> A few thoughts come to mind:
> 
>  * Traditional submission queues are no longer needed and can be
>    replaced by an enqueue register. NVMe sqs and VIRTIO avail rings
>    aren't needed anymore, although the sqes and vring descriptors are
>    still needed to represent commands and buffers.

yes, this is one possible enhancement.

> 
>    Or the enqueue register can be used simply as a doorbell to start DMA
>    reading requests from a traditional submission queue. In this case
>    the advantage is that a single shared hardware unit (ADI) can emulate
>    multiple queues at the same time.
> 
>  * In order to support submitting multiple requests in a single enqueue
>    register access there needs to be some kind of chaining mechanism.
>    For example, the Device Specific Command field contains a num_reqs
>    field telling the device how many requests to DMA.

yes, this leaves to device specific format. e.g. DSA defines a chaining
descript type to chain multiple requests together.

> 
> I don't know much about ENQCMD and am trying to figure out where it fits
> in. Please let me know if this matches how this feature is intended to
> be used.
> 

I think you captured most of the intention of  ENQCMD. 😊

Thanks
Kevin

Re: ENQCMD

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Fri, Oct 30, 2020 at 08:04:54AM +0000, Tian, Kevin wrote:
> > From: Stefan Hajnoczi <stefanha@redhat.com>
> > Sent: Friday, October 30, 2020 3:51 PM
> > 
> > Hi,
> > The "Scalable Work Submission in Device Virtualization" talk at KVM
> > Forum 2020 was interesting and I have some beginner questions about
> > ENQCMD:
> > https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Su
> > bmission_In_Device_Virtualization.pdf
> > 
> > Security
> > --------
> > If the ENQCMD instruction is allowed for userspace applications, how can
> > they be prevented from writing to the MMIO address directly (without the
> > ENQCMD instruction) and faking the 64-byte enqueue register data format?
> > For example, they could set the PRIV bit or an arbitrary PASID.
> 
> ENQCMD payload is transmitted through DMWr transactions (slide 10), which
> cannot be triggered through other memory instructions. The device portal
> only handles DMWr transactions.

Thanks, that explains it! I was wondering the a regular write
transaction could fool the device :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ENQCMD

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Fri, Oct 30, 2020 at 08:04:54AM +0000, Tian, Kevin wrote:
> > From: Stefan Hajnoczi <stefanha@redhat.com>
> > Sent: Friday, October 30, 2020 3:51 PM
> > 
> > Hi,
> > The "Scalable Work Submission in Device Virtualization" talk at KVM
> > Forum 2020 was interesting and I have some beginner questions about
> > ENQCMD:
> > https://static.sched.com/hosted_files/kvmforum2020/22/Scalable_Work_Su
> > bmission_In_Device_Virtualization.pdf
> > 
> > Security
> > --------
> > If the ENQCMD instruction is allowed for userspace applications, how can
> > they be prevented from writing to the MMIO address directly (without the
> > ENQCMD instruction) and faking the 64-byte enqueue register data format?
> > For example, they could set the PRIV bit or an arbitrary PASID.
> 
> ENQCMD payload is transmitted through DMWr transactions (slide 10), which
> cannot be triggered through other memory instructions. The device portal
> only handles DMWr transactions.

Thanks, that explains it! I was wondering the a regular write
transaction could fool the device :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]