dmaengine Archive on
 help / color / Atom feed
From: Dave Jiang <>
To:,,,,,,,,,,, David.Laight@ACULAB.COM
Subject: [PATCH v6 0/5] Add shared workqueue support for idxd driver
Date: Thu, 24 Sep 2020 11:00:36 -0700
Message-ID: <> (raw)

- Fixup MOBDIR64B inline asm input/output constraints

- Fixup commit headers
- Fixup var names for movdir64b()
- Move enqcmds() to special_insns.h
- Fix up comments for enqcmds()
- Change enqcmds() to reflect instruction return. 0 as success, -EAGAIN for fail.

- Fixup enqcmds() gas constraints

- Rebased against latest dmaengine/next tree
- Split out enqcmd and pasid dependency.

- Rebased against latest dmaengine/next tree.
- Updated API doc with new kernel version and dates.
- Changed to allow driver to load without ENQCMD support.
- Break out some patches that can be sent ahead of this series for inclusion.

- Dropped device feature enabling (GregKH)
- Dropped PCI device feature enabling (Bjorn)
- After some internal discussion, we have decided to hold off on the enabling of DMWR due to the
  following reasons. 1. Most first gen hw will not have the feature bits. 2. First gen hw that
  support the feature are all Root Complex integrated endpoints. 3. PCI devices that are not
  RCiEP’s with this capability won’t surface for a few years so we can wait until we can test the
  full code.
- Dropped special ioremap (hch)
- Added proper support for WQ flush (tony, dan)
- Changed descriptor submission to use sbitmap_queue for blocking. (dan)

Driver stage 1 postings for context: [1]

The patch series has compilation and functional dependency on Fenghua's "Tag application
address space for devices" patch series for the ENQCMD CPU command enumeration and the PASID MSR
support. [2] 

== Background ==
A typical DMA device requires the driver to translate application buffers to hardware addresses,
and a kernel-user transition to notify the hardware of new work. Shared Virtual Addressing (SVA)
allows the processor and device to use the same virtual addresses without requiring software to
translate between the address spaces. ENQCMD is a new instruction on Intel Platforms that allows
user applications to directly notify hardware of new work, much like how doorbells are used in
some hardware, but it carries a payload along with it. ENQCMDS is the supervisor version (ring0)

Introduce enqcmds(), a helper funciton that copies an input payload to a 64B aligned
destination and confirms whether the payload was accepted by the device or not.
enqcmds() wraps the new ENQCMDS CPU instruction. The ENQCMDS is a ring 0 CPU instruction that
performs similar to the ENQCMD instruction. Descriptor submission must use ENQCMD(S) for shared
workqueues (swq) on an Intel DSA device. 

== Shared WQ support ==
Introduce shared workqueue (swq) support for the idxd driver. The current idxd driver contains
dedicated workqueue (dwq) support only. A dwq accepts descriptors from a MOVDIR64B instruction.
MOVDIR64B is a posted instruction on the PCIe bus, it does not wait for any response from the
device. If the wq is full, submitted descriptors are dropped. A swq utilizes the ENQCMDS in
ring 0, which is a non-posted instruction. The zero flag would be set to 1 if the device rejects
the descriptor or if the wq is full. A swq can be shared between multiple users
(kernel or userspace) due to not having to keep track of the wq full condition for submission.
A swq requires PASID and can only run with SVA support. 

== IDXD SVA support ==
Add utilization of PASID to support Shared Virtual Addressing (SVA). With PASID support,
the descriptors can be programmed with host virtual address (HVA) rather than IOVA.
The hardware will work with the IOMMU in fulfilling page requests. With SVA support,
a user app using the char device interface can now submit descriptors without having to pin the
virtual memory range it wants to DMA in its own address space. 

The series does not add SVA support for the dmaengine subsystem. That support is coming at a
later time.

[8]: idxd-stage2


Dave Jiang (5):
  x86/asm: Carve out a generic movdir64b() helper for general usage
  x86/asm: Add enqcmds() to support ENQCMDS instruction
  dmaengine: idxd: Add shared workqueue support
  dmaengine: idxd: Clean up descriptors with fault error
  dmaengine: idxd: Add ABI documentation for shared wq

 .../ABI/stable/sysfs-driver-dma-idxd          |  14 ++
 arch/x86/include/asm/io.h                     |  17 +--
 arch/x86/include/asm/special_insns.h          |  56 +++++++
 drivers/dma/Kconfig                           |  10 ++
 drivers/dma/idxd/cdev.c                       |  49 +++++-
 drivers/dma/idxd/device.c                     |  91 ++++++++++-
 drivers/dma/idxd/dma.c                        |   9 --
 drivers/dma/idxd/idxd.h                       |  33 +++-
 drivers/dma/idxd/init.c                       |  92 ++++++++---
 drivers/dma/idxd/irq.c                        | 143 ++++++++++++++++--
 drivers/dma/idxd/registers.h                  |  14 ++
 drivers/dma/idxd/submit.c                     |  35 ++++-
 drivers/dma/idxd/sysfs.c                      | 127 ++++++++++++++++
 13 files changed, 620 insertions(+), 70 deletions(-)


             reply index

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-24 18:00 Dave Jiang [this message]
2020-09-24 18:00 ` [PATCH v6 1/5] x86/asm: Carve out a generic movdir64b() helper for general usage Dave Jiang
2020-09-24 18:33   ` Borislav Petkov
2020-09-24 18:00 ` [PATCH v6 2/5] x86/asm: Add enqcmds() to support ENQCMDS instruction Dave Jiang
2020-09-24 18:58   ` Borislav Petkov
2020-09-24 21:14     ` Dave Jiang
2020-09-24 22:09   ` David Laight
2020-09-24 18:00 ` [PATCH v6 3/5] dmaengine: idxd: Add shared workqueue support Dave Jiang
2020-10-05  4:35   ` Vinod Koul
2020-09-24 18:00 ` [PATCH v6 4/5] dmaengine: idxd: Clean up descriptors with fault error Dave Jiang
2020-10-05  4:42   ` Vinod Koul
2020-10-05  4:55     ` Dave Jiang
2020-10-05  5:45       ` Vinod Koul
2020-09-24 18:00 ` [PATCH v6 5/5] dmaengine: idxd: Add ABI documentation for shared wq Dave Jiang
2020-10-05  4:43   ` Vinod Koul
2020-09-24 21:32 ` [PATCH v6 0/5] Add shared workqueue support for idxd driver Dave Jiang
2020-09-24 21:51   ` Borislav Petkov
2020-09-24 22:05     ` Dave Jiang
2020-09-30 22:19     ` Dave Jiang
2020-10-01  4:29       ` Vinod Koul
2020-10-01  7:30         ` Borislav Petkov
2020-10-01 12:18         ` Jiang, Dave

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \
    --cc=David.Laight@ACULAB.COM \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

dmaengine Archive on

Archives are clonable:
	git clone --mirror dmaengine/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dmaengine dmaengine/ \
	public-inbox-index dmaengine

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone