All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v10 00/20] dlb: introduce DLB device driver
@ 2021-02-10 17:54 Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 01/20] dlb: add skeleton for DLB driver Mike Ximing Chen
                   ` (20 more replies)
  0 siblings, 21 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev; +Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart

---------------------------------------------------------
This is a device driver for a new HW IPC accelerator. It was submitted
to linux-kernel group. Per Greg's (maintainer for drivers/misc) suggestion
(see below), we could like to get the patch set reviewed/acked by the
networking driver community. Thanks.

>As this is a networking related thing, I would like you to get the
>proper reviews/acks from the networking maintainers before I can take
>this.
>
>Or, if they think it has nothing to do with networking, that's fine too,
>but please do not try to route around them.
>
>thanks,
>
>greg k-
---------------------------------------------------------

Introduce a new misc device driver for the Intel(r) Dynamic Load Balancer
(Intel(r) DLB). The Intel DLB is a PCIe device that provides
load-balanced, prioritized scheduling of core-to-core communication.

Intel DLB is an accelerator for the event-driven programming model of
DPDK's Event Device Library[2]. The library is used in packet processing
pipelines that arrange for multi-core scalability, dynamic load-balancing,
and variety of packet distribution and synchronization schemes

These distribution schemes include "parallel" (packets are load-balanced
across multiple cores and processed in parallel), "ordered" (similar to
"parallel" but packets are reordered into ingress order by the device), and
"atomic" (packet flows are scheduled to a single core at a time such that
locks are not required to access per-flow data, and dynamically migrated to
ensure load-balance).

This submission supports Intel DLB 2.0 only.

The Intel DLB consists of queues and arbiters that connect producer
cores and consumer cores. The device implements load-balanced queueing
features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The fundamental unit of communication through the device is a queue entry
(QE), which consists of 8B of data and 8B of metadata (destination queue,
priority, etc.). The data field can be any type that fits within 8B.

A core's interface to the device, a "port," consists of a memory-mappable
region through which the core enqueues a queue entry, and an in-memory
queue (the "consumer queue") to which the device schedules QEs. Each QE
is enqueued to a device-managed queue, and from there scheduled to a port.
Software specifies the "linking" of queues and ports; i.e. which ports the
device is allowed to schedule to for a given queue. The device uses a
credit scheme to prevent overflow of the on-device queue storage.

Applications can interface directly with the device by mapping the port's
memory and MMIO regions into the application's address space for enqueue
and dequeue operations, but call into the kernel driver for configuration
operations. An application can also be polling- or interrupt-driven;
Intel DLB supports both modes of operation.

Device resources -- i.e. ports, queues, and credits -- are contained within
a scheduling domain. Scheduling domains are isolated from one another; a
port can only enqueue to and dequeue from queues within its scheduling
domain. A scheduling domain's resources are configured through a scheduling
domain file, which is acquired through an ioctl.

Intel DLB supports SR-IOV and Scalable IOV, and allows for a flexible
division of its resources among the PF and its virtual devices. The virtual
devices are incapable of configuring the device directly; they use a
hardware mailbox to proxy configuration requests to the PF driver. This
driver supports both PF and virtual devices, as there is significant code
re-use between the two, with device-specific behavior handled through a
callback interface.  Virtualization support will be added in a later patch
set.

The dlb driver uses ioctls as its primary interface (it makes use of sysfs
as well, to a lesser extent). The dlb device file supports a different
ioctl interface than the scheduling domain file; the dlb device file
is used for device-wide operations (including scheduling domain creation),
and the scheduling domain file supports operations on the scheduling
domain's resources (primarily resource configuration). Scheduling domains
are created dynamically (using a dlb device file ioctl) by user-space
software, and the scheduling domain file is created from an anonymous file
that is installed in the ioctl's calling process's file descriptor table.

[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html

v10:
- Addressed an issue reported by kernel test robot <lkp@intel.com>
-- Add "WITH Linux-syscall-note" to the SPDX-License-Identifier in uapi
   header file dlb.h.

v9:
- Addressed all of Greg's feecback on v8, including
-- Remove function name (__func__) from dev_err() messages, that could spam log.
-- Replace list and function pointer calls in dlb_ioctl() with switch-case
   and real function calls for ioctl.
-- Drop the compat_ptr_ioctl in dlb_ops (struct file_operations).
-- Change ioctl magic number for DLB to unused 0x81 (from 'h').
-- Remove all placeholder/dummy functions in the patch set.
-- Re-arrange the comments in dlb.h so that the order is consistent with that
   of data structures referred.
-- Correct the comments on SPDX License and DLB versions in dlb.h.
-- Replace BIT_SET() and BITS_CLR() marcos with direct coding.   
-- Remove NULL pointer checking (f->private_data) in dlb_ioctl().
-- Use whole line whenever possible and not wrapping lines unnecessarily.
-- Remove __attribute__((unused)).
-- Merge dlb_ioctl.h and dlb_file.h into dlb_main.h

v8:
- Add a functional block diagram in dlb.rst 
- Modify change logs to reflect the links between patches and DPDK
  eventdev library.
- Add a check of power-of-2 for CQ depth.
- Move call to INIT_WORK() to dlb_open().
- Clean dlb workqueue by calling flush_scheduled_work().
- Add unmap_mapping_range() in dlb_port_close().

v7 (Intel internal version):
- Address all of Dan's feedback, including
-- Drop DLB 2.0 throughout the patch set, use DLB only.
-- Fix license and copyright statements
-- Use pcim_enable_device() and pcim_iomap_regions(), instead of
   unmanaged version.
-- Move cdev_add() to dlb_init() and add all devices at once.
-- Fix Makefile, using "+=" style.
-- Remove FLR description and mention movdir64/enqcmd usage in doc.
-- Make the permission for the domain same as that for device for
   ioctl access.
-- Use idr instead of ida.
-- Add a lock in dlb_close() to prevent driver unbinding while ioctl
   coomands are in progress.
-- Remove wrappers that are used for code sharing between kernel driver
   and DPDK. 
- Address Pierre-Louis' feedback, including
-- Clean the warinings from checkpatch
-- Fix the warnings from "make W=1"

v6 (Intel internal version):
- Change the module name to dlb(from dlb2), which currently supports Intel
  DLB 2.0 only.
- Address all of Pierre-Louis' feedback on v5, including
-- Consolidate the two near-identical for loops in dlb2_release_domain_memory().
-- Remove an unnecessary "port = NULL" initialization
-- Consistently use curly braces on the *_LIST_FOR macros
   when the for-loop contents spans multiple lines.
-- Add a comment to the definition of DLB2FS_MAGIC
-- Remove always true if statemnets
-- Move the get_cos_bw mutex unlock call earlier to shorten the critical
   section.
- Address all of Dan's feedbacks, including
-- Replace the unions for register bits access with bitmask and shifts
-- Centralize the "to/from" user memory copies for ioctl functions.
-- Review ioctl design against Documentation/process/botching-up-ioctls.rst
-- Remove wraper functions for memory barriers.
-- Use ilog() to simplify a switch code block.
-- Add base-commit to cover letter.

v5 (Intel internal version):
- Reduce the scope of the initial patch set (drop the last 8 patches)
- Further decompose some of the remaining patches into multiple patches.
- Address all of Pierre-Louis' feedback, including:
-- Move kerneldoc to *.c files
-- Fix SPDX comment style
-- Add BAR macros
-- Improve/clarify struct dlb2_dev and struct device variable naming
-- Add const where missing
-- Clarify existing comments and add new ones in various places
-- Remove unnecessary memsets and zero-initialization
-- Remove PM abstraction, fix missing pm_runtime_allow(), and don't
   update PM refcnt when port files are opened and closed.
-- Convert certain ternary operations into if-statements
-- Out-line the CQ depth valid check
-- De-duplicate the logic in dlb2_release_device_memory()
-- Limit use of devm functions to allocating/freeing struct dlb2
- Address Ira's comments on dlb2.rst and correct commit messages that
  don't use the imperative voice.

v4:
- Move PCI device ID definitions into dlb2_hw_types.h, drop the VF definition
- Remove dlb2_dev_list
- Remove open/close functions and fops structure (unused)
- Remove "(char *)" cast from PCI driver name
- Unwind init failures properly
- Remove ID alloc helper functions and call IDA interfaces directly instead

v3:
- Remove DLB2_PCI_REG_READ/WRITE macros

v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command

Mike Ximing Chen (20):
  dlb: add skeleton for DLB driver
  dlb: initialize device
  dlb: add resource and device initialization
  dlb: add device ioctl layer and first three ioctls
  dlb: add scheduling domain configuration
  dlb: add domain software reset
  dlb: add low-level register reset operations
  dlb: add runtime power-management support
  dlb: add queue create, reset, get-depth ioctls
  dlb: add register operations for queue management
  dlb: add ioctl to configure ports and query poll mode
  dlb: add register operations for port management
  dlb: add port mmap support
  dlb: add start domain ioctl
  dlb: add queue map, unmap, and pending unmap operations
  dlb: add port map/unmap state machine
  dlb: add static queue map register operations
  dlb: add dynamic queue map register operations
  dlb: add queue unmap register operations
  dlb: queue map/unmap workqueue

 Documentation/misc-devices/dlb.rst            |  259 +
 Documentation/misc-devices/index.rst          |    1 +
 .../userspace-api/ioctl/ioctl-number.rst      |    1 +
 MAINTAINERS                                   |    8 +
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/dlb/Kconfig                      |   18 +
 drivers/misc/dlb/Makefile                     |   11 +
 drivers/misc/dlb/dlb_bitmap.h                 |  210 +
 drivers/misc/dlb/dlb_file.c                   |  149 +
 drivers/misc/dlb/dlb_hw_types.h               |  311 +
 drivers/misc/dlb/dlb_ioctl.c                  |  498 ++
 drivers/misc/dlb/dlb_main.c                   |  614 ++
 drivers/misc/dlb/dlb_main.h                   |  178 +
 drivers/misc/dlb/dlb_pf_ops.c                 |  277 +
 drivers/misc/dlb/dlb_regs.h                   | 3640 +++++++++++
 drivers/misc/dlb/dlb_resource.c               | 5469 +++++++++++++++++
 drivers/misc/dlb/dlb_resource.h               |   94 +
 include/uapi/linux/dlb.h                      |  602 ++
 19 files changed, 12342 insertions(+)
 create mode 100644 Documentation/misc-devices/dlb.rst
 create mode 100644 drivers/misc/dlb/Kconfig
 create mode 100644 drivers/misc/dlb/Makefile
 create mode 100644 drivers/misc/dlb/dlb_bitmap.h
 create mode 100644 drivers/misc/dlb/dlb_file.c
 create mode 100644 drivers/misc/dlb/dlb_hw_types.h
 create mode 100644 drivers/misc/dlb/dlb_ioctl.c
 create mode 100644 drivers/misc/dlb/dlb_main.c
 create mode 100644 drivers/misc/dlb/dlb_main.h
 create mode 100644 drivers/misc/dlb/dlb_pf_ops.c
 create mode 100644 drivers/misc/dlb/dlb_regs.h
 create mode 100644 drivers/misc/dlb/dlb_resource.c
 create mode 100644 drivers/misc/dlb/dlb_resource.h
 create mode 100644 include/uapi/linux/dlb.h


base-commit: e71ba9452f0b5b2e8dc8aa5445198cd9214a6a62
-- 
2.17.1


^ permalink raw reply	[flat|nested] 50+ messages in thread

* [PATCH v10 01/20] dlb: add skeleton for DLB driver
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-18  7:34   ` Chen, Mike Ximing
  2021-02-10 17:54 ` [PATCH v10 02/20] dlb: initialize device Mike Ximing Chen
                   ` (19 subsequent siblings)
  20 siblings, 1 reply; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add basic driver functionality (load, unload, probe, and remove callbacks)
for the DLB driver.

Add documentation which describes in detail the hardware, the user
interface, device interrupts, and the driver's power-management strategy.
For more details about the driver see the documentation in the patch.

Add a DLB entry to the MAINTAINERS file.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
 Documentation/misc-devices/index.rst |   1 +
 MAINTAINERS                          |   8 +
 drivers/misc/Kconfig                 |   1 +
 drivers/misc/Makefile                |   1 +
 drivers/misc/dlb/Kconfig             |  18 ++
 drivers/misc/dlb/Makefile            |   9 +
 drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
 drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
 drivers/misc/dlb/dlb_main.h          |  37 ++++
 10 files changed, 522 insertions(+)
 create mode 100644 Documentation/misc-devices/dlb.rst
 create mode 100644 drivers/misc/dlb/Kconfig
 create mode 100644 drivers/misc/dlb/Makefile
 create mode 100644 drivers/misc/dlb/dlb_hw_types.h
 create mode 100644 drivers/misc/dlb/dlb_main.c
 create mode 100644 drivers/misc/dlb/dlb_main.h

diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst
new file mode 100644
index 000000000000..aa79be07ee49
--- /dev/null
+++ b/Documentation/misc-devices/dlb.rst
@@ -0,0 +1,259 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+===========================================
+Intel(R) Dynamic Load Balancer Overview
+===========================================
+
+:Authors: Gage Eads and Mike Ximing Chen
+
+Contents
+========
+
+- Introduction
+- Scheduling
+- Queue Entry
+- Port
+- Queue
+- Credits
+- Scheduling Domain
+- Interrupts
+- Power Management
+- User Interface
+- Reset
+
+Introduction
+============
+
+The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
+provides load-balanced, prioritized scheduling of core-to-core communication.
+
+Intel DLB is an accelerator for the event-driven programming model of
+DPDK's Event Device Library[2]. The library is used in packet processing
+pipelines that arrange for multi-core scalability, dynamic load-balancing, and
+variety of packet distribution and synchronization schemes.
+
+Intel DLB device consists of queues and arbiters that connect producer
+cores and consumer cores. The device implements load-balanced queueing features
+including:
+- Lock-free multi-producer/multi-consumer operation.
+- Multiple priority levels for varying traffic types.
+- 'Direct' traffic (i.e. multi-producer/single-consumer)
+- Simple unordered load-balanced distribution.
+- Atomic lock free load balancing across multiple consumers.
+- Queue element reordering feature allowing ordered load-balanced distribution.
+
+Note: this document uses 'DLB' when discussing the device hardware and 'dlb' when
+discussing the driver implementation.
+
+Following diagram illustrates the functional blocks of an Intel DLB device.
+
+                                       +----+
+                                       |    |
+                        +----------+   |    |   +-------+
+                       /|   IQ     |---|----|--/|       |
+                      / +----------+   |    | / |  CP   |
+                     /                 |    |/  +-------+
+        +--------+  /                  |    |
+        |        | /    +----------+   |   /|   +-------+
+        |  PP    |------|   IQ     |---|----|---|       |
+        +--------+ \    +----------+   | /  |   |  CP   |
+                    \                  |/   |   +-------+
+           ...       \     ...         |    |
+        +--------+    \               /|    |   +-------+
+        |        |     \+----------+ / |    |   |       |
+        |  PP    |------|   IQ     |/--|----|---|  CP   |
+        +--------+      +----------+   |    |   +-------+
+                                       |    |
+                                       +----+     ...
+PP: Producer Port                        |
+CP: Consumer Port                        |
+IQ: Internal Queue                   DLB Scheduler
+
+
+Scheduling Types
+================
+
+Intel DLB supports four types of scheduling of 'events' (using DPDK
+terminology), where an event can represent any type of data (e.g. a network
+packet). The first, ``directed``, is multi-producer/single-consumer style
+core-to-core communication. The remaining three are
+multi-producer/multi-consumer, and support load-balancing across the consumers.
+
+- ``Directed``: events are scheduled to a single consumer.
+
+- ``Unordered``: events are load-balanced across consumers without any ordering
+                 guarantees.
+
+- ``Ordered``: events are load-balanced across consumers, and the consumer can
+               re-enqueue its events so the device re-orders them into the
+               original order. This scheduling type allows software to
+               parallelize ordered event processing without the synchronization
+               cost of re-ordering packets.
+
+- ``Atomic``: events are load-balanced across consumers, with the guarantee that
+              events from a particular 'flow' are only scheduled to a single
+              consumer at a time (but can migrate over time). This allows, for
+              example, packet processing applications to parallelize while
+              avoiding locks on per-flow data and maintaining ordering within a
+              flow.
+
+Intel DLB provides hierarchical priority scheduling, with eight priority
+levels within each. Each consumer selects up to eight queues to receive events
+from, and assigns a priority to each of these 'connected' queues. To schedule
+an event to a consumer, the device selects the highest priority non-empty queue
+of the (up to) eight connected queues. Within that queue, the device selects
+the highest priority event available (selecting a lower priority event for
+starvation avoidance 1% of the time, by default).
+
+The device also supports four load-balanced scheduler classes of service. Each
+class of service receives a (user-configurable) guaranteed percentage of the
+scheduler bandwidth, and any unreserved bandwidth is divided evenly among the
+four classes.
+
+Queue Entry
+===========
+
+Each event is contained in a queue entry (QE), the fundamental unit of
+communication through the device, which consists of 8B of data and 8B of
+metadata, as depicted below.
+
+QE structure format
+::
+    data     :64
+    opaque   :16
+    qid      :8
+    sched    :2
+    priority :3
+    msg_type :3
+    lock_id  :16
+    rsvd     :8
+    cmd      :8
+
+The ``data`` field can be any type that fits within 8B (pointer, integer,
+etc.); DLB merely copies this field from producer to consumer. The
+``opaque`` and ``msg_type`` fields behave the same way.
+
+``qid`` is set by the producer to specify to which DLB 2.0 queue it wishes to
+enqueue this QE. The ID spaces for load-balanced and directed queues are both
+zero-based.
+
+``sched`` controls the scheduling type: atomic, unordered, ordered, or
+directed. The first three scheduling types are only valid for load-balanced
+queues, and the directed scheduling type is only valid for directed queues.
+This field distinguishes whether ``qid`` is load-balanced or directed, since
+their ID spaces overlap.
+
+``priority`` is the priority with which this QE should be scheduled.
+
+``lock_id``, used for atomic scheduling and ignored for ordered and unordered
+scheduling, identifies the atomic flow to which the QE belongs. When sending a
+directed event, ``lock_id`` is simply copied like the ``data``, ``opaque``, and
+``msg_type`` fields.
+
+``cmd`` specifies the operation, such as:
+- Enqueue a new QE
+- Forward a QE that was dequeued
+- Complete/terminate a QE that was dequeued
+- Return one or more consumer queue tokens.
+- Arm the port's consumer queue interrupt.
+
+Port
+====
+
+A core's interface to the DLB is called a "port," and consists of an MMIO
+region (producer port) through which the core enqueues a queue entry, and an
+in-memory queue (the "consumer queue" or cosumer port) to which the device
+schedules QEs. A core enqueues a QE to a device queue, then the device
+schedules the event to a port. Software specifies the connection of queues
+and ports; i.e. for each queue, to which ports the device is allowed to
+schedule its events. The device uses a credit scheme to prevent overflow of
+the on-device queue storage.
+
+Applications interface directly with the device by mapping the port's memory
+and MMIO regions into the application's address space for enqueue and dequeue
+operations, but call into the kernel driver for configuration operations. An
+application can be polling- or interrupt-driven; DLB supports both modes
+of operation.
+
+Internal Queue
+==============
+
+A DLB device supports an implementation specific and runtime discoverable
+number of load-balanced (i.e. capable of atomic, ordered, and unordered
+scheduling) and directed queues. Each internal queue supports a set of
+priority levels.
+
+A load-balanced queue is capable of scheduling its events to any combination of
+load-balanced ports, whereas each directed queue has one-to-one mapping with a
+directed port. There is no restriction on port or queue types when a port
+enqueues an event to a queue; that is, a load-balanced port can enqueue to a
+directed queue and vice versa.
+
+Credits
+=======
+
+The Intel DLB uses a credit scheme to prevent overflow of the on-device
+queue storage, with separate credits for load-balanced and directed queues. A
+port spends one credit when it enqueues a QE, and one credit is replenished
+when a QE is scheduled to a consumer queue. Each scheduling domain has one pool
+of load-balanced credits and one pool of directed credits; software is
+responsible for managing the allocation and replenishment of these credits among
+the scheduling domain's ports.
+
+Scheduling Domain
+=================
+
+Device resources -- including ports, queues, and credits -- are contained
+within a scheduling domain. Scheduling domains are isolated from one another; a
+port can only enqueue to and dequeue from queues within its scheduling domain.
+
+The scheduling domain creation ioctl returns a domain file descriptor, through
+which the domain's resources are configured. For a multi-process scenario, the
+owner of this descriptor must share it with the other processes (e.g. inherited
+through fork() or shared over a unix domain socket).
+
+Consumer Queue Interrupts
+=========================
+
+Each port has its own interrupt which fires, if armed, when the consumer queue
+depth becomes non-zero. Software arms an interrupt by enqueueing a special
+'interrupt arm' command to the device through the port's MMIO window.
+
+Power Management
+================
+
+The kernel driver keeps the device in D3Hot when not in use. The driver
+transitions the device to D0 when the first device file is opened, and keeps it
+there until there are no open device files or memory mappings.
+
+User Interface
+==============
+
+The dlb driver uses ioctls as its primary interface. It provides two types of
+files: the dlb device file and the scheduling domain file.
+
+The two types support different ioctl interfaces; the dlb device file is used
+for device-wide operations (including scheduling domain creation), and the
+scheduling domain device file supports operations on the scheduling domain's
+resources such as port and queue configuration.
+
+The dlb device file is created during driver probe and is located at
+/dev/dlb<N>, where N is the zero-based device ID. The scheduling domain fd is
+an anonymous inode created by a dlb device ioctl.
+
+The driver also exports an mmap interface through port files, which are
+acquired through scheduling domain ioctls. This mmap interface is used to map
+a port's memory and MMIO window into the process's address space. Once the
+ports are mapped, applications may use 64-byte direct-store instructions such
+movdir64b or enqcmd to enqueue the events for better performance.
+
+Reset
+=====
+
+The dlb driver currently supports scheduling domain reset.
+
+Scheduling domain reset occurs when an application stops using its domain.
+Specifically, when no more file references or memory mappings exist. At this
+time, the driver resets all the domain's resources (flushes its queues and
+ports) and puts them in their respective available-resource lists for later
+use.
diff --git a/Documentation/misc-devices/index.rst b/Documentation/misc-devices/index.rst
index 64420b3314fe..c9954e8b72a3 100644
--- a/Documentation/misc-devices/index.rst
+++ b/Documentation/misc-devices/index.rst
@@ -17,6 +17,7 @@ fit into other categories.
    ad525x_dpot
    apds990x
    bh1770glc
+   dlb
    eeprom
    c2port
    ibmvmc
diff --git a/MAINTAINERS b/MAINTAINERS
index 6eff4f720c72..b408c1ec8fdf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8896,6 +8896,14 @@ L:	linux-kernel@vger.kernel.org
 S:	Supported
 F:	arch/x86/include/asm/intel-family.h
 
+INTEL DYNAMIC LOAD BALANCER DRIVER
+M:	Mike Ximing Chen <mike.ximing.chen@intel.com>
+M:	Gage Eads <gage.eads@intel.com>
+S:	Maintained
+F:	Documentation/ABI/testing/sysfs-driver-dlb
+F:	drivers/misc/dlb/
+F:	include/uapi/linux/dlb_user.h
+
 INTEL DRM DRIVERS (excluding Poulsbo, Moorestown and derivative chipsets)
 M:	Jani Nikula <jani.nikula@linux.intel.com>
 M:	Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index fafa8b0d8099..fef26819eb1e 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -481,4 +481,5 @@ source "drivers/misc/ocxl/Kconfig"
 source "drivers/misc/cardreader/Kconfig"
 source "drivers/misc/habanalabs/Kconfig"
 source "drivers/misc/uacce/Kconfig"
+source "drivers/misc/dlb/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index d23231e73330..a0bafe336277 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_HABANA_AI)		+= habanalabs/
 obj-$(CONFIG_UACCE)		+= uacce/
 obj-$(CONFIG_XILINX_SDFEC)	+= xilinx_sdfec.o
 obj-$(CONFIG_HISI_HIKEY_USB)	+= hisi_hikey_usb.o
+obj-$(CONFIG_INTEL_DLB)		+= dlb/
diff --git a/drivers/misc/dlb/Kconfig b/drivers/misc/dlb/Kconfig
new file mode 100644
index 000000000000..cfa978c705bd
--- /dev/null
+++ b/drivers/misc/dlb/Kconfig
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config INTEL_DLB
+	tristate "Intel Dynamic Load Balancer Driver"
+	depends on 64BIT && PCI && X86
+	help
+	  This driver supports the Intel Dynamic Load Balancer (DLB), a
+	  PCIe device (PCI ID 8086:27xx) that provides load-balanced,
+	  prioritized scheduling of core-to-core communication and improves
+	  DPDK Event Device library performance.
+
+	  The user-space interface is described in
+	  include/uapi/linux/dlb_user.h
+
+	  To compile this driver as a module, choose M here. The module
+	  will be called dlb.
+
+	  If unsure, select N.
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
new file mode 100644
index 000000000000..8911375effd2
--- /dev/null
+++ b/drivers/misc/dlb/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+#
+# Makefile for the Intel(R) Dynamic Load Balancer (dlb.ko) driver
+#
+
+obj-$(CONFIG_INTEL_DLB) := dlb.o
+
+dlb-objs := dlb_main.o
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
new file mode 100644
index 000000000000..778ec8665ea0
--- /dev/null
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_HW_TYPES_H
+#define __DLB_HW_TYPES_H
+
+#define DLB_MAX_NUM_VDEVS			16
+#define DLB_MAX_NUM_DOMAINS			32
+#define DLB_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB_MAX_NUM_LDB_PORTS			64
+#define DLB_MAX_NUM_DIR_PORTS			DLB_MAX_NUM_DIR_QUEUES
+#define DLB_MAX_NUM_LDB_CREDITS			8192
+#define DLB_MAX_NUM_DIR_CREDITS			2048
+#define DLB_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB_MAX_NUM_AQED_ENTRIES		2048
+#define DLB_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB_QID_PRIORITIES			8
+#define DLB_NUM_ARB_WEIGHTS			8
+#define DLB_MAX_WEIGHT				255
+#define DLB_NUM_COS_DOMAINS			4
+#define DLB_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#define DLB_HZ					800000000
+#define DLB_FUNC_BAR				0
+#define DLB_CSR_BAR				2
+
+#define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
+
+#endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
new file mode 100644
index 000000000000..fbd77203b398
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.c
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/aer.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include "dlb_main.h"
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
+
+static struct class *dlb_class;
+static dev_t dlb_devt;
+static DEFINE_IDR(dlb_ids);
+static DEFINE_SPINLOCK(dlb_ids_lock);
+
+/**********************************/
+/****** PCI driver callbacks ******/
+/**********************************/
+
+static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
+{
+	struct dlb *dlb;
+	int ret;
+
+	dlb = devm_kzalloc(&pdev->dev, sizeof(*dlb), GFP_KERNEL);
+	if (!dlb)
+		return -ENOMEM;
+
+	pci_set_drvdata(pdev, dlb);
+
+	dlb->pdev = pdev;
+
+	spin_lock(&dlb_ids_lock);
+	dlb->id = idr_alloc(&dlb_ids, (void *)dlb, 0, DLB_MAX_NUM_DEVICES - 1,
+			    GFP_KERNEL);
+	spin_unlock(&dlb_ids_lock);
+
+	if (dlb->id < 0) {
+		dev_err(&pdev->dev, "probe: device ID allocation failed\n");
+
+		ret = dlb->id;
+		goto alloc_id_fail;
+	}
+
+	ret = pcim_enable_device(pdev);
+	if (ret != 0) {
+		dev_err(&pdev->dev, "pcim_enable_device() returned %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	ret = pcim_iomap_regions(pdev,
+				 (1U << DLB_CSR_BAR) | (1U << DLB_FUNC_BAR),
+				 "dlb");
+	if (ret != 0) {
+		dev_err(&pdev->dev, "pcim_iomap_regions(): returned %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	pci_set_master(pdev);
+
+	ret = pci_enable_pcie_error_reporting(pdev);
+	if (ret != 0)
+		dev_info(&pdev->dev, "Failed to enable AER %d\n", ret);
+
+	return 0;
+
+pci_enable_device_fail:
+	spin_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	spin_unlock(&dlb_ids_lock);
+alloc_id_fail:
+	return ret;
+}
+
+static void dlb_remove(struct pci_dev *pdev)
+{
+	struct dlb *dlb = pci_get_drvdata(pdev);
+
+	pci_disable_pcie_error_reporting(pdev);
+
+	spin_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	spin_unlock(&dlb_ids_lock);
+}
+
+static struct pci_device_id dlb_id_table[] = {
+	{ PCI_DEVICE_DATA(INTEL, DLB_PF, DLB_PF) },
+	{ 0 }
+};
+MODULE_DEVICE_TABLE(pci, dlb_id_table);
+
+static struct pci_driver dlb_pci_driver = {
+	.name		 = "dlb",
+	.id_table	 = dlb_id_table,
+	.probe		 = dlb_probe,
+	.remove		 = dlb_remove,
+};
+
+static int __init dlb_init_module(void)
+{
+	int err;
+
+	dlb_class = class_create(THIS_MODULE, "dlb");
+
+	if (IS_ERR(dlb_class)) {
+		pr_err("dlb: class_create() returned %ld\n",
+		       PTR_ERR(dlb_class));
+
+		return PTR_ERR(dlb_class);
+	}
+
+	err = alloc_chrdev_region(&dlb_devt, 0, DLB_MAX_NUM_DEVICES, "dlb");
+
+	if (err < 0) {
+		pr_err("dlb: alloc_chrdev_region() returned %d\n", err);
+
+		goto alloc_chrdev_fail;
+	}
+
+	err = pci_register_driver(&dlb_pci_driver);
+	if (err < 0) {
+		pr_err("dlb: pci_register_driver() returned %d\n", err);
+
+		goto pci_register_fail;
+	}
+
+	return 0;
+
+pci_register_fail:
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+alloc_chrdev_fail:
+	class_destroy(dlb_class);
+
+	return err;
+}
+
+static void __exit dlb_exit_module(void)
+{
+	pci_unregister_driver(&dlb_pci_driver);
+
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+
+	class_destroy(dlb_class);
+}
+
+module_init(dlb_init_module);
+module_exit(dlb_exit_module);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
new file mode 100644
index 000000000000..ab2a2014bafd
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_MAIN_H
+#define __DLB_MAIN_H
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+#include "dlb_hw_types.h"
+
+/*
+ * The dlb driver uses a different minor number for each device file, of which
+ * there are:
+ * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
+ * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
+ * - Up to 16 PFs per system
+ */
+#define DLB_MAX_NUM_PFS	  16
+#define DLB_NUM_FUNCS_PER_DEVICE (1 + DLB_MAX_NUM_VDEVS)
+#define DLB_MAX_NUM_DEVICES	 (DLB_MAX_NUM_PFS * DLB_NUM_FUNCS_PER_DEVICE)
+
+enum dlb_device_type {
+	DLB_PF,
+};
+
+struct dlb {
+	struct pci_dev *pdev;
+	int id;
+};
+
+#endif /* __DLB_MAIN_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 02/20] dlb: initialize device
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 01/20] dlb: add skeleton for DLB driver Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 03/20] dlb: add resource and device initialization Mike Ximing Chen
                   ` (18 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Assign the physical function device 'ops' callbacks, map the PCI BAR space,
create a char device, set the DMA API mask for 64-bit addressing. Add the
corresponding undo operations to the remove callback.

The ops callbacks are implement behavior that differs depending on device
type -- physical function (PF, as implemented here) or virtual
function/device (support to be added later).

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/Makefile       |  1 +
 drivers/misc/dlb/dlb_hw_types.h | 23 +++++++++++++
 drivers/misc/dlb/dlb_main.c     | 59 +++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_main.h     | 14 ++++++++
 drivers/misc/dlb/dlb_pf_ops.c   | 50 ++++++++++++++++++++++++++++
 5 files changed, 147 insertions(+)
 create mode 100644 drivers/misc/dlb/dlb_pf_ops.c

diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index 8911375effd2..a33bf774e6a8 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -7,3 +7,4 @@
 obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
+dlb-objs += dlb_pf_ops.o
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index 778ec8665ea0..a4ce28c157de 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -4,6 +4,20 @@
 #ifndef __DLB_HW_TYPES_H
 #define __DLB_HW_TYPES_H
 
+#include <linux/io.h>
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB_CSR_REG_ADDR(a, reg)   ((a)->csr_kva + (reg))
+#define DLB_CSR_RD(hw, reg)	    ioread32(DLB_CSR_REG_ADDR((hw), (reg)))
+#define DLB_CSR_WR(hw, reg, value) iowrite32((value), \
+					      DLB_CSR_REG_ADDR((hw), (reg)))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB_FUNC_REG_ADDR(a, reg)   ((a)->func_kva + (reg))
+#define DLB_FUNC_RD(hw, reg)	     ioread32(DLB_FUNC_REG_ADDR((hw), (reg)))
+#define DLB_FUNC_WR(hw, reg, value) iowrite32((value), \
+					       DLB_FUNC_REG_ADDR((hw), (reg)))
+
 #define DLB_MAX_NUM_VDEVS			16
 #define DLB_MAX_NUM_DOMAINS			32
 #define DLB_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
@@ -29,4 +43,13 @@
 
 #define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
 
+struct dlb_hw {
+	/* BAR 0 address */
+	void __iomem *csr_kva;
+	unsigned long csr_phys_addr;
+	/* BAR 2 address */
+	void __iomem *func_kva;
+	unsigned long func_phys_addr;
+};
+
 #endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index fbd77203b398..7fb6e9c360c8 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -16,10 +16,38 @@ MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
 
 static struct class *dlb_class;
+static struct cdev dlb_cdev;
 static dev_t dlb_devt;
 static DEFINE_IDR(dlb_ids);
 static DEFINE_SPINLOCK(dlb_ids_lock);
 
+static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
+{
+	/*
+	 * Create a new device in order to create a /dev/dlb node. This device
+	 * is a child of the DLB PCI device.
+	 */
+	dlb->dev_number = MKDEV(MAJOR(dlb_devt), dlb->id);
+	dlb->dev = device_create(dlb_class, &pdev->dev, dlb->dev_number, dlb,
+				 "dlb%d", dlb->id);
+	if (IS_ERR(dlb->dev)) {
+		dev_err(dlb->dev, "device_create() returned %ld\n",
+			PTR_ERR(dlb->dev));
+
+		return PTR_ERR(dlb->dev);
+	}
+
+	return 0;
+}
+
+/********************************/
+/****** Char dev callbacks ******/
+/********************************/
+
+static const struct file_operations dlb_fops = {
+	.owner   = THIS_MODULE,
+};
+
 /**********************************/
 /****** PCI driver callbacks ******/
 /**********************************/
@@ -33,6 +61,8 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (!dlb)
 		return -ENOMEM;
 
+	dlb->ops = &dlb_pf_ops;
+
 	pci_set_drvdata(pdev, dlb);
 
 	dlb->pdev = pdev;
@@ -71,8 +101,24 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret != 0)
 		dev_info(&pdev->dev, "Failed to enable AER %d\n", ret);
 
+	ret = dlb->ops->map_pci_bar_space(dlb, pdev);
+	if (ret)
+		goto map_pci_bar_fail;
+
+	ret = dlb_device_create(dlb, pdev);
+	if (ret)
+		goto map_pci_bar_fail;
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret)
+		goto dma_set_mask_fail;
+
 	return 0;
 
+dma_set_mask_fail:
+	device_destroy(dlb_class, dlb->dev_number);
+map_pci_bar_fail:
+	pci_disable_pcie_error_reporting(pdev);
 pci_enable_device_fail:
 	spin_lock(&dlb_ids_lock);
 	idr_remove(&dlb_ids, dlb->id);
@@ -85,6 +131,8 @@ static void dlb_remove(struct pci_dev *pdev)
 {
 	struct dlb *dlb = pci_get_drvdata(pdev);
 
+	device_destroy(dlb_class, dlb->dev_number);
+
 	pci_disable_pcie_error_reporting(pdev);
 
 	spin_lock(&dlb_ids_lock);
@@ -107,6 +155,7 @@ static struct pci_driver dlb_pci_driver = {
 
 static int __init dlb_init_module(void)
 {
+	int dlb_major;
 	int err;
 
 	dlb_class = class_create(THIS_MODULE, "dlb");
@@ -126,6 +175,12 @@ static int __init dlb_init_module(void)
 		goto alloc_chrdev_fail;
 	}
 
+	dlb_major = MAJOR(dlb_devt);
+	cdev_init(&dlb_cdev, &dlb_fops);
+	err = cdev_add(&dlb_cdev, MKDEV(dlb_major, 0), DLB_MAX_NUM_DEVICES);
+	if (err)
+		goto cdev_add_fail;
+
 	err = pci_register_driver(&dlb_pci_driver);
 	if (err < 0) {
 		pr_err("dlb: pci_register_driver() returned %d\n", err);
@@ -136,6 +191,8 @@ static int __init dlb_init_module(void)
 	return 0;
 
 pci_register_fail:
+	cdev_del(&dlb_cdev);
+cdev_add_fail:
 	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
 alloc_chrdev_fail:
 	class_destroy(dlb_class);
@@ -147,6 +204,8 @@ static void __exit dlb_exit_module(void)
 {
 	pci_unregister_driver(&dlb_pci_driver);
 
+	cdev_del(&dlb_cdev);
+
 	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
 
 	class_destroy(dlb_class);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index ab2a2014bafd..21570b206419 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -29,9 +29,23 @@ enum dlb_device_type {
 	DLB_PF,
 };
 
+struct dlb;
+
+struct dlb_device_ops {
+	int (*map_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
+	void (*unmap_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
+};
+
+extern struct dlb_device_ops dlb_pf_ops;
+
 struct dlb {
 	struct pci_dev *pdev;
+	struct dlb_hw hw;
+	struct dlb_device_ops *ops;
+	struct device *dev;
+	enum dlb_device_type type;
 	int id;
+	dev_t dev_number;
 };
 
 #endif /* __DLB_MAIN_H */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
new file mode 100644
index 000000000000..0951c99f6183
--- /dev/null
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -0,0 +1,50 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include "dlb_main.h"
+
+/********************************/
+/****** PCI BAR management ******/
+/********************************/
+
+static void
+dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+{
+	pcim_iounmap(pdev, dlb->hw.csr_kva);
+	pcim_iounmap(pdev, dlb->hw.func_kva);
+}
+
+static int
+dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+{
+	dlb->hw.func_kva = pcim_iomap_table(pdev)[DLB_FUNC_BAR];
+	dlb->hw.func_phys_addr = pci_resource_start(pdev, DLB_FUNC_BAR);
+
+	if (!dlb->hw.func_kva) {
+		dev_err(&pdev->dev, "Cannot iomap BAR 0 (size %llu)\n",
+			pci_resource_len(pdev, 0));
+
+		return -EIO;
+	}
+
+	dlb->hw.csr_kva = pcim_iomap_table(pdev)[DLB_CSR_BAR];
+	dlb->hw.csr_phys_addr = pci_resource_start(pdev, DLB_CSR_BAR);
+
+	if (!dlb->hw.csr_kva) {
+		dev_err(&pdev->dev, "Cannot iomap BAR 2 (size %llu)\n",
+			pci_resource_len(pdev, 2));
+
+		return -EIO;
+	}
+
+	return 0;
+}
+
+/********************************/
+/****** DLB PF Device Ops ******/
+/********************************/
+
+struct dlb_device_ops dlb_pf_ops = {
+	.map_pci_bar_space = dlb_pf_map_pci_bar_space,
+	.unmap_pci_bar_space = dlb_pf_unmap_pci_bar_space,
+};
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 03/20] dlb: add resource and device initialization
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 01/20] dlb: add skeleton for DLB driver Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 02/20] dlb: initialize device Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-03-09  9:24   ` Greg KH
  2021-02-10 17:54 ` [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls Mike Ximing Chen
                   ` (17 subsequent siblings)
  20 siblings, 1 reply; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the hardware resource data structures, functions for
their initialization/teardown, and a function for device power-on. In
subsequent commits, dlb_resource.c will be expanded to hold the dlb
resource-management and configuration logic (using the data structures
defined in dlb_hw_types.h).

Introduce dlb_bitmap_* functions, a thin convenience layer wrapping the
Linux bitmap interfaces, used by the bitmaps in the dlb hardware types.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/Makefile       |   2 +-
 drivers/misc/dlb/dlb_bitmap.h   |  76 +++++++++++
 drivers/misc/dlb/dlb_hw_types.h | 177 +++++++++++++++++++++++++
 drivers/misc/dlb/dlb_main.c     |  47 +++++++
 drivers/misc/dlb/dlb_main.h     |   8 ++
 drivers/misc/dlb/dlb_pf_ops.c   |  66 +++++++++-
 drivers/misc/dlb/dlb_regs.h     | 119 +++++++++++++++++
 drivers/misc/dlb/dlb_resource.c | 220 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |  17 +++
 9 files changed, 727 insertions(+), 5 deletions(-)
 create mode 100644 drivers/misc/dlb/dlb_bitmap.h
 create mode 100644 drivers/misc/dlb/dlb_regs.h
 create mode 100644 drivers/misc/dlb/dlb_resource.c
 create mode 100644 drivers/misc/dlb/dlb_resource.h

diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index a33bf774e6a8..8a49ea5fd752 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -7,4 +7,4 @@
 obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
-dlb-objs += dlb_pf_ops.o
+dlb-objs += dlb_pf_ops.o dlb_resource.o
diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
new file mode 100644
index 000000000000..fb3ef52a306d
--- /dev/null
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -0,0 +1,76 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_OSDEP_BITMAP_H
+#define __DLB_OSDEP_BITMAP_H
+
+#include <linux/bitmap.h>
+#include <linux/slab.h>
+
+#include "dlb_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb_bitmap {
+	unsigned long *map;
+	unsigned int len;
+};
+
+/**
+ * dlb_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb_bitmap_alloc(struct dlb_bitmap **bitmap,
+				   unsigned int len)
+{
+	struct dlb_bitmap *bm;
+
+	if (!bitmap || len == 0)
+		return -EINVAL;
+
+	bm = kzalloc(sizeof(*bm), GFP_KERNEL);
+	if (!bm)
+		return -ENOMEM;
+
+	bm->map = bitmap_zalloc(len, GFP_KERNEL);
+	if (!bm->map) {
+		kfree(bm);
+		return -ENOMEM;
+	}
+
+	bm->len = len;
+
+	*bitmap = bm;
+
+	return 0;
+}
+
+/**
+ * dlb_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb_bitmap_alloc().
+ */
+static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
+{
+	if (!bitmap)
+		return;
+
+	bitmap_free(bitmap->map);
+
+	kfree(bitmap);
+}
+
+#endif /*  __DLB_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index a4ce28c157de..3e03b061d5ff 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -5,6 +5,13 @@
 #define __DLB_HW_TYPES_H
 
 #include <linux/io.h>
+#include <linux/types.h>
+
+#include "dlb_bitmap.h"
+
+#define BITS_SET(x, val, mask)	(x = ((x) & ~(mask))     \
+				 | (((val) << (mask##_LOC)) & (mask)))
+#define BITS_GET(x, mask)       (((x) & (mask)) >> (mask##_LOC))
 
 /* Read/write register 'reg' in the CSR BAR space */
 #define DLB_CSR_REG_ADDR(a, reg)   ((a)->csr_kva + (reg))
@@ -43,6 +50,169 @@
 
 #define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
 
+struct dlb_resource_id {
+	u32 phys_id;
+	u32 virt_id;
+	u8 vdev_owned;
+	u8 vdev_id;
+};
+
+struct dlb_freelist {
+	u32 base;
+	u32 bound;
+	u32 offset;
+};
+
+static inline u32 dlb_freelist_count(struct dlb_freelist *list)
+{
+	return list->bound - list->base - list->offset;
+}
+
+struct dlb_ldb_queue {
+	struct list_head domain_list;
+	struct list_head func_list;
+	struct dlb_resource_id id;
+	struct dlb_resource_id domain_id;
+	u32 num_qid_inflights;
+	u32 aqed_limit;
+	u32 sn_group; /* sn == sequence number */
+	u32 sn_slot;
+	u32 num_mappings;
+	u8 sn_cfg_valid;
+	u8 num_pending_additions;
+	u8 owned;
+	u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb_dir_pq_pair {
+	struct list_head domain_list;
+	struct list_head func_list;
+	struct dlb_resource_id id;
+	struct dlb_resource_id domain_id;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 queue_configured;
+	u8 port_configured;
+	u8 owned;
+	u8 enabled;
+};
+
+enum dlb_qid_map_state {
+	/* The slot doesn't contain a valid queue mapping */
+	DLB_QUEUE_UNMAPPED,
+	/* The slot contains a valid queue mapping */
+	DLB_QUEUE_MAPPED,
+	/* The driver is mapping a queue into this slot */
+	DLB_QUEUE_MAP_IN_PROG,
+	/* The driver is unmapping a queue from this slot */
+	DLB_QUEUE_UNMAP_IN_PROG,
+	/*
+	 * The driver is unmapping a queue from this slot, and once complete
+	 * will replace it with another mapping.
+	 */
+	DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb_ldb_port_qid_map {
+	enum dlb_qid_map_state state;
+	u16 qid;
+	u16 pending_qid;
+	u8 priority;
+	u8 pending_priority;
+};
+
+struct dlb_ldb_port {
+	struct list_head domain_list;
+	struct list_head func_list;
+	struct dlb_resource_id id;
+	struct dlb_resource_id domain_id;
+	/* The qid_map represents the hardware QID mapping state. */
+	struct dlb_ldb_port_qid_map qid_map[DLB_MAX_NUM_QIDS_PER_LDB_CQ];
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_limit;
+	u32 ref_cnt;
+	u8 init_tkn_cnt;
+	u8 num_pending_removals;
+	u8 num_mappings;
+	u8 owned;
+	u8 enabled;
+	u8 configured;
+};
+
+struct dlb_sn_group {
+	u32 mode;
+	u32 sequence_numbers_per_queue;
+	u32 slot_use_bitmap;
+	u32 id;
+};
+
+struct dlb_hw_domain {
+	struct dlb_function_resources *parent_func;
+	struct list_head func_list;
+	struct list_head used_ldb_queues;
+	struct list_head used_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head used_dir_pq_pairs;
+	struct list_head avail_ldb_queues;
+	struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head avail_dir_pq_pairs;
+	u32 total_hist_list_entries;
+	u32 avail_hist_list_entries;
+	u32 hist_list_entry_base;
+	u32 hist_list_entry_offset;
+	u32 num_ldb_credits;
+	u32 num_dir_credits;
+	u32 num_avail_aqed_entries;
+	u32 num_used_aqed_entries;
+	struct dlb_resource_id id;
+	int num_pending_removals;
+	int num_pending_additions;
+	u8 configured;
+	u8 started;
+};
+
+struct dlb_function_resources {
+	struct list_head avail_domains;
+	struct list_head used_domains;
+	struct list_head avail_ldb_queues;
+	struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	struct list_head avail_dir_pq_pairs;
+	struct dlb_bitmap *avail_hist_list_entries;
+	u32 num_avail_domains;
+	u32 num_avail_ldb_queues;
+	u32 num_avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+	u32 num_avail_dir_pq_pairs;
+	u32 num_avail_qed_entries;
+	u32 num_avail_dqed_entries;
+	u32 num_avail_aqed_entries;
+	u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ *	by the PF and not allocated to a dlb scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ *	resources not allocated to a dlb scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ *	resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ *	resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb_hw_resources {
+	struct dlb_ldb_queue ldb_queues[DLB_MAX_NUM_LDB_QUEUES];
+	struct dlb_ldb_port ldb_ports[DLB_MAX_NUM_LDB_PORTS];
+	struct dlb_dir_pq_pair dir_pq_pairs[DLB_MAX_NUM_DIR_PORTS];
+	struct dlb_sn_group sn_groups[DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
 struct dlb_hw {
 	/* BAR 0 address */
 	void __iomem *csr_kva;
@@ -50,6 +220,13 @@ struct dlb_hw {
 	/* BAR 2 address */
 	void __iomem *func_kva;
 	unsigned long func_phys_addr;
+
+	/* Resource tracking */
+	struct dlb_hw_resources rsrcs;
+	struct dlb_function_resources pf;
+	struct dlb_function_resources vdev[DLB_MAX_NUM_VDEVS];
+	struct dlb_hw_domain domains[DLB_MAX_NUM_DOMAINS];
+	u8 cos_reservation[DLB_NUM_COS_DOMAINS];
 };
 
 #endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 7fb6e9c360c8..12707b23ab3e 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -11,6 +11,7 @@
 #include <linux/uaccess.h>
 
 #include "dlb_main.h"
+#include "dlb_resource.h"
 
 MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
@@ -21,6 +22,23 @@ static dev_t dlb_devt;
 static DEFINE_IDR(dlb_ids);
 static DEFINE_SPINLOCK(dlb_ids_lock);
 
+static int dlb_reset_device(struct pci_dev *pdev)
+{
+	int ret;
+
+	ret = pci_save_state(pdev);
+	if (ret)
+		return ret;
+
+	ret = __pci_reset_function_locked(pdev);
+	if (ret)
+		return ret;
+
+	pci_restore_state(pdev);
+
+	return 0;
+}
+
 static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
 {
 	/*
@@ -113,8 +131,35 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret)
 		goto dma_set_mask_fail;
 
+	/*
+	 * PM enable must be done before any other MMIO accesses, and this
+	 * setting is persistent across device reset.
+	 */
+	dlb->ops->enable_pm(dlb);
+
+	ret = dlb->ops->wait_for_device_ready(dlb, pdev);
+	if (ret)
+		goto wait_for_device_ready_fail;
+
+	ret = dlb_reset_device(pdev);
+	if (ret)
+		goto dlb_reset_fail;
+
+	ret = dlb_resource_init(&dlb->hw);
+	if (ret)
+		goto resource_init_fail;
+
+	ret = dlb->ops->init_driver_state(dlb);
+	if (ret)
+		goto init_driver_state_fail;
+
 	return 0;
 
+init_driver_state_fail:
+	dlb_resource_free(&dlb->hw);
+resource_init_fail:
+dlb_reset_fail:
+wait_for_device_ready_fail:
 dma_set_mask_fail:
 	device_destroy(dlb_class, dlb->dev_number);
 map_pci_bar_fail:
@@ -131,6 +176,8 @@ static void dlb_remove(struct pci_dev *pdev)
 {
 	struct dlb *dlb = pci_get_drvdata(pdev);
 
+	dlb_resource_free(&dlb->hw);
+
 	device_destroy(dlb_class, dlb->dev_number);
 
 	pci_disable_pcie_error_reporting(pdev);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 21570b206419..ec5eb7bd8f54 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -34,6 +34,9 @@ struct dlb;
 struct dlb_device_ops {
 	int (*map_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
 	void (*unmap_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
+	int (*init_driver_state)(struct dlb *dlb);
+	void (*enable_pm)(struct dlb *dlb);
+	int (*wait_for_device_ready)(struct dlb *dlb, struct pci_dev *pdev);
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
@@ -43,6 +46,11 @@ struct dlb {
 	struct dlb_hw hw;
 	struct dlb_device_ops *ops;
 	struct device *dev;
+	/*
+	 * The resource mutex serializes access to driver data structures and
+	 * hardware registers.
+	 */
+	struct mutex resource_mutex;
 	enum dlb_device_type type;
 	int id;
 	dev_t dev_number;
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 0951c99f6183..124b4fee8564 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -1,21 +1,23 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
 
+#include <linux/delay.h>
+
 #include "dlb_main.h"
+#include "dlb_regs.h"
+#include "dlb_resource.h"
 
 /********************************/
 /****** PCI BAR management ******/
 /********************************/
 
-static void
-dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+static void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
 {
 	pcim_iounmap(pdev, dlb->hw.csr_kva);
 	pcim_iounmap(pdev, dlb->hw.func_kva);
 }
 
-static int
-dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+static int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
 {
 	dlb->hw.func_kva = pcim_iomap_table(pdev)[DLB_FUNC_BAR];
 	dlb->hw.func_phys_addr = pci_resource_start(pdev, DLB_FUNC_BAR);
@@ -40,6 +42,59 @@ dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
 	return 0;
 }
 
+/*******************************/
+/****** Driver management ******/
+/*******************************/
+
+static int dlb_pf_init_driver_state(struct dlb *dlb)
+{
+	mutex_init(&dlb->resource_mutex);
+
+	return 0;
+}
+
+static void dlb_pf_enable_pm(struct dlb *dlb)
+{
+	/*
+	 * Clear the power-management-disable register to power on the bulk of
+	 * the device's hardware.
+	 */
+	dlb_clr_pmcsr_disable(&dlb->hw);
+}
+
+#define DLB_READY_RETRY_LIMIT 1000
+static int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
+{
+	u32 retries = DLB_READY_RETRY_LIMIT;
+
+	/* Allow at least 1s for the device to become active after power-on */
+	do {
+		u32 idle, pm_st, addr;
+
+		addr = CM_CFG_PM_STATUS;
+
+		pm_st = DLB_CSR_RD(&dlb->hw, addr);
+
+		addr = CM_CFG_DIAGNOSTIC_IDLE_STATUS;
+
+		idle = DLB_CSR_RD(&dlb->hw, addr);
+
+		if (BITS_GET(pm_st, CM_CFG_PM_STATUS_PMSM) == 1 &&
+		    BITS_GET(idle, CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE)
+		    == 1)
+			break;
+
+		usleep_range(1000, 2000);
+	} while (--retries);
+
+	if (!retries) {
+		dev_err(&pdev->dev, "Device idle test failed\n");
+		return -EIO;
+	}
+
+	return 0;
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -47,4 +102,7 @@ dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
 struct dlb_device_ops dlb_pf_ops = {
 	.map_pci_bar_space = dlb_pf_map_pci_bar_space,
 	.unmap_pci_bar_space = dlb_pf_unmap_pci_bar_space,
+	.init_driver_state = dlb_pf_init_driver_state,
+	.enable_pm = dlb_pf_enable_pm,
+	.wait_for_device_ready = dlb_pf_wait_for_device_ready,
 };
diff --git a/drivers/misc/dlb/dlb_regs.h b/drivers/misc/dlb/dlb_regs.h
new file mode 100644
index 000000000000..72f3cb22b933
--- /dev/null
+++ b/drivers/misc/dlb/dlb_regs.h
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_REGS_H
+#define __DLB_REGS_H
+
+#include <linux/types.h>
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE		0x00000001
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE		0x00000002
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE		0x00000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE		0x00000008
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE		0x00000010
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE		0x00000020
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE		0x00000040
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE		0x00000080
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE		0x00000100
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE		0x00000200
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE		0x00000400
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE		0x00000800
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE		0x00001000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE		0x00002000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE		0x00004000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE		0x00008000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE		0x00010000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE		0x00020000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE		0x00040000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE		0x00080000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1			0x00F00000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE	0x01000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE	0x02000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B		0x04000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE		0x08000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED	0x10000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0			0x60000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE		0x80000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE_LOC		0
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE_LOC		1
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE_LOC		2
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE_LOC		3
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE_LOC		4
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE_LOC		5
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE_LOC		6
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE_LOC		7
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE_LOC		8
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE_LOC		9
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE_LOC		10
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE_LOC		11
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE_LOC		12
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE_LOC	13
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE_LOC		14
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE_LOC		15
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE_LOC		16
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE_LOC	17
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE_LOC	18
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE_LOC		19
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1_LOC			20
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE_LOC	24
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE_LOC	25
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B_LOC	26
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_LOC	27
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED_LOC	28
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0_LOC			29
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE_LOC		31
+
+#define CM_CFG_PM_STATUS 0xb4000014
+#define CM_CFG_PM_STATUS_RST 0x100403e
+
+#define CM_CFG_PM_STATUS_PROCHOT		0x00000001
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE		0x00000002
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B	0x00000004
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B	0x00000008
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B	0x00000010
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B	0x00000020
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B	0x00000040
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B		0x00000080
+#define CM_CFG_PM_STATUS_RSVZ0			0x00000100
+#define CM_CFG_PM_STATUS_RSVZ1			0x00000200
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON		0x00000400
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE	0x00000800
+#define CM_CFG_PM_STATUS_RSVZ2			0x00001000
+#define CM_CFG_PM_STATUS_RSVZ3			0x00002000
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK	0x00004000
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK	0x00008000
+#define CM_CFG_PM_STATUS_DLB_IN_D3		0x00010000
+#define CM_CFG_PM_STATUS_RSVZ4			0x00FE0000
+#define CM_CFG_PM_STATUS_PMSM			0xFF000000
+#define CM_CFG_PM_STATUS_PROCHOT_LOC			0
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE_LOC		1
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B_LOC	2
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B_LOC		3
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B_LOC		4
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B_LOC		5
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B_LOC		6
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B_LOC		7
+#define CM_CFG_PM_STATUS_RSVZ0_LOC			8
+#define CM_CFG_PM_STATUS_RSVZ1_LOC			9
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON_LOC		10
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE_LOC		11
+#define CM_CFG_PM_STATUS_RSVZ2_LOC			12
+#define CM_CFG_PM_STATUS_RSVZ3_LOC			13
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK_LOC		14
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK_LOC		15
+#define CM_CFG_PM_STATUS_DLB_IN_D3_LOC			16
+#define CM_CFG_PM_STATUS_RSVZ4_LOC			17
+#define CM_CFG_PM_STATUS_PMSM_LOC			24
+
+#define CM_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define CM_CFG_PM_PMCSR_DISABLE_RST 0x1
+
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE	0x00000001
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0	0xFFFFFFFE
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE_LOC	0
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0_LOC	1
+
+#endif /* __DLB_REGS_H */
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
new file mode 100644
index 000000000000..fca444c46aca
--- /dev/null
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include "dlb_bitmap.h"
+#include "dlb_hw_types.h"
+#include "dlb_regs.h"
+#include "dlb_resource.h"
+
+static void dlb_init_fn_rsrc_lists(struct dlb_function_resources *rsrc)
+{
+	int i;
+
+	INIT_LIST_HEAD(&rsrc->avail_domains);
+	INIT_LIST_HEAD(&rsrc->used_domains);
+	INIT_LIST_HEAD(&rsrc->avail_ldb_queues);
+	INIT_LIST_HEAD(&rsrc->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&rsrc->avail_ldb_ports[i]);
+}
+
+static void dlb_init_domain_rsrc_lists(struct dlb_hw_domain *domain)
+{
+	int i;
+
+	INIT_LIST_HEAD(&domain->used_ldb_queues);
+	INIT_LIST_HEAD(&domain->used_dir_pq_pairs);
+	INIT_LIST_HEAD(&domain->avail_ldb_queues);
+	INIT_LIST_HEAD(&domain->avail_dir_pq_pairs);
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&domain->used_ldb_ports[i]);
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		INIT_LIST_HEAD(&domain->avail_ldb_ports[i]);
+}
+
+/**
+ * dlb_resource_free() - free device state memory
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb_resource_free(struct dlb_hw *hw)
+{
+	int i;
+
+	if (hw->pf.avail_hist_list_entries)
+		dlb_bitmap_free(hw->pf.avail_hist_list_entries);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+		if (hw->vdev[i].avail_hist_list_entries)
+			dlb_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+	}
+}
+
+/**
+ * dlb_resource_init() - initialize the device
+ * @hw: pointer to struct dlb_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization, and the dlb_hw structure should
+ * be zero-initialized before calling the function.
+ *
+ * The dlb_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb_resource_init(struct dlb_hw *hw)
+{
+	struct dlb_bitmap *map;
+	struct list_head *list;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * For optimal load-balancing, ports that map to one or more QIDs in
+	 * common should not be in numerical sequence. The port->QID mapping is
+	 * application dependent, but the driver interleaves port IDs as much
+	 * as possible to reduce the likelihood of sequential ports mapping to
+	 * the same QID(s). This initial allocation of port IDs maximizes the
+	 * average distance between an ID and its immediate neighbors (i.e.
+	 * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+	 * 3, etc.).
+	 */
+	const u8 init_ldb_port_allocation[DLB_MAX_NUM_LDB_PORTS] = {
+		0,  7,  14,  5, 12,  3, 10,  1,  8, 15,  6, 13,  4, 11,  2,  9,
+		16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+		32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+		48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+	};
+
+	dlb_init_fn_rsrc_lists(&hw->pf);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++)
+		dlb_init_fn_rsrc_lists(&hw->vdev[i]);
+
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+		dlb_init_domain_rsrc_lists(&hw->domains[i]);
+		hw->domains[i].parent_func = &hw->pf;
+	}
+
+	/* Give all resources to the PF driver */
+	hw->pf.num_avail_domains = DLB_MAX_NUM_DOMAINS;
+	for (i = 0; i < hw->pf.num_avail_domains; i++) {
+		list = &hw->domains[i].func_list;
+
+		list_add(list, &hw->pf.avail_domains);
+	}
+
+	hw->pf.num_avail_ldb_queues = DLB_MAX_NUM_LDB_QUEUES;
+	for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+		list = &hw->rsrcs.ldb_queues[i].func_list;
+
+		list_add(list, &hw->pf.avail_ldb_queues);
+	}
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		hw->pf.num_avail_ldb_ports[i] =
+			DLB_MAX_NUM_LDB_PORTS / DLB_NUM_COS_DOMAINS;
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+		int cos_id = i >> DLB_NUM_COS_DOMAINS;
+		struct dlb_ldb_port *port;
+
+		port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+		list_add(&port->func_list, &hw->pf.avail_ldb_ports[cos_id]);
+	}
+
+	hw->pf.num_avail_dir_pq_pairs = DLB_MAX_NUM_DIR_PORTS;
+	for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+		list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+		list_add(list, &hw->pf.avail_dir_pq_pairs);
+	}
+
+	hw->pf.num_avail_qed_entries = DLB_MAX_NUM_LDB_CREDITS;
+	hw->pf.num_avail_dqed_entries = DLB_MAX_NUM_DIR_CREDITS;
+	hw->pf.num_avail_aqed_entries = DLB_MAX_NUM_AQED_ENTRIES;
+
+	ret = dlb_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+			       DLB_MAX_NUM_HIST_LIST_ENTRIES);
+	if (ret)
+		goto unwind;
+
+	map = hw->pf.avail_hist_list_entries;
+	bitmap_fill(map->map, map->len);
+
+	for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+		ret = dlb_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+				       DLB_MAX_NUM_HIST_LIST_ENTRIES);
+		if (ret)
+			goto unwind;
+
+		map = hw->vdev[i].avail_hist_list_entries;
+		bitmap_zero(map->map, map->len);
+	}
+
+	/* Initialize the hardware resource IDs */
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+		hw->domains[i].id.phys_id = i;
+		hw->domains[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_QUEUES; i++) {
+		hw->rsrcs.ldb_queues[i].id.phys_id = i;
+		hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+		hw->rsrcs.ldb_ports[i].id.phys_id = i;
+		hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB_MAX_NUM_DIR_PORTS; i++) {
+		hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+		hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+	}
+
+	for (i = 0; i < DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		hw->rsrcs.sn_groups[i].id = i;
+		/* Default mode (0) is 64 sequence numbers per queue */
+		hw->rsrcs.sn_groups[i].mode = 0;
+		hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+		hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+	}
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		hw->cos_reservation[i] = 100 / DLB_NUM_COS_DOMAINS;
+
+	return 0;
+
+unwind:
+	dlb_resource_free(hw);
+
+	return ret;
+}
+
+/**
+ * dlb_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw)
+{
+	u32 pmcsr_dis;
+
+	pmcsr_dis = DLB_CSR_RD(hw, CM_CFG_PM_PMCSR_DISABLE);
+
+	/* Clear register bits */
+	pmcsr_dis &= ~CM_CFG_PM_PMCSR_DISABLE_DISABLE;
+
+	DLB_CSR_WR(hw, CM_CFG_PM_PMCSR_DISABLE, pmcsr_dis);
+}
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
new file mode 100644
index 000000000000..2229813d9c45
--- /dev/null
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_RESOURCE_H
+#define __DLB_RESOURCE_H
+
+#include <linux/types.h>
+
+#include "dlb_hw_types.h"
+
+int dlb_resource_init(struct dlb_hw *hw);
+
+void dlb_resource_free(struct dlb_hw *hw);
+
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
+
+#endif /* __DLB_RESOURCE_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (2 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 03/20] dlb: add resource and device initialization Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-03-09  9:26   ` Greg KH
  2021-02-10 17:54 ` [PATCH v10 05/20] dlb: add scheduling domain configuration Mike Ximing Chen
                   ` (16 subsequent siblings)
  20 siblings, 1 reply; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Introduce the dlb device ioctl layer and the first three ioctls: query
device version, query available resources, and create a scheduling domain.
Also introduce the user-space interface file dlb_user.h.

The device version query is designed to allow each DLB device version/type
to have its own unique ioctl API through the /dev/dlb%d node. Each such API
would share in common the device version command as its first command, and
all subsequent commands can be unique to the particular device.

The hardware operation for scheduling domain creation will be added in a
subsequent commit.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 drivers/misc/dlb/Makefile                     |   2 +-
 drivers/misc/dlb/dlb_bitmap.h                 |  32 ++++
 drivers/misc/dlb/dlb_ioctl.c                  | 103 +++++++++++
 drivers/misc/dlb/dlb_main.c                   |   1 +
 drivers/misc/dlb/dlb_main.h                   |  10 ++
 drivers/misc/dlb/dlb_pf_ops.c                 |  22 +++
 drivers/misc/dlb/dlb_resource.c               |  62 +++++++
 drivers/misc/dlb/dlb_resource.h               |   4 +
 include/uapi/linux/dlb.h                      | 167 ++++++++++++++++++
 10 files changed, 403 insertions(+), 1 deletion(-)
 create mode 100644 drivers/misc/dlb/dlb_ioctl.c
 create mode 100644 include/uapi/linux/dlb.h

diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..747b48b141c8 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -300,6 +300,7 @@ Code  Seq#    Include File                                           Comments
 'z'   10-4F  drivers/s390/crypto/zcrypt_api.h                        conflict!
 '|'   00-7F  linux/media.h
 0x80  00-1F  linux/fb.h
+0x81  00-1F  uapi/linux/dlb.h
 0x89  00-06  arch/x86/include/asm/sockios.h
 0x89  0B-DF  linux/sockios.h
 0x89  E0-EF  linux/sockios.h                                         SIOCPROTOPRIVATE range
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index 8a49ea5fd752..aaafb3086d8d 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -7,4 +7,4 @@
 obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
-dlb-objs += dlb_pf_ops.o dlb_resource.o
+dlb-objs += dlb_pf_ops.o dlb_resource.o dlb_ioctl.o
diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
index fb3ef52a306d..3ea78b42c79f 100644
--- a/drivers/misc/dlb/dlb_bitmap.h
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -73,4 +73,36 @@ static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
 	kfree(bitmap);
 }
 
+/**
+ * dlb_bitmap_longest_set_range() - returns longest contiguous range of set
+ *				     bits
+ * @bitmap: pointer to dlb_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb_bitmap_longest_set_range(struct dlb_bitmap *bitmap)
+{
+	int max_len, len;
+	int start, end;
+
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap_weight(bitmap->map, bitmap->len) == 0)
+		return 0;
+
+	max_len = 0;
+	bitmap_for_each_set_region(bitmap->map, start, end, 0, bitmap->len) {
+		len = end - start;
+		if (max_len < len)
+			max_len = len;
+	}
+	return max_len;
+}
+
 #endif /*  __DLB_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
new file mode 100644
index 000000000000..47d6cab773d4
--- /dev/null
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -0,0 +1,103 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/uaccess.h>
+
+#include <uapi/linux/dlb.h>
+
+#include "dlb_main.h"
+
+/* [7:0]: device revision, [15:8]: device version */
+#define DLB_SET_DEVICE_VERSION(ver, rev) (((ver) << 8) | (rev))
+
+static int dlb_ioctl_get_device_version(unsigned long user_arg)
+{
+	struct dlb_get_device_version_args arg;
+	u8 revision;
+
+	switch (boot_cpu_data.x86_stepping) {
+	case 0:
+		revision = DLB_REV_A0;
+		break;
+	case 1:
+		revision = DLB_REV_A1;
+		break;
+	case 2:
+		revision = DLB_REV_A2;
+		break;
+	default:
+		/* Treat all revisions >= 3 as B0 */
+		revision = DLB_REV_B0;
+		break;
+	}
+
+	arg.response.status = 0;
+	arg.response.id = DLB_SET_DEVICE_VERSION(2, revision);
+
+	if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg)
+{
+	struct dlb_create_sched_domain_args __user *uarg;
+	struct dlb_create_sched_domain_args arg;
+	struct dlb_cmd_response response = {0};
+	int ret;
+
+	uarg = (void __user *)user_arg;
+	if (copy_from_user(&arg, uarg, sizeof(arg)))
+		return -EFAULT;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	ret = dlb->ops->create_sched_domain(&dlb->hw, &arg, &response);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	response.status = ret;
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+		return -EFAULT;
+
+	return ret;
+}
+
+static int dlb_ioctl_get_num_resources(struct dlb *dlb, unsigned long user_arg)
+{
+	struct dlb_get_num_resources_args arg = {0};
+	int ret;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	ret = dlb->ops->get_num_resources(&dlb->hw, &arg);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+	arg.response.status = ret;
+
+	if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+		return -EFAULT;
+
+	return ret;
+}
+
+long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+	struct dlb *dlb = f->private_data;
+
+	switch (cmd) {
+	case DLB_IOC_GET_DEVICE_VERSION:
+		return dlb_ioctl_get_device_version(arg);
+	case DLB_IOC_CREATE_SCHED_DOMAIN:
+		return dlb_ioctl_create_sched_domain(dlb, arg);
+	case DLB_IOC_GET_NUM_RESOURCES:
+		return dlb_ioctl_get_num_resources(dlb, arg);
+	default:
+		return -ENOTTY;
+	}
+}
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 12707b23ab3e..d92956b1643d 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -64,6 +64,7 @@ static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
 
 static const struct file_operations dlb_fops = {
 	.owner   = THIS_MODULE,
+	.unlocked_ioctl = dlb_ioctl,
 };
 
 /**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index ec5eb7bd8f54..3089a66a3560 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -12,6 +12,8 @@
 #include <linux/pci.h>
 #include <linux/types.h>
 
+#include <uapi/linux/dlb.h>
+
 #include "dlb_hw_types.h"
 
 /*
@@ -37,6 +39,11 @@ struct dlb_device_ops {
 	int (*init_driver_state)(struct dlb *dlb);
 	void (*enable_pm)(struct dlb *dlb);
 	int (*wait_for_device_ready)(struct dlb *dlb, struct pci_dev *pdev);
+	int (*create_sched_domain)(struct dlb_hw *hw,
+				   struct dlb_create_sched_domain_args *args,
+				   struct dlb_cmd_response *resp);
+	int (*get_num_resources)(struct dlb_hw *hw,
+				 struct dlb_get_num_resources_args *args);
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
@@ -56,4 +63,7 @@ struct dlb {
 	dev_t dev_number;
 };
 
+/* Prototypes for dlb_ioctl.c */
+long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+
 #endif /* __DLB_MAIN_H */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 124b4fee8564..125ef6fe6c70 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -95,6 +95,26 @@ static int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
 	return 0;
 }
 
+/*****************************/
+/****** IOCTL callbacks ******/
+/*****************************/
+
+static int dlb_pf_create_sched_domain(struct dlb_hw *hw,
+				      struct dlb_create_sched_domain_args *args,
+				      struct dlb_cmd_response *resp)
+{
+	resp->id = 0;
+	resp->status = 0;
+
+	return 0;
+}
+
+static int dlb_pf_get_num_resources(struct dlb_hw *hw,
+				    struct dlb_get_num_resources_args *args)
+{
+	return dlb_hw_get_num_resources(hw, args, false, 0);
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -105,4 +125,6 @@ struct dlb_device_ops dlb_pf_ops = {
 	.init_driver_state = dlb_pf_init_driver_state,
 	.enable_pm = dlb_pf_enable_pm,
 	.wait_for_device_ready = dlb_pf_wait_for_device_ready,
+	.create_sched_domain = dlb_pf_create_sched_domain,
+	.get_num_resources = dlb_pf_get_num_resources,
 };
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index fca444c46aca..9d75b12eb793 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -200,6 +200,68 @@ int dlb_resource_init(struct dlb_hw *hw)
 	return ret;
 }
 
+/**
+ * dlb_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_req is true and vdev_id is
+ * invalid.
+ */
+int dlb_hw_get_num_resources(struct dlb_hw *hw,
+			     struct dlb_get_num_resources_args *arg,
+			     bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_function_resources *rsrcs;
+	struct dlb_bitmap *map;
+	int i;
+
+	if (vdev_req && vdev_id >= DLB_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	if (vdev_req)
+		rsrcs = &hw->vdev[vdev_id];
+	else
+		rsrcs = &hw->pf;
+
+	arg->num_sched_domains = rsrcs->num_avail_domains;
+
+	arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+	arg->num_ldb_ports = 0;
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+		arg->num_cos_ldb_ports[i] = rsrcs->num_avail_ldb_ports[i];
+
+	arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+	arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+	map = rsrcs->avail_hist_list_entries;
+
+	arg->num_hist_list_entries = bitmap_weight(map->map, map->len);
+
+	arg->max_contiguous_hist_list_entries =
+		dlb_bitmap_longest_set_range(map);
+
+	arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+	arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+	return 0;
+}
+
 /**
  * dlb_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
  * @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 2229813d9c45..3e6d419796bc 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -12,6 +12,10 @@ int dlb_resource_init(struct dlb_hw *hw);
 
 void dlb_resource_free(struct dlb_hw *hw);
 
+int dlb_hw_get_num_resources(struct dlb_hw *hw,
+			     struct dlb_get_num_resources_args *arg,
+			     bool vdev_req, unsigned int vdev_id);
+
 void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
 
 #endif /* __DLB_RESOURCE_H */
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
new file mode 100644
index 000000000000..87ba1bfa75ed
--- /dev/null
+++ b/include/uapi/linux/dlb.h
@@ -0,0 +1,167 @@
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_H
+#define __DLB_H
+
+#include <linux/types.h>
+
+struct dlb_cmd_response {
+	__u32 status; /* Interpret using enum dlb_error */
+	__u32 id;
+};
+
+/********************************/
+/* 'dlb' device file commands  */
+/********************************/
+
+#define DLB_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb_revisions {
+	DLB_REV_A0 = 0,
+	DLB_REV_A1,
+	DLB_REV_A2,
+	DLB_REV_B0,
+};
+
+/*
+ * DLB_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ *	All DLB device versions have the same ioctl API. Each version may have
+ *	different resource and feature set. The device revision is provided
+ *	in case of any hardware errata.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id[7:0]: Device revision.
+ * @response.id[15:8]: Device version.
+ */
+
+struct dlb_get_device_version_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+};
+
+/*
+ * DLB_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ *	its hardware resources. This command returns the newly created domain
+ *	ID and a file descriptor for accessing the domain.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: domain ID.
+ * @domain_fd: file descriptor for performing the domain's ioctl operations
+ * @padding0: Reserved for future use.
+ *
+ * Input parameters:
+ * @num_ldb_queues: Number of load-balanced queues.
+ * @num_ldb_ports: Number of load-balanced ports that can be allocated from
+ *	any class-of-service with available ports.
+ * @num_cos_ldb_ports[4]: Number of load-balanced ports from
+ *	classes-of-service 0-3.
+ * @num_dir_ports: Number of directed ports. A directed port has one directed
+ *	queue, so no num_dir_queues argument is necessary.
+ * @num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for the domain. This storage is divided among the domain's
+ *	load-balanced queues that are configured for atomic scheduling.
+ * @num_hist_list_entries: Amount of history list storage. This is divided
+ *	among the domain's CQs.
+ * @num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ *	space until they are scheduled to a load-balanced CQ. One credit
+ *	represents the storage for one QE.
+ * @num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ *	space until they are scheduled to a directed CQ. One credit represents
+ *	the storage for one QE.
+ * @cos_strict: If set, return an error if there are insufficient ports in
+ *	class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ *	unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ *	classes-of-service if class N does not contain enough free ports.
+ * @padding1: Reserved for future use.
+ */
+struct dlb_create_sched_domain_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	__u32 domain_fd;
+	__u32 padding0;
+	/* Input parameters */
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+	__u8 cos_strict;
+	__u8 padding1[3];
+};
+
+/*
+ * DLB_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ *	(queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @num_domains: Number of available scheduling domains.
+ * @num_ldb_queues: Number of available load-balanced queues.
+ * @num_ldb_ports: Total number of available load-balanced ports.
+ * @num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ *	classes-of-service 0-3.
+ * @num_dir_ports: Number of available directed ports. There is one directed
+ *	queue for every directed port.
+ * @num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * @num_hist_list_entries: Amount of history list storage.
+ * @max_contiguous_hist_list_entries: History list storage is allocated in
+ *	a contiguous chunk, and this return value is the longest available
+ *	contiguous range of history list entries.
+ * @num_ldb_credits: Amount of available load-balanced QE storage.
+ * @num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb_get_num_resources_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	__u32 num_sched_domains;
+	__u32 num_ldb_queues;
+	__u32 num_ldb_ports;
+	__u32 num_cos_ldb_ports[4];
+	__u32 num_dir_ports;
+	__u32 num_atomic_inflights;
+	__u32 num_hist_list_entries;
+	__u32 max_contiguous_hist_list_entries;
+	__u32 num_ldb_credits;
+	__u32 num_dir_credits;
+};
+
+enum dlb_user_interface_commands {
+	DLB_CMD_GET_DEVICE_VERSION,
+	DLB_CMD_CREATE_SCHED_DOMAIN,
+	DLB_CMD_GET_NUM_RESOURCES,
+
+	/* NUM_DLB_CMD must be last */
+	NUM_DLB_CMD,
+};
+
+/********************/
+/* dlb ioctl codes */
+/********************/
+
+#define DLB_IOC_MAGIC	0x81
+
+#define DLB_IOC_GET_DEVICE_VERSION				\
+		_IOR(DLB_IOC_MAGIC,				\
+		     DLB_CMD_GET_DEVICE_VERSION,		\
+		     struct dlb_get_device_version_args)
+#define DLB_IOC_CREATE_SCHED_DOMAIN				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_CMD_CREATE_SCHED_DOMAIN,		\
+		      struct dlb_create_sched_domain_args)
+#define DLB_IOC_GET_NUM_RESOURCES				\
+		_IOR(DLB_IOC_MAGIC,				\
+		     DLB_CMD_GET_NUM_RESOURCES,			\
+		     struct dlb_get_num_resources_args)
+
+#endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 05/20] dlb: add scheduling domain configuration
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (3 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-03-09  9:28   ` Greg KH
  2021-02-10 17:54 ` [PATCH v10 06/20] dlb: add domain software reset Mike Ximing Chen
                   ` (15 subsequent siblings)
  20 siblings, 1 reply; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add support for configuring a scheduling domain, creating the domain fd,
and reserving the domain's resources.

A scheduling domain serves as a container of DLB resources -- e.g. ports,
queues, and credits -- with the property that a port can only enqueue
to and dequeue from queues within its domain. A scheduling domain is
created on-demand by a user-space application, whose request includes
the DLB resource allocation.

When a user requests to create a scheduling domain, the requested resources
are validated against the number currently available, and then reserved for
the scheduling domain. Finally, the ioctl handler allocates an anonymous
file descriptor for the domain and installs this in the calling process's
file descriptor table.

Once created, user-space can use this file descriptor to configure the
scheduling domain's resources (to be added in a subsequent commit). For
multiprocess applications, this descriptor can be shared over a unix
domain socket.

The driver maintains a reference count for each scheduling domain,
incrementing it each time user-space requests a file descriptor and
decrementing it in the file's release callback.

When the reference count transitions from 1->0 the driver automatically
resets the scheduling domain's resources and makes them available for use
by future applications. This ensures that applications that crash without
explicitly cleaning up do not orphan device resources. The code to perform
the domain reset will be added in subsequent commits.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_bitmap.h   |  74 ++++
 drivers/misc/dlb/dlb_ioctl.c    |  36 +-
 drivers/misc/dlb/dlb_main.c     |  68 ++++
 drivers/misc/dlb/dlb_main.h     |  24 ++
 drivers/misc/dlb/dlb_pf_ops.c   |   5 +-
 drivers/misc/dlb/dlb_regs.h     |  18 +
 drivers/misc/dlb/dlb_resource.c | 605 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |   7 +
 include/uapi/linux/dlb.h        |  14 +
 9 files changed, 845 insertions(+), 6 deletions(-)

diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
index 3ea78b42c79f..5cebf833fab4 100644
--- a/drivers/misc/dlb/dlb_bitmap.h
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -73,6 +73,80 @@ static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
 	kfree(bitmap);
 }
 
+/**
+ * dlb_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb_bitmap_clear_range(struct dlb_bitmap *bitmap,
+					 unsigned int bit,
+					 unsigned int len)
+{
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	bitmap_clear(bitmap->map, bit, len);
+
+	return 0;
+}
+
+/**
+ * dlb_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb_bitmap_find_set_bit_range(struct dlb_bitmap *bitmap,
+						unsigned int len)
+{
+	struct dlb_bitmap *complement_mask = NULL;
+	int ret;
+
+	if (!bitmap || !bitmap->map || len == 0)
+		return -EINVAL;
+
+	if (bitmap->len < len)
+		return -ENOENT;
+
+	ret = dlb_bitmap_alloc(&complement_mask, bitmap->len);
+	if (ret)
+		return ret;
+
+	bitmap_zero(complement_mask->map, complement_mask->len);
+
+	bitmap_complement(complement_mask->map, bitmap->map, bitmap->len);
+
+	ret = bitmap_find_next_zero_area(complement_mask->map,
+					 complement_mask->len,
+					 0,
+					 len,
+					 0);
+
+	dlb_bitmap_free(complement_mask);
+
+	/* No set bit range of length len? */
+	return (ret >= (int)bitmap->len) ? -ENOENT : ret;
+}
+
 /**
  * dlb_bitmap_longest_set_range() - returns longest contiguous range of set
  *				     bits
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 47d6cab773d4..7871d0cea118 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
 
+#include <linux/anon_inodes.h>
 #include <linux/uaccess.h>
 
 #include <uapi/linux/dlb.h>
@@ -45,7 +46,10 @@ static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg
 	struct dlb_create_sched_domain_args __user *uarg;
 	struct dlb_create_sched_domain_args arg;
 	struct dlb_cmd_response response = {0};
-	int ret;
+	struct dlb_domain *domain;
+	u32 flags = O_RDONLY;
+	size_t offset;
+	int ret, fd;
 
 	uarg = (void __user *)user_arg;
 	if (copy_from_user(&arg, uarg, sizeof(arg)))
@@ -54,10 +58,38 @@ static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg
 	mutex_lock(&dlb->resource_mutex);
 
 	ret = dlb->ops->create_sched_domain(&dlb->hw, &arg, &response);
+	if (ret)
+		goto unlock;
+
+	ret = dlb_init_domain(dlb, response.id);
+	if (ret)
+		goto unlock;
+
+	domain = dlb->sched_domains[response.id];
+
+	if (dlb->f->f_mode & FMODE_WRITE)
+		flags = O_RDWR;
+
+	fd = anon_inode_getfd("[dlbdomain]", &dlb_domain_fops,
+			      domain, flags);
+
+	if (fd < 0) {
+		dev_err(dlb->dev, "Failed to get anon fd.\n");
+		kref_put(&domain->refcnt, dlb_free_domain);
+		ret = fd;
+		goto unlock;
+	}
+
+	offset = offsetof(struct dlb_create_sched_domain_args, domain_fd);
+
+	if (copy_to_user((void __user *)(user_arg + offset), &fd, sizeof(fd))) {
+		mutex_unlock(&dlb->resource_mutex);
+		return -EFAULT;
+	}
 
+unlock:
 	mutex_unlock(&dlb->resource_mutex);
 
-	response.status = ret;
 	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
 
 	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index d92956b1643d..a4ed413eee2f 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -62,11 +62,79 @@ static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
 /****** Char dev callbacks ******/
 /********************************/
 
+static int dlb_open(struct inode *i, struct file *f)
+{
+	struct dlb *dlb;
+
+	spin_lock(&dlb_ids_lock);
+	dlb = idr_find(&dlb_ids, iminor(i));
+	spin_unlock(&dlb_ids_lock);
+
+	f->private_data = dlb;
+	dlb->f = f;
+
+	return 0;
+}
+
 static const struct file_operations dlb_fops = {
 	.owner   = THIS_MODULE,
+	.open    = dlb_open,
 	.unlocked_ioctl = dlb_ioctl,
 };
 
+int dlb_init_domain(struct dlb *dlb, u32 domain_id)
+{
+	struct dlb_domain *domain;
+
+	domain = kzalloc(sizeof(*domain), GFP_KERNEL);
+	if (!domain)
+		return -ENOMEM;
+
+	domain->id = domain_id;
+
+	kref_init(&domain->refcnt);
+	domain->dlb = dlb;
+
+	dlb->sched_domains[domain_id] = domain;
+
+	return 0;
+}
+
+static int __dlb_free_domain(struct dlb_domain *domain)
+{
+	struct dlb *dlb = domain->dlb;
+
+	dlb->sched_domains[domain->id] = NULL;
+
+	kfree(domain);
+
+	return 0;
+}
+
+void dlb_free_domain(struct kref *kref)
+{
+	__dlb_free_domain(container_of(kref, struct dlb_domain, refcnt));
+}
+
+static int dlb_domain_close(struct inode *i, struct file *f)
+{
+	struct dlb_domain *domain = f->private_data;
+	struct dlb *dlb = domain->dlb;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	kref_put(&domain->refcnt, dlb_free_domain);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	return 0;
+}
+
+const struct file_operations dlb_domain_fops = {
+	.owner   = THIS_MODULE,
+	.release = dlb_domain_close,
+};
+
 /**********************************/
 /****** PCI driver callbacks ******/
 /**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 3089a66a3560..824416e6cdcf 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -47,12 +47,21 @@ struct dlb_device_ops {
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
+extern const struct file_operations dlb_domain_fops;
+
+struct dlb_domain {
+	struct dlb *dlb;
+	struct kref refcnt;
+	u8 id;
+};
 
 struct dlb {
 	struct pci_dev *pdev;
 	struct dlb_hw hw;
 	struct dlb_device_ops *ops;
 	struct device *dev;
+	struct dlb_domain *sched_domains[DLB_MAX_NUM_DOMAINS];
+	struct file *f;
 	/*
 	 * The resource mutex serializes access to driver data structures and
 	 * hardware registers.
@@ -66,4 +75,19 @@ struct dlb {
 /* Prototypes for dlb_ioctl.c */
 long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
 
+int dlb_init_domain(struct dlb *dlb, u32 domain_id);
+void dlb_free_domain(struct kref *kref);
+
+#define DLB_HW_ERR(hw, ...) do {		  \
+	struct dlb *dlb;			  \
+	dlb = container_of(hw, struct dlb, hw); \
+	dev_err(dlb->dev, __VA_ARGS__);	  \
+} while (0)
+
+#define DLB_HW_DBG(hw, ...) do {		  \
+	struct dlb *dlb;			  \
+	dlb = container_of(hw, struct dlb, hw); \
+	dev_dbg(dlb->dev, __VA_ARGS__);	  \
+} while (0)
+
 #endif /* __DLB_MAIN_H */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 125ef6fe6c70..b59e9eaa600d 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -103,10 +103,7 @@ static int dlb_pf_create_sched_domain(struct dlb_hw *hw,
 				      struct dlb_create_sched_domain_args *args,
 				      struct dlb_cmd_response *resp)
 {
-	resp->id = 0;
-	resp->status = 0;
-
-	return 0;
+	return dlb_hw_create_sched_domain(hw, args, resp, false, 0);
 }
 
 static int dlb_pf_get_num_resources(struct dlb_hw *hw,
diff --git a/drivers/misc/dlb/dlb_regs.h b/drivers/misc/dlb/dlb_regs.h
index 72f3cb22b933..0fd499f384de 100644
--- a/drivers/misc/dlb/dlb_regs.h
+++ b/drivers/misc/dlb/dlb_regs.h
@@ -6,6 +6,24 @@
 
 #include <linux/types.h>
 
+#define CHP_CFG_DIR_VAS_CRD(x) \
+	(0x40000000 + (x) * 0x1000)
+#define CHP_CFG_DIR_VAS_CRD_RST 0x0
+
+#define CHP_CFG_DIR_VAS_CRD_COUNT	0x00003FFF
+#define CHP_CFG_DIR_VAS_CRD_RSVD0	0xFFFFC000
+#define CHP_CFG_DIR_VAS_CRD_COUNT_LOC	0
+#define CHP_CFG_DIR_VAS_CRD_RSVD0_LOC	14
+
+#define CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define CHP_CFG_LDB_VAS_CRD_RST 0x0
+
+#define CHP_CFG_LDB_VAS_CRD_COUNT	0x00007FFF
+#define CHP_CFG_LDB_VAS_CRD_RSVD0	0xFFFF8000
+#define CHP_CFG_LDB_VAS_CRD_COUNT_LOC	0
+#define CHP_CFG_LDB_VAS_CRD_RSVD0_LOC	15
+
 #define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
 #define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
 
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 9d75b12eb793..b7df23c6a158 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -3,6 +3,7 @@
 
 #include "dlb_bitmap.h"
 #include "dlb_hw_types.h"
+#include "dlb_main.h"
 #include "dlb_regs.h"
 #include "dlb_resource.h"
 
@@ -200,6 +201,610 @@ int dlb_resource_init(struct dlb_hw *hw)
 	return ret;
 }
 
+static int dlb_attach_ldb_queues(struct dlb_hw *hw,
+				 struct dlb_function_resources *rsrcs,
+				 struct dlb_hw_domain *domain, u32 num_queues,
+				 struct dlb_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_queues < num_queues) {
+		resp->status = DLB_ST_LDB_QUEUES_UNAVAILABLE;
+		DLB_HW_DBG(hw, "[%s()] Internal error: %d\n", __func__,
+			   resp->status);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_queues; i++) {
+		struct dlb_ldb_queue *queue;
+
+		queue = list_first_entry_or_null(&rsrcs->avail_ldb_queues,
+						 typeof(*queue), func_list);
+		if (!queue) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: domain validation failed\n",
+				   __func__);
+			return -EFAULT;
+		}
+
+		list_del(&queue->func_list);
+
+		queue->domain_id = domain->id;
+		queue->owned = true;
+
+		list_add(&queue->domain_list, &domain->avail_ldb_queues);
+	}
+
+	rsrcs->num_avail_ldb_queues -= num_queues;
+
+	return 0;
+}
+
+static struct dlb_ldb_port *
+dlb_get_next_ldb_port(struct dlb_hw *hw, struct dlb_function_resources *rsrcs,
+		      u32 domain_id, u32 cos_id)
+{
+	struct dlb_ldb_port *port;
+
+	/*
+	 * To reduce the odds of consecutive load-balanced ports mapping to the
+	 * same queue(s), the driver attempts to allocate ports whose neighbors
+	 * are owned by a different domain.
+	 */
+	list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[next].owned ||
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+			continue;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned ||
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+			continue;
+
+		return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with one neighbor owned by
+	 * a different domain and the other unallocated.
+	 */
+	list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+			return port;
+
+		if (!hw->rsrcs.ldb_ports[next].owned &&
+		    hw->rsrcs.ldb_ports[prev].owned &&
+		    hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+			return port;
+	}
+
+	/*
+	 * Failing that, the driver looks for a port with both neighbors
+	 * unallocated.
+	 */
+	list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+		u32 next, prev;
+		u32 phys_id;
+
+		phys_id = port->id.phys_id;
+		next = phys_id + 1;
+		prev = phys_id - 1;
+
+		if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+			next = 0;
+		if (phys_id == 0)
+			prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+		if (!hw->rsrcs.ldb_ports[prev].owned &&
+		    !hw->rsrcs.ldb_ports[next].owned)
+			return port;
+	}
+
+	/* If all else fails, the driver returns the next available port. */
+	return list_first_entry_or_null(&rsrcs->avail_ldb_ports[cos_id],
+					typeof(*port), func_list);
+}
+
+static int __dlb_attach_ldb_ports(struct dlb_hw *hw,
+				  struct dlb_function_resources *rsrcs,
+				  struct dlb_hw_domain *domain, u32 num_ports,
+				  u32 cos_id, struct dlb_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+		resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+		DLB_HW_DBG(hw,
+			   "[%s()] Internal error: %d\n",
+			   __func__,
+			   resp->status);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb_ldb_port *port;
+
+		port = dlb_get_next_ldb_port(hw, rsrcs,
+					     domain->id.phys_id, cos_id);
+		if (!port) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: domain validation failed\n",
+				   __func__);
+			return -EFAULT;
+		}
+
+		list_del(&port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		list_add(&port->domain_list,
+			 &domain->avail_ldb_ports[cos_id]);
+	}
+
+	rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+	return 0;
+}
+
+static int dlb_attach_ldb_ports(struct dlb_hw *hw,
+				struct dlb_function_resources *rsrcs,
+				struct dlb_hw_domain *domain,
+				struct dlb_create_sched_domain_args *args,
+				struct dlb_cmd_response *resp)
+{
+	unsigned int i, j;
+	int ret;
+
+	if (args->cos_strict) {
+		for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+			u32 num = args->num_cos_ldb_ports[i];
+
+			/* Allocate ports from specific classes-of-service */
+			ret = __dlb_attach_ldb_ports(hw, rsrcs, domain, num,
+						     i, resp);
+			if (ret)
+				return ret;
+		}
+	} else {
+		unsigned int k;
+		u32 cos_id;
+
+		/*
+		 * Attempt to allocate from specific class-of-service, but
+		 * fallback to the other classes if that fails.
+		 */
+		for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+			for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+				for (k = 0; k < DLB_NUM_COS_DOMAINS; k++) {
+					cos_id = (i + k) % DLB_NUM_COS_DOMAINS;
+
+					ret = __dlb_attach_ldb_ports(hw, rsrcs, domain,
+								     1, cos_id, resp);
+					if (ret == 0)
+						break;
+				}
+
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	/* Allocate num_ldb_ports from any class-of-service */
+	for (i = 0; i < args->num_ldb_ports; i++) {
+		for (j = 0; j < DLB_NUM_COS_DOMAINS; j++) {
+			ret = __dlb_attach_ldb_ports(hw, rsrcs, domain, 1, j, resp);
+			if (ret == 0)
+				break;
+		}
+
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+static int dlb_attach_dir_ports(struct dlb_hw *hw,
+				struct dlb_function_resources *rsrcs,
+				struct dlb_hw_domain *domain, u32 num_ports,
+				struct dlb_cmd_response *resp)
+{
+	unsigned int i;
+
+	if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+		resp->status = DLB_ST_DIR_PORTS_UNAVAILABLE;
+		DLB_HW_DBG(hw,
+			   "[%s()] Internal error: %d\n",
+			   __func__,
+			   resp->status);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < num_ports; i++) {
+		struct dlb_dir_pq_pair *port;
+
+		port = list_first_entry_or_null(&rsrcs->avail_dir_pq_pairs,
+						typeof(*port), func_list);
+		if (!port) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: domain validation failed\n",
+				   __func__);
+			return -EFAULT;
+		}
+
+		list_del(&port->func_list);
+
+		port->domain_id = domain->id;
+		port->owned = true;
+
+		list_add(&port->domain_list, &domain->avail_dir_pq_pairs);
+	}
+
+	rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+	return 0;
+}
+
+static int dlb_attach_ldb_credits(struct dlb_function_resources *rsrcs,
+				  struct dlb_hw_domain *domain, u32 num_credits,
+				  struct dlb_cmd_response *resp)
+{
+	if (rsrcs->num_avail_qed_entries < num_credits) {
+		resp->status = DLB_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_qed_entries -= num_credits;
+	domain->num_ldb_credits += num_credits;
+	return 0;
+}
+
+static int dlb_attach_dir_credits(struct dlb_function_resources *rsrcs,
+				  struct dlb_hw_domain *domain, u32 num_credits,
+				  struct dlb_cmd_response *resp)
+{
+	if (rsrcs->num_avail_dqed_entries < num_credits) {
+		resp->status = DLB_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_dqed_entries -= num_credits;
+	domain->num_dir_credits += num_credits;
+	return 0;
+}
+
+static int dlb_attach_atomic_inflights(struct dlb_function_resources *rsrcs,
+				       struct dlb_hw_domain *domain,
+				       u32 num_atomic_inflights,
+				       struct dlb_cmd_response *resp)
+{
+	if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+		resp->status = DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+	domain->num_avail_aqed_entries += num_atomic_inflights;
+	return 0;
+}
+
+static int
+dlb_attach_domain_hist_list_entries(struct dlb_function_resources *rsrcs,
+				    struct dlb_hw_domain *domain,
+				    u32 num_hist_list_entries,
+				    struct dlb_cmd_response *resp)
+{
+	struct dlb_bitmap *bitmap;
+	int base;
+
+	if (num_hist_list_entries) {
+		bitmap = rsrcs->avail_hist_list_entries;
+
+		base = dlb_bitmap_find_set_bit_range(bitmap,
+						     num_hist_list_entries);
+		if (base < 0)
+			goto error;
+
+		domain->total_hist_list_entries = num_hist_list_entries;
+		domain->avail_hist_list_entries = num_hist_list_entries;
+		domain->hist_list_entry_base = base;
+		domain->hist_list_entry_offset = 0;
+
+		dlb_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+	}
+	return 0;
+
+error:
+	resp->status = DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+	return -EINVAL;
+}
+
+static int
+dlb_verify_create_sched_dom_args(struct dlb_function_resources *rsrcs,
+				 struct dlb_create_sched_domain_args *args,
+				 struct dlb_cmd_response *resp,
+				 struct dlb_hw_domain **out_domain)
+{
+	u32 num_avail_ldb_ports, req_ldb_ports;
+	struct dlb_bitmap *avail_hl_entries;
+	unsigned int max_contig_hl_range;
+	struct dlb_hw_domain *domain;
+	int i;
+
+	avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+	max_contig_hl_range = dlb_bitmap_longest_set_range(avail_hl_entries);
+
+	num_avail_ldb_ports = 0;
+	req_ldb_ports = 0;
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+		req_ldb_ports += args->num_cos_ldb_ports[i];
+	}
+
+	req_ldb_ports += args->num_ldb_ports;
+
+	if (rsrcs->num_avail_domains < 1) {
+		resp->status = DLB_ST_DOMAIN_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	domain = list_first_entry_or_null(&rsrcs->avail_domains,
+					  typeof(*domain), func_list);
+	if (!domain) {
+		resp->status = DLB_ST_DOMAIN_UNAVAILABLE;
+		return -EFAULT;
+	}
+
+	if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+		resp->status = DLB_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (req_ldb_ports > num_avail_ldb_ports) {
+		resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	for (i = 0; args->cos_strict && i < DLB_NUM_COS_DOMAINS; i++) {
+		if (args->num_cos_ldb_ports[i] >
+		    rsrcs->num_avail_ldb_ports[i]) {
+			resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+		resp->status = DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+		resp->status = DLB_ST_DIR_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+		resp->status = DLB_ST_LDB_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+		resp->status = DLB_ST_DIR_CREDITS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (max_contig_hl_range < args->num_hist_list_entries) {
+		resp->status = DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+
+	return 0;
+}
+
+static void dlb_configure_domain_credits(struct dlb_hw *hw,
+					 struct dlb_hw_domain *domain)
+{
+	u32 reg = 0;
+
+	BITS_SET(reg, domain->num_ldb_credits, CHP_CFG_LDB_VAS_CRD_COUNT);
+	DLB_CSR_WR(hw, CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, domain->num_dir_credits, CHP_CFG_DIR_VAS_CRD_COUNT);
+	DLB_CSR_WR(hw, CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), reg);
+}
+
+static int
+dlb_domain_attach_resources(struct dlb_hw *hw,
+			    struct dlb_function_resources *rsrcs,
+			    struct dlb_hw_domain *domain,
+			    struct dlb_create_sched_domain_args *args,
+			    struct dlb_cmd_response *resp)
+{
+	int ret;
+
+	ret = dlb_attach_ldb_queues(hw, rsrcs, domain, args->num_ldb_queues, resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_ldb_ports(hw, rsrcs, domain, args, resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_dir_ports(hw, rsrcs, domain, args->num_dir_ports, resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_ldb_credits(rsrcs, domain,
+				     args->num_ldb_credits, resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_dir_credits(rsrcs, domain, args->num_dir_credits, resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_domain_hist_list_entries(rsrcs, domain,
+						  args->num_hist_list_entries,
+						  resp);
+	if (ret)
+		return ret;
+
+	ret = dlb_attach_atomic_inflights(rsrcs, domain,
+					  args->num_atomic_inflights, resp);
+	if (ret)
+		return ret;
+
+	dlb_configure_domain_credits(hw, domain);
+
+	domain->configured = true;
+
+	domain->started = false;
+
+	rsrcs->num_avail_domains--;
+
+	return 0;
+}
+
+static void
+dlb_log_create_sched_domain_args(struct dlb_hw *hw,
+				 struct dlb_create_sched_domain_args *args,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB create sched domain arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tNumber of LDB queues:          %d\n",
+		   args->num_ldb_queues);
+	DLB_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+		   args->num_ldb_ports);
+	DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 0):   %d\n",
+		   args->num_cos_ldb_ports[0]);
+	DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 1):   %d\n",
+		   args->num_cos_ldb_ports[1]);
+	DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 2):   %d\n",
+		   args->num_cos_ldb_ports[2]);
+	DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 3):   %d\n",
+		   args->num_cos_ldb_ports[3]);
+	DLB_HW_DBG(hw, "\tStrict CoS allocation:         %d\n",
+		   args->cos_strict);
+	DLB_HW_DBG(hw, "\tNumber of DIR ports:           %d\n",
+		   args->num_dir_ports);
+	DLB_HW_DBG(hw, "\tNumber of ATM inflights:       %d\n",
+		   args->num_atomic_inflights);
+	DLB_HW_DBG(hw, "\tNumber of hist list entries:   %d\n",
+		   args->num_hist_list_entries);
+	DLB_HW_DBG(hw, "\tNumber of LDB credits:         %d\n",
+		   args->num_ldb_credits);
+	DLB_HW_DBG(hw, "\tNumber of DIR credits:         %d\n",
+		   args->num_dir_credits);
+}
+
+/**
+ * dlb_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ *	    is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_sched_domain(struct dlb_hw *hw,
+			       struct dlb_create_sched_domain_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_function_resources *rsrcs;
+	struct dlb_hw_domain *domain;
+	int ret;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	dlb_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_create_sched_dom_args(rsrcs, args, resp, &domain);
+	if (ret)
+		return ret;
+
+	dlb_init_domain_rsrc_lists(domain);
+
+	ret = dlb_domain_attach_resources(hw, rsrcs, domain, args, resp);
+	if (ret) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: failed to verify args.\n",
+			   __func__);
+
+		return ret;
+	}
+
+	list_del(&domain->func_list);
+
+	list_add(&domain->func_list, &rsrcs->used_domains);
+
+	resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+	resp->status = 0;
+
+	return 0;
+}
+
 /**
  * dlb_hw_get_num_resources() - query the PCI function's available resources
  * @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 3e6d419796bc..efc5140970cd 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -6,12 +6,19 @@
 
 #include <linux/types.h>
 
+#include <uapi/linux/dlb.h>
+
 #include "dlb_hw_types.h"
 
 int dlb_resource_init(struct dlb_hw *hw);
 
 void dlb_resource_free(struct dlb_hw *hw);
 
+int dlb_hw_create_sched_domain(struct dlb_hw *hw,
+			       struct dlb_create_sched_domain_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id);
+
 int dlb_hw_get_num_resources(struct dlb_hw *hw,
 			     struct dlb_get_num_resources_args *arg,
 			     bool vdev_req, unsigned int vdev_id);
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 87ba1bfa75ed..0b152d29f9e4 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -6,6 +6,20 @@
 
 #include <linux/types.h>
 
+enum dlb_error {
+	DLB_ST_SUCCESS = 0,
+	DLB_ST_DOMAIN_UNAVAILABLE,
+	DLB_ST_LDB_PORTS_UNAVAILABLE,
+	DLB_ST_DIR_PORTS_UNAVAILABLE,
+	DLB_ST_LDB_QUEUES_UNAVAILABLE,
+	DLB_ST_LDB_CREDITS_UNAVAILABLE,
+	DLB_ST_DIR_CREDITS_UNAVAILABLE,
+	DLB_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+	DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+	DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+	DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+};
+
 struct dlb_cmd_response {
 	__u32 status; /* Interpret using enum dlb_error */
 	__u32 id;
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 06/20] dlb: add domain software reset
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (4 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 05/20] dlb: add scheduling domain configuration Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 07/20] dlb: add low-level register reset operations Mike Ximing Chen
                   ` (14 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add operation to reset a domain's resource's software state when its
reference count reaches zero, and re-inserts those resources in their
respective available-resources linked lists, for use by future scheduling
domains.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_bitmap.h   |  28 +++++
 drivers/misc/dlb/dlb_hw_types.h |   6 +
 drivers/misc/dlb/dlb_ioctl.c    |  10 +-
 drivers/misc/dlb/dlb_main.c     |  10 +-
 drivers/misc/dlb/dlb_main.h     |   2 +
 drivers/misc/dlb/dlb_pf_ops.c   |   7 ++
 drivers/misc/dlb/dlb_resource.c | 217 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |   3 +
 include/uapi/linux/dlb.h        |   1 +
 9 files changed, 282 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
index 5cebf833fab4..332135689dd9 100644
--- a/drivers/misc/dlb/dlb_bitmap.h
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -73,6 +73,34 @@ static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
 	kfree(bitmap);
 }
 
+/**
+ * dlb_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ *	    length.
+ */
+static inline int dlb_bitmap_set_range(struct dlb_bitmap *bitmap,
+				       unsigned int bit,
+				       unsigned int len)
+{
+	if (!bitmap || !bitmap->map)
+		return -EINVAL;
+
+	if (bitmap->len <= bit)
+		return -EINVAL;
+
+	bitmap_set(bitmap->map, bit, len);
+
+	return 0;
+}
+
 /**
  * dlb_bitmap_clear_range() - clear a range of bitmap entries
  * @bitmap: pointer to dlb_bitmap structure.
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index 3e03b061d5ff..c486ea344292 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -150,6 +150,12 @@ struct dlb_sn_group {
 	u32 id;
 };
 
+static inline void
+dlb_sn_group_free_slot(struct dlb_sn_group *group, int slot)
+{
+	group->slot_use_bitmap &= ~(BIT(slot));
+}
+
 struct dlb_hw_domain {
 	struct dlb_function_resources *parent_func;
 	struct list_head func_list;
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 7871d0cea118..75892966f061 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -57,13 +57,21 @@ static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg
 
 	mutex_lock(&dlb->resource_mutex);
 
+	if (dlb->domain_reset_failed) {
+		response.status = DLB_ST_DOMAIN_RESET_FAILED;
+		ret = -EINVAL;
+		goto unlock;
+	}
+
 	ret = dlb->ops->create_sched_domain(&dlb->hw, &arg, &response);
 	if (ret)
 		goto unlock;
 
 	ret = dlb_init_domain(dlb, response.id);
-	if (ret)
+	if (ret) {
+		dlb->ops->reset_domain(&dlb->hw, response.id);
 		goto unlock;
+	}
 
 	domain = dlb->sched_domains[response.id];
 
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index a4ed413eee2f..70030d779033 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -103,12 +103,20 @@ int dlb_init_domain(struct dlb *dlb, u32 domain_id)
 static int __dlb_free_domain(struct dlb_domain *domain)
 {
 	struct dlb *dlb = domain->dlb;
+	int ret;
+
+	ret = dlb->ops->reset_domain(&dlb->hw, domain->id);
+	if (ret) {
+		dlb->domain_reset_failed = true;
+		dev_err(dlb->dev,
+			"Internal error: Domain reset failed. To recover, reset the device.\n");
+	}
 
 	dlb->sched_domains[domain->id] = NULL;
 
 	kfree(domain);
 
-	return 0;
+	return ret;
 }
 
 void dlb_free_domain(struct kref *kref)
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 824416e6cdcf..ecfda11b297b 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -44,6 +44,7 @@ struct dlb_device_ops {
 				   struct dlb_cmd_response *resp);
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
+	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
@@ -70,6 +71,7 @@ struct dlb {
 	enum dlb_device_type type;
 	int id;
 	dev_t dev_number;
+	u8 domain_reset_failed;
 };
 
 /* Prototypes for dlb_ioctl.c */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index b59e9eaa600d..5dea0037d14b 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -112,6 +112,12 @@ static int dlb_pf_get_num_resources(struct dlb_hw *hw,
 	return dlb_hw_get_num_resources(hw, args, false, 0);
 }
 
+static int
+dlb_pf_reset_domain(struct dlb_hw *hw, u32 id)
+{
+	return dlb_reset_domain(hw, id, false, 0);
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -124,4 +130,5 @@ struct dlb_device_ops dlb_pf_ops = {
 	.wait_for_device_ready = dlb_pf_wait_for_device_ready,
 	.create_sched_domain = dlb_pf_create_sched_domain,
 	.get_num_resources = dlb_pf_get_num_resources,
+	.reset_domain = dlb_pf_reset_domain,
 };
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index b7df23c6a158..6d73c2479819 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -201,6 +201,29 @@ int dlb_resource_init(struct dlb_hw *hw)
 	return ret;
 }
 
+static struct dlb_hw_domain *dlb_get_domain_from_id(struct dlb_hw *hw, u32 id,
+						    bool vdev_req,
+						    unsigned int vdev_id)
+{
+	struct dlb_function_resources *rsrcs;
+	struct dlb_hw_domain *domain;
+
+	if (id >= DLB_MAX_NUM_DOMAINS)
+		return NULL;
+
+	if (!vdev_req)
+		return &hw->domains[id];
+
+	rsrcs = &hw->vdev[vdev_id];
+
+	list_for_each_entry(domain, &rsrcs->used_domains, func_list) {
+		if (domain->id.virt_id == id)
+			return domain;
+	}
+
+	return NULL;
+}
+
 static int dlb_attach_ldb_queues(struct dlb_hw *hw,
 				 struct dlb_function_resources *rsrcs,
 				 struct dlb_hw_domain *domain, u32 num_queues,
@@ -805,6 +828,200 @@ int dlb_hw_create_sched_domain(struct dlb_hw *hw,
 	return 0;
 }
 
+static int dlb_domain_reset_software_state(struct dlb_hw *hw,
+					   struct dlb_hw_domain *domain)
+{
+	struct dlb *dlb = container_of(hw, struct dlb, hw);
+	struct dlb_dir_pq_pair *tmp_dir_port;
+	struct dlb_function_resources *rsrcs;
+	struct dlb_ldb_queue *tmp_ldb_queue;
+	struct dlb_ldb_port *tmp_ldb_port;
+	struct dlb_dir_pq_pair *dir_port;
+	struct dlb_ldb_queue *ldb_queue;
+	struct dlb_ldb_port *ldb_port;
+	int ret, i;
+
+	lockdep_assert_held(&dlb->resource_mutex);
+
+	rsrcs = domain->parent_func;
+
+	/* Move the domain's ldb queues to the function's avail list */
+	list_for_each_entry_safe(ldb_queue, tmp_ldb_queue,
+				 &domain->used_ldb_queues, domain_list) {
+		if (ldb_queue->sn_cfg_valid) {
+			struct dlb_sn_group *grp;
+
+			grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+			dlb_sn_group_free_slot(grp, ldb_queue->sn_slot);
+			ldb_queue->sn_cfg_valid = false;
+		}
+
+		ldb_queue->owned = false;
+		ldb_queue->num_mappings = 0;
+		ldb_queue->num_pending_additions = 0;
+
+		list_del(&ldb_queue->domain_list);
+		list_add(&ldb_queue->func_list, &rsrcs->avail_ldb_queues);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	list_for_each_entry_safe(ldb_queue, tmp_ldb_queue,
+				 &domain->avail_ldb_queues, domain_list) {
+		ldb_queue->owned = false;
+
+		list_del(&ldb_queue->domain_list);
+		list_add(&ldb_queue->func_list, &rsrcs->avail_ldb_queues);
+		rsrcs->num_avail_ldb_queues++;
+	}
+
+	/* Move the domain's ldb ports to the function's avail list */
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry_safe(ldb_port, tmp_ldb_port,
+					 &domain->used_ldb_ports[i], domain_list) {
+			int j;
+
+			ldb_port->owned = false;
+			ldb_port->configured = false;
+			ldb_port->num_pending_removals = 0;
+			ldb_port->num_mappings = 0;
+			ldb_port->init_tkn_cnt = 0;
+			for (j = 0; j < DLB_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+				ldb_port->qid_map[j].state =
+					DLB_QUEUE_UNMAPPED;
+
+			list_del(&ldb_port->domain_list);
+			list_add(&ldb_port->func_list,
+				 &rsrcs->avail_ldb_ports[i]);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+
+		list_for_each_entry_safe(ldb_port, tmp_ldb_port,
+					 &domain->avail_ldb_ports[i], domain_list) {
+			ldb_port->owned = false;
+
+			list_del(&ldb_port->domain_list);
+			list_add(&ldb_port->func_list,
+				 &rsrcs->avail_ldb_ports[i]);
+			rsrcs->num_avail_ldb_ports[i]++;
+		}
+	}
+
+	/* Move the domain's dir ports to the function's avail list */
+	list_for_each_entry_safe(dir_port, tmp_dir_port,
+				 &domain->used_dir_pq_pairs, domain_list) {
+		dir_port->owned = false;
+		dir_port->port_configured = false;
+		dir_port->init_tkn_cnt = 0;
+
+		list_del(&dir_port->domain_list);
+
+		list_add(&dir_port->func_list, &rsrcs->avail_dir_pq_pairs);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	list_for_each_entry_safe(dir_port, tmp_dir_port,
+				 &domain->avail_dir_pq_pairs, domain_list) {
+		dir_port->owned = false;
+
+		list_del(&dir_port->domain_list);
+
+		list_add(&dir_port->func_list, &rsrcs->avail_dir_pq_pairs);
+		rsrcs->num_avail_dir_pq_pairs++;
+	}
+
+	/* Return hist list entries to the function */
+	ret = dlb_bitmap_set_range(rsrcs->avail_hist_list_entries,
+				   domain->hist_list_entry_base,
+				   domain->total_hist_list_entries);
+	if (ret) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+			   __func__);
+		return ret;
+	}
+
+	domain->total_hist_list_entries = 0;
+	domain->avail_hist_list_entries = 0;
+	domain->hist_list_entry_base = 0;
+	domain->hist_list_entry_offset = 0;
+
+	rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+	domain->num_ldb_credits = 0;
+
+	rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+	domain->num_dir_credits = 0;
+
+	rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+	rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+	domain->num_avail_aqed_entries = 0;
+	domain->num_used_aqed_entries = 0;
+
+	domain->num_pending_removals = 0;
+	domain->num_pending_additions = 0;
+	domain->configured = false;
+	domain->started = false;
+
+	/*
+	 * Move the domain out of the used_domains list and back to the
+	 * function's avail_domains list.
+	 */
+	list_del(&domain->func_list);
+	list_add(&domain->func_list, &rsrcs->avail_domains);
+	rsrcs->num_avail_domains++;
+
+	return 0;
+}
+
+static void dlb_log_reset_domain(struct dlb_hw *hw, u32 domain_id,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB reset domain:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb_reset_domain() - reset a scheduling domain
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ *	    is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
+		     unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+
+	dlb_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain || !domain->configured)
+		return -EINVAL;
+
+	return dlb_domain_reset_software_state(hw, domain);
+}
+
 /**
  * dlb_hw_get_num_resources() - query the PCI function's available resources
  * @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index efc5140970cd..8c50f449cb9b 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -19,6 +19,9 @@ int dlb_hw_create_sched_domain(struct dlb_hw *hw,
 			       struct dlb_cmd_response *resp,
 			       bool vdev_req, unsigned int vdev_id);
 
+int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
+		     unsigned int vdev_id);
+
 int dlb_hw_get_num_resources(struct dlb_hw *hw,
 			     struct dlb_get_num_resources_args *arg,
 			     bool vdev_req, unsigned int vdev_id);
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 0b152d29f9e4..0513116072a7 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -18,6 +18,7 @@ enum dlb_error {
 	DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
 	DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
 	DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+	DLB_ST_DOMAIN_RESET_FAILED,
 };
 
 struct dlb_cmd_response {
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 07/20] dlb: add low-level register reset operations
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (5 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 06/20] dlb: add domain software reset Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 08/20] dlb: add runtime power-management support Mike Ximing Chen
                   ` (13 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Program all registers used to configure the domain's resources back to
their reset values during scheduling domain reset. This ensures the device
is in a known good state if/when it is configured again in the future.

Additional work is required if a resource is in-use (e.g. a queue is
non-empty) at that time. Support for these cases will be added in
subsequent commits.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_regs.h     | 3527 ++++++++++++++++++++++++++++++-
 drivers/misc/dlb/dlb_resource.c |  387 ++++
 2 files changed, 3902 insertions(+), 12 deletions(-)

diff --git a/drivers/misc/dlb/dlb_regs.h b/drivers/misc/dlb/dlb_regs.h
index 0fd499f384de..5e4609504a01 100644
--- a/drivers/misc/dlb/dlb_regs.h
+++ b/drivers/misc/dlb/dlb_regs.h
@@ -4,25 +4,3475 @@
 #ifndef __DLB_REGS_H
 #define __DLB_REGS_H
 
-#include <linux/types.h>
+#define PF_VF2PF_MAILBOX_BYTES 256
+#define PF_VF2PF_MAILBOX(vf_id, x) \
+	(0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define PF_VF2PF_MAILBOX_RST 0x0
+
+#define PF_VF2PF_MAILBOX_MSG	0xFFFFFFFF
+#define PF_VF2PF_MAILBOX_MSG_LOC	0
+
+#define PF_VF2PF_MAILBOX_ISR(vf_id) \
+	(0x1f00 + (vf_id) * 0x10000)
+#define PF_VF2PF_MAILBOX_ISR_RST 0x0
+
+#define PF_VF2PF_MAILBOX_ISR_VF0_ISR	0x00000001
+#define PF_VF2PF_MAILBOX_ISR_VF1_ISR	0x00000002
+#define PF_VF2PF_MAILBOX_ISR_VF2_ISR	0x00000004
+#define PF_VF2PF_MAILBOX_ISR_VF3_ISR	0x00000008
+#define PF_VF2PF_MAILBOX_ISR_VF4_ISR	0x00000010
+#define PF_VF2PF_MAILBOX_ISR_VF5_ISR	0x00000020
+#define PF_VF2PF_MAILBOX_ISR_VF6_ISR	0x00000040
+#define PF_VF2PF_MAILBOX_ISR_VF7_ISR	0x00000080
+#define PF_VF2PF_MAILBOX_ISR_VF8_ISR	0x00000100
+#define PF_VF2PF_MAILBOX_ISR_VF9_ISR	0x00000200
+#define PF_VF2PF_MAILBOX_ISR_VF10_ISR	0x00000400
+#define PF_VF2PF_MAILBOX_ISR_VF11_ISR	0x00000800
+#define PF_VF2PF_MAILBOX_ISR_VF12_ISR	0x00001000
+#define PF_VF2PF_MAILBOX_ISR_VF13_ISR	0x00002000
+#define PF_VF2PF_MAILBOX_ISR_VF14_ISR	0x00004000
+#define PF_VF2PF_MAILBOX_ISR_VF15_ISR	0x00008000
+#define PF_VF2PF_MAILBOX_ISR_RSVD0	0xFFFF0000
+#define PF_VF2PF_MAILBOX_ISR_VF0_ISR_LOC	0
+#define PF_VF2PF_MAILBOX_ISR_VF1_ISR_LOC	1
+#define PF_VF2PF_MAILBOX_ISR_VF2_ISR_LOC	2
+#define PF_VF2PF_MAILBOX_ISR_VF3_ISR_LOC	3
+#define PF_VF2PF_MAILBOX_ISR_VF4_ISR_LOC	4
+#define PF_VF2PF_MAILBOX_ISR_VF5_ISR_LOC	5
+#define PF_VF2PF_MAILBOX_ISR_VF6_ISR_LOC	6
+#define PF_VF2PF_MAILBOX_ISR_VF7_ISR_LOC	7
+#define PF_VF2PF_MAILBOX_ISR_VF8_ISR_LOC	8
+#define PF_VF2PF_MAILBOX_ISR_VF9_ISR_LOC	9
+#define PF_VF2PF_MAILBOX_ISR_VF10_ISR_LOC	10
+#define PF_VF2PF_MAILBOX_ISR_VF11_ISR_LOC	11
+#define PF_VF2PF_MAILBOX_ISR_VF12_ISR_LOC	12
+#define PF_VF2PF_MAILBOX_ISR_VF13_ISR_LOC	13
+#define PF_VF2PF_MAILBOX_ISR_VF14_ISR_LOC	14
+#define PF_VF2PF_MAILBOX_ISR_VF15_ISR_LOC	15
+#define PF_VF2PF_MAILBOX_ISR_RSVD0_LOC		16
+
+#define PF_VF2PF_FLR_ISR(vf_id) \
+	(0x1f04 + (vf_id) * 0x10000)
+#define PF_VF2PF_FLR_ISR_RST 0x0
+
+#define PF_VF2PF_FLR_ISR_VF0_ISR	0x00000001
+#define PF_VF2PF_FLR_ISR_VF1_ISR	0x00000002
+#define PF_VF2PF_FLR_ISR_VF2_ISR	0x00000004
+#define PF_VF2PF_FLR_ISR_VF3_ISR	0x00000008
+#define PF_VF2PF_FLR_ISR_VF4_ISR	0x00000010
+#define PF_VF2PF_FLR_ISR_VF5_ISR	0x00000020
+#define PF_VF2PF_FLR_ISR_VF6_ISR	0x00000040
+#define PF_VF2PF_FLR_ISR_VF7_ISR	0x00000080
+#define PF_VF2PF_FLR_ISR_VF8_ISR	0x00000100
+#define PF_VF2PF_FLR_ISR_VF9_ISR	0x00000200
+#define PF_VF2PF_FLR_ISR_VF10_ISR	0x00000400
+#define PF_VF2PF_FLR_ISR_VF11_ISR	0x00000800
+#define PF_VF2PF_FLR_ISR_VF12_ISR	0x00001000
+#define PF_VF2PF_FLR_ISR_VF13_ISR	0x00002000
+#define PF_VF2PF_FLR_ISR_VF14_ISR	0x00004000
+#define PF_VF2PF_FLR_ISR_VF15_ISR	0x00008000
+#define PF_VF2PF_FLR_ISR_RSVD0		0xFFFF0000
+#define PF_VF2PF_FLR_ISR_VF0_ISR_LOC	0
+#define PF_VF2PF_FLR_ISR_VF1_ISR_LOC	1
+#define PF_VF2PF_FLR_ISR_VF2_ISR_LOC	2
+#define PF_VF2PF_FLR_ISR_VF3_ISR_LOC	3
+#define PF_VF2PF_FLR_ISR_VF4_ISR_LOC	4
+#define PF_VF2PF_FLR_ISR_VF5_ISR_LOC	5
+#define PF_VF2PF_FLR_ISR_VF6_ISR_LOC	6
+#define PF_VF2PF_FLR_ISR_VF7_ISR_LOC	7
+#define PF_VF2PF_FLR_ISR_VF8_ISR_LOC	8
+#define PF_VF2PF_FLR_ISR_VF9_ISR_LOC	9
+#define PF_VF2PF_FLR_ISR_VF10_ISR_LOC	10
+#define PF_VF2PF_FLR_ISR_VF11_ISR_LOC	11
+#define PF_VF2PF_FLR_ISR_VF12_ISR_LOC	12
+#define PF_VF2PF_FLR_ISR_VF13_ISR_LOC	13
+#define PF_VF2PF_FLR_ISR_VF14_ISR_LOC	14
+#define PF_VF2PF_FLR_ISR_VF15_ISR_LOC	15
+#define PF_VF2PF_FLR_ISR_RSVD0_LOC	16
+
+#define PF_VF2PF_ISR_PEND(vf_id) \
+	(0x1f10 + (vf_id) * 0x10000)
+#define PF_VF2PF_ISR_PEND_RST 0x0
+
+#define PF_VF2PF_ISR_PEND_ISR_PEND	0x00000001
+#define PF_VF2PF_ISR_PEND_RSVD0		0xFFFFFFFE
+#define PF_VF2PF_ISR_PEND_ISR_PEND_LOC	0
+#define PF_VF2PF_ISR_PEND_RSVD0_LOC	1
+
+#define PF_PF2VF_MAILBOX_BYTES 64
+#define PF_PF2VF_MAILBOX(vf_id, x) \
+	(0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define PF_PF2VF_MAILBOX_RST 0x0
+
+#define PF_PF2VF_MAILBOX_MSG	0xFFFFFFFF
+#define PF_PF2VF_MAILBOX_MSG_LOC	0
+
+#define PF_PF2VF_MAILBOX_ISR(vf_id) \
+	(0x2f00 + (vf_id) * 0x10000)
+#define PF_PF2VF_MAILBOX_ISR_RST 0x0
+
+#define PF_PF2VF_MAILBOX_ISR_VF0_ISR	0x00000001
+#define PF_PF2VF_MAILBOX_ISR_VF1_ISR	0x00000002
+#define PF_PF2VF_MAILBOX_ISR_VF2_ISR	0x00000004
+#define PF_PF2VF_MAILBOX_ISR_VF3_ISR	0x00000008
+#define PF_PF2VF_MAILBOX_ISR_VF4_ISR	0x00000010
+#define PF_PF2VF_MAILBOX_ISR_VF5_ISR	0x00000020
+#define PF_PF2VF_MAILBOX_ISR_VF6_ISR	0x00000040
+#define PF_PF2VF_MAILBOX_ISR_VF7_ISR	0x00000080
+#define PF_PF2VF_MAILBOX_ISR_VF8_ISR	0x00000100
+#define PF_PF2VF_MAILBOX_ISR_VF9_ISR	0x00000200
+#define PF_PF2VF_MAILBOX_ISR_VF10_ISR	0x00000400
+#define PF_PF2VF_MAILBOX_ISR_VF11_ISR	0x00000800
+#define PF_PF2VF_MAILBOX_ISR_VF12_ISR	0x00001000
+#define PF_PF2VF_MAILBOX_ISR_VF13_ISR	0x00002000
+#define PF_PF2VF_MAILBOX_ISR_VF14_ISR	0x00004000
+#define PF_PF2VF_MAILBOX_ISR_VF15_ISR	0x00008000
+#define PF_PF2VF_MAILBOX_ISR_RSVD0	0xFFFF0000
+#define PF_PF2VF_MAILBOX_ISR_VF0_ISR_LOC	0
+#define PF_PF2VF_MAILBOX_ISR_VF1_ISR_LOC	1
+#define PF_PF2VF_MAILBOX_ISR_VF2_ISR_LOC	2
+#define PF_PF2VF_MAILBOX_ISR_VF3_ISR_LOC	3
+#define PF_PF2VF_MAILBOX_ISR_VF4_ISR_LOC	4
+#define PF_PF2VF_MAILBOX_ISR_VF5_ISR_LOC	5
+#define PF_PF2VF_MAILBOX_ISR_VF6_ISR_LOC	6
+#define PF_PF2VF_MAILBOX_ISR_VF7_ISR_LOC	7
+#define PF_PF2VF_MAILBOX_ISR_VF8_ISR_LOC	8
+#define PF_PF2VF_MAILBOX_ISR_VF9_ISR_LOC	9
+#define PF_PF2VF_MAILBOX_ISR_VF10_ISR_LOC	10
+#define PF_PF2VF_MAILBOX_ISR_VF11_ISR_LOC	11
+#define PF_PF2VF_MAILBOX_ISR_VF12_ISR_LOC	12
+#define PF_PF2VF_MAILBOX_ISR_VF13_ISR_LOC	13
+#define PF_PF2VF_MAILBOX_ISR_VF14_ISR_LOC	14
+#define PF_PF2VF_MAILBOX_ISR_VF15_ISR_LOC	15
+#define PF_PF2VF_MAILBOX_ISR_RSVD0_LOC		16
+
+#define PF_VF_RESET_IN_PROGRESS(vf_id) \
+	(0x3000 + (vf_id) * 0x10000)
+#define PF_VF_RESET_IN_PROGRESS_RST 0xffff
+
+#define PF_VF_RESET_IN_PROGRESS_VF0_RESET_IN_PROGRESS	0x00000001
+#define PF_VF_RESET_IN_PROGRESS_VF1_RESET_IN_PROGRESS	0x00000002
+#define PF_VF_RESET_IN_PROGRESS_VF2_RESET_IN_PROGRESS	0x00000004
+#define PF_VF_RESET_IN_PROGRESS_VF3_RESET_IN_PROGRESS	0x00000008
+#define PF_VF_RESET_IN_PROGRESS_VF4_RESET_IN_PROGRESS	0x00000010
+#define PF_VF_RESET_IN_PROGRESS_VF5_RESET_IN_PROGRESS	0x00000020
+#define PF_VF_RESET_IN_PROGRESS_VF6_RESET_IN_PROGRESS	0x00000040
+#define PF_VF_RESET_IN_PROGRESS_VF7_RESET_IN_PROGRESS	0x00000080
+#define PF_VF_RESET_IN_PROGRESS_VF8_RESET_IN_PROGRESS	0x00000100
+#define PF_VF_RESET_IN_PROGRESS_VF9_RESET_IN_PROGRESS	0x00000200
+#define PF_VF_RESET_IN_PROGRESS_VF10_RESET_IN_PROGRESS	0x00000400
+#define PF_VF_RESET_IN_PROGRESS_VF11_RESET_IN_PROGRESS	0x00000800
+#define PF_VF_RESET_IN_PROGRESS_VF12_RESET_IN_PROGRESS	0x00001000
+#define PF_VF_RESET_IN_PROGRESS_VF13_RESET_IN_PROGRESS	0x00002000
+#define PF_VF_RESET_IN_PROGRESS_VF14_RESET_IN_PROGRESS	0x00004000
+#define PF_VF_RESET_IN_PROGRESS_VF15_RESET_IN_PROGRESS	0x00008000
+#define PF_VF_RESET_IN_PROGRESS_RSVD0			0xFFFF0000
+#define PF_VF_RESET_IN_PROGRESS_VF0_RESET_IN_PROGRESS_LOC	0
+#define PF_VF_RESET_IN_PROGRESS_VF1_RESET_IN_PROGRESS_LOC	1
+#define PF_VF_RESET_IN_PROGRESS_VF2_RESET_IN_PROGRESS_LOC	2
+#define PF_VF_RESET_IN_PROGRESS_VF3_RESET_IN_PROGRESS_LOC	3
+#define PF_VF_RESET_IN_PROGRESS_VF4_RESET_IN_PROGRESS_LOC	4
+#define PF_VF_RESET_IN_PROGRESS_VF5_RESET_IN_PROGRESS_LOC	5
+#define PF_VF_RESET_IN_PROGRESS_VF6_RESET_IN_PROGRESS_LOC	6
+#define PF_VF_RESET_IN_PROGRESS_VF7_RESET_IN_PROGRESS_LOC	7
+#define PF_VF_RESET_IN_PROGRESS_VF8_RESET_IN_PROGRESS_LOC	8
+#define PF_VF_RESET_IN_PROGRESS_VF9_RESET_IN_PROGRESS_LOC	9
+#define PF_VF_RESET_IN_PROGRESS_VF10_RESET_IN_PROGRESS_LOC	10
+#define PF_VF_RESET_IN_PROGRESS_VF11_RESET_IN_PROGRESS_LOC	11
+#define PF_VF_RESET_IN_PROGRESS_VF12_RESET_IN_PROGRESS_LOC	12
+#define PF_VF_RESET_IN_PROGRESS_VF13_RESET_IN_PROGRESS_LOC	13
+#define PF_VF_RESET_IN_PROGRESS_VF14_RESET_IN_PROGRESS_LOC	14
+#define PF_VF_RESET_IN_PROGRESS_VF15_RESET_IN_PROGRESS_LOC	15
+#define PF_VF_RESET_IN_PROGRESS_RSVD0_LOC			16
+
+#define MSIX_VECTOR_CTRL(x) \
+	(0x100000c + (x) * 0x10)
+#define MSIX_VECTOR_CTRL_RST 0x1
+
+#define MSIX_VECTOR_CTRL_VEC_MASK	0x00000001
+#define MSIX_VECTOR_CTRL_RSVD0		0xFFFFFFFE
+#define MSIX_VECTOR_CTRL_VEC_MASK_LOC	0
+#define MSIX_VECTOR_CTRL_RSVD0_LOC	1
+
+#define IOSF_SMON_COMP_MASK1(x) \
+	(0x8002024 + (x) * 0x40)
+#define IOSF_SMON_COMP_MASK1_RST 0xffffffff
+
+#define IOSF_SMON_COMP_MASK1_COMP_MASK1	0xFFFFFFFF
+#define IOSF_SMON_COMP_MASK1_COMP_MASK1_LOC	0
+
+#define IOSF_SMON_COMP_MASK0(x) \
+	(0x8002020 + (x) * 0x40)
+#define IOSF_SMON_COMP_MASK0_RST 0xffffffff
+
+#define IOSF_SMON_COMP_MASK0_COMP_MASK0	0xFFFFFFFF
+#define IOSF_SMON_COMP_MASK0_COMP_MASK0_LOC	0
+
+#define IOSF_SMON_MAX_TMR(x) \
+	(0x800201c + (x) * 0x40)
+#define IOSF_SMON_MAX_TMR_RST 0x0
+
+#define IOSF_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define IOSF_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define IOSF_SMON_TMR(x) \
+	(0x8002018 + (x) * 0x40)
+#define IOSF_SMON_TMR_RST 0x0
+
+#define IOSF_SMON_TMR_TIMER_VAL	0xFFFFFFFF
+#define IOSF_SMON_TMR_TIMER_VAL_LOC	0
+
+#define IOSF_SMON_ACTIVITYCNTR1(x) \
+	(0x8002014 + (x) * 0x40)
+#define IOSF_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define IOSF_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define IOSF_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define IOSF_SMON_ACTIVITYCNTR0(x) \
+	(0x8002010 + (x) * 0x40)
+#define IOSF_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define IOSF_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define IOSF_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define IOSF_SMON_COMPARE1(x) \
+	(0x800200c + (x) * 0x40)
+#define IOSF_SMON_COMPARE1_RST 0x0
+
+#define IOSF_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define IOSF_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define IOSF_SMON_COMPARE0(x) \
+	(0x8002008 + (x) * 0x40)
+#define IOSF_SMON_COMPARE0_RST 0x0
+
+#define IOSF_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define IOSF_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define IOSF_SMON_CFG1(x) \
+	(0x8002004 + (x) * 0x40)
+#define IOSF_SMON_CFG1_RST 0x0
+
+#define IOSF_SMON_CFG1_MODE0	0x000000FF
+#define IOSF_SMON_CFG1_MODE1	0x0000FF00
+#define IOSF_SMON_CFG1_RSVD	0xFFFF0000
+#define IOSF_SMON_CFG1_MODE0_LOC	0
+#define IOSF_SMON_CFG1_MODE1_LOC	8
+#define IOSF_SMON_CFG1_RSVD_LOC		16
+
+#define IOSF_SMON_CFG0(x) \
+	(0x8002000 + (x) * 0x40)
+#define IOSF_SMON_CFG0_RST 0x40000000
+
+#define IOSF_SMON_CFG0_SMON_ENABLE		0x00000001
+#define IOSF_SMON_CFG0_RSVD2			0x0000000E
+#define IOSF_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define IOSF_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define IOSF_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define IOSF_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define IOSF_SMON_CFG0_SMON_MODE		0x0000F000
+#define IOSF_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define IOSF_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define IOSF_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define IOSF_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define IOSF_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define IOSF_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define IOSF_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define IOSF_SMON_CFG0_RSVD1			0x00800000
+#define IOSF_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define IOSF_SMON_CFG0_RSVD0			0x20000000
+#define IOSF_SMON_CFG0_VERSION			0xC0000000
+#define IOSF_SMON_CFG0_SMON_ENABLE_LOC			0
+#define IOSF_SMON_CFG0_RSVD2_LOC			1
+#define IOSF_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define IOSF_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define IOSF_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define IOSF_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define IOSF_SMON_CFG0_SMON_MODE_LOC			12
+#define IOSF_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define IOSF_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define IOSF_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define IOSF_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define IOSF_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define IOSF_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define IOSF_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define IOSF_SMON_CFG0_RSVD1_LOC			23
+#define IOSF_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define IOSF_SMON_CFG0_RSVD0_LOC			29
+#define IOSF_SMON_CFG0_VERSION_LOC			30
+
+#define IOSF_FUNC_VF_BAR_DSBL(x) \
+	(0x20 + (x) * 0x4)
+#define IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+
+#define IOSF_FUNC_VF_BAR_DSBL_FUNC_VF_BAR_DIS	0x00000001
+#define IOSF_FUNC_VF_BAR_DSBL_RSVD0		0xFFFFFFFE
+#define IOSF_FUNC_VF_BAR_DSBL_FUNC_VF_BAR_DIS_LOC	0
+#define IOSF_FUNC_VF_BAR_DSBL_RSVD0_LOC			1
+
+#define SYS_TOTAL_VAS 0x1000011c
+#define SYS_TOTAL_VAS_RST 0x20
+
+#define SYS_TOTAL_VAS_TOTAL_VAS	0xFFFFFFFF
+#define SYS_TOTAL_VAS_TOTAL_VAS_LOC	0
+
+#define SYS_TOTAL_DIR_PORTS 0x10000118
+#define SYS_TOTAL_DIR_PORTS_RST 0x40
+
+#define SYS_TOTAL_DIR_PORTS_TOTAL_DIR_PORTS	0xFFFFFFFF
+#define SYS_TOTAL_DIR_PORTS_TOTAL_DIR_PORTS_LOC	0
+
+#define SYS_TOTAL_LDB_PORTS 0x10000114
+#define SYS_TOTAL_LDB_PORTS_RST 0x40
+
+#define SYS_TOTAL_LDB_PORTS_TOTAL_LDB_PORTS	0xFFFFFFFF
+#define SYS_TOTAL_LDB_PORTS_TOTAL_LDB_PORTS_LOC	0
+
+#define SYS_TOTAL_DIR_QID 0x10000110
+#define SYS_TOTAL_DIR_QID_RST 0x40
+
+#define SYS_TOTAL_DIR_QID_TOTAL_DIR_QID	0xFFFFFFFF
+#define SYS_TOTAL_DIR_QID_TOTAL_DIR_QID_LOC	0
+
+#define SYS_TOTAL_LDB_QID 0x1000010c
+#define SYS_TOTAL_LDB_QID_RST 0x20
+
+#define SYS_TOTAL_LDB_QID_TOTAL_LDB_QID	0xFFFFFFFF
+#define SYS_TOTAL_LDB_QID_TOTAL_LDB_QID_LOC	0
+
+#define SYS_TOTAL_DIR_CRDS 0x10000108
+#define SYS_TOTAL_DIR_CRDS_RST 0x1000
+
+#define SYS_TOTAL_DIR_CRDS_TOTAL_DIR_CREDITS	0xFFFFFFFF
+#define SYS_TOTAL_DIR_CRDS_TOTAL_DIR_CREDITS_LOC	0
+
+#define SYS_TOTAL_LDB_CRDS 0x10000104
+#define SYS_TOTAL_LDB_CRDS_RST 0x2000
+
+#define SYS_TOTAL_LDB_CRDS_TOTAL_LDB_CREDITS	0xFFFFFFFF
+#define SYS_TOTAL_LDB_CRDS_TOTAL_LDB_CREDITS_LOC	0
+
+#define SYS_ALARM_PF_SYND2 0x10000508
+#define SYS_ALARM_PF_SYND2_RST 0x0
+
+#define SYS_ALARM_PF_SYND2_LOCK_ID	0x0000FFFF
+#define SYS_ALARM_PF_SYND2_MEAS		0x00010000
+#define SYS_ALARM_PF_SYND2_DEBUG	0x00FE0000
+#define SYS_ALARM_PF_SYND2_CQ_POP	0x01000000
+#define SYS_ALARM_PF_SYND2_QE_UHL	0x02000000
+#define SYS_ALARM_PF_SYND2_QE_ORSP	0x04000000
+#define SYS_ALARM_PF_SYND2_QE_VALID	0x08000000
+#define SYS_ALARM_PF_SYND2_CQ_INT_REARM	0x10000000
+#define SYS_ALARM_PF_SYND2_DSI_ERROR	0x20000000
+#define SYS_ALARM_PF_SYND2_RSVD0	0xC0000000
+#define SYS_ALARM_PF_SYND2_LOCK_ID_LOC		0
+#define SYS_ALARM_PF_SYND2_MEAS_LOC		16
+#define SYS_ALARM_PF_SYND2_DEBUG_LOC		17
+#define SYS_ALARM_PF_SYND2_CQ_POP_LOC		24
+#define SYS_ALARM_PF_SYND2_QE_UHL_LOC		25
+#define SYS_ALARM_PF_SYND2_QE_ORSP_LOC		26
+#define SYS_ALARM_PF_SYND2_QE_VALID_LOC		27
+#define SYS_ALARM_PF_SYND2_CQ_INT_REARM_LOC	28
+#define SYS_ALARM_PF_SYND2_DSI_ERROR_LOC	29
+#define SYS_ALARM_PF_SYND2_RSVD0_LOC		30
+
+#define SYS_ALARM_PF_SYND1 0x10000504
+#define SYS_ALARM_PF_SYND1_RST 0x0
+
+#define SYS_ALARM_PF_SYND1_DSI		0x0000FFFF
+#define SYS_ALARM_PF_SYND1_QID		0x00FF0000
+#define SYS_ALARM_PF_SYND1_QTYPE	0x03000000
+#define SYS_ALARM_PF_SYND1_QPRI		0x1C000000
+#define SYS_ALARM_PF_SYND1_MSG_TYPE	0xE0000000
+#define SYS_ALARM_PF_SYND1_DSI_LOC	0
+#define SYS_ALARM_PF_SYND1_QID_LOC	16
+#define SYS_ALARM_PF_SYND1_QTYPE_LOC	24
+#define SYS_ALARM_PF_SYND1_QPRI_LOC	26
+#define SYS_ALARM_PF_SYND1_MSG_TYPE_LOC	29
+
+#define SYS_ALARM_PF_SYND0 0x10000500
+#define SYS_ALARM_PF_SYND0_RST 0x0
+
+#define SYS_ALARM_PF_SYND0_SYNDROME	0x000000FF
+#define SYS_ALARM_PF_SYND0_RTYPE	0x00000300
+#define SYS_ALARM_PF_SYND0_RSVD0	0x00001C00
+#define SYS_ALARM_PF_SYND0_IS_LDB	0x00002000
+#define SYS_ALARM_PF_SYND0_CLS		0x0000C000
+#define SYS_ALARM_PF_SYND0_AID		0x003F0000
+#define SYS_ALARM_PF_SYND0_UNIT		0x03C00000
+#define SYS_ALARM_PF_SYND0_SOURCE	0x3C000000
+#define SYS_ALARM_PF_SYND0_MORE		0x40000000
+#define SYS_ALARM_PF_SYND0_VALID	0x80000000
+#define SYS_ALARM_PF_SYND0_SYNDROME_LOC	0
+#define SYS_ALARM_PF_SYND0_RTYPE_LOC	8
+#define SYS_ALARM_PF_SYND0_RSVD0_LOC	10
+#define SYS_ALARM_PF_SYND0_IS_LDB_LOC	13
+#define SYS_ALARM_PF_SYND0_CLS_LOC	14
+#define SYS_ALARM_PF_SYND0_AID_LOC	16
+#define SYS_ALARM_PF_SYND0_UNIT_LOC	22
+#define SYS_ALARM_PF_SYND0_SOURCE_LOC	26
+#define SYS_ALARM_PF_SYND0_MORE_LOC	30
+#define SYS_ALARM_PF_SYND0_VALID_LOC	31
+
+#define SYS_VF_LDB_VPP_V(x) \
+	(0x10000f00 + (x) * 0x1000)
+#define SYS_VF_LDB_VPP_V_RST 0x0
+
+#define SYS_VF_LDB_VPP_V_VPP_V	0x00000001
+#define SYS_VF_LDB_VPP_V_RSVD0	0xFFFFFFFE
+#define SYS_VF_LDB_VPP_V_VPP_V_LOC	0
+#define SYS_VF_LDB_VPP_V_RSVD0_LOC	1
+
+#define SYS_VF_LDB_VPP2PP(x) \
+	(0x10000f04 + (x) * 0x1000)
+#define SYS_VF_LDB_VPP2PP_RST 0x0
+
+#define SYS_VF_LDB_VPP2PP_PP	0x0000003F
+#define SYS_VF_LDB_VPP2PP_RSVD0	0xFFFFFFC0
+#define SYS_VF_LDB_VPP2PP_PP_LOC	0
+#define SYS_VF_LDB_VPP2PP_RSVD0_LOC	6
+
+#define SYS_VF_DIR_VPP_V(x) \
+	(0x10000f08 + (x) * 0x1000)
+#define SYS_VF_DIR_VPP_V_RST 0x0
+
+#define SYS_VF_DIR_VPP_V_VPP_V	0x00000001
+#define SYS_VF_DIR_VPP_V_RSVD0	0xFFFFFFFE
+#define SYS_VF_DIR_VPP_V_VPP_V_LOC	0
+#define SYS_VF_DIR_VPP_V_RSVD0_LOC	1
+
+#define SYS_VF_DIR_VPP2PP(x) \
+	(0x10000f0c + (x) * 0x1000)
+#define SYS_VF_DIR_VPP2PP_RST 0x0
+
+#define SYS_VF_DIR_VPP2PP_PP	0x0000003F
+#define SYS_VF_DIR_VPP2PP_RSVD0	0xFFFFFFC0
+#define SYS_VF_DIR_VPP2PP_PP_LOC	0
+#define SYS_VF_DIR_VPP2PP_RSVD0_LOC	6
+
+#define SYS_VF_LDB_VQID_V(x) \
+	(0x10000f10 + (x) * 0x1000)
+#define SYS_VF_LDB_VQID_V_RST 0x0
+
+#define SYS_VF_LDB_VQID_V_VQID_V	0x00000001
+#define SYS_VF_LDB_VQID_V_RSVD0		0xFFFFFFFE
+#define SYS_VF_LDB_VQID_V_VQID_V_LOC	0
+#define SYS_VF_LDB_VQID_V_RSVD0_LOC	1
+
+#define SYS_VF_LDB_VQID2QID(x) \
+	(0x10000f14 + (x) * 0x1000)
+#define SYS_VF_LDB_VQID2QID_RST 0x0
+
+#define SYS_VF_LDB_VQID2QID_QID		0x0000001F
+#define SYS_VF_LDB_VQID2QID_RSVD0	0xFFFFFFE0
+#define SYS_VF_LDB_VQID2QID_QID_LOC	0
+#define SYS_VF_LDB_VQID2QID_RSVD0_LOC	5
+
+#define SYS_LDB_QID2VQID(x) \
+	(0x10000f18 + (x) * 0x1000)
+#define SYS_LDB_QID2VQID_RST 0x0
+
+#define SYS_LDB_QID2VQID_VQID	0x0000001F
+#define SYS_LDB_QID2VQID_RSVD0	0xFFFFFFE0
+#define SYS_LDB_QID2VQID_VQID_LOC	0
+#define SYS_LDB_QID2VQID_RSVD0_LOC	5
+
+#define SYS_VF_DIR_VQID_V(x) \
+	(0x10000f1c + (x) * 0x1000)
+#define SYS_VF_DIR_VQID_V_RST 0x0
+
+#define SYS_VF_DIR_VQID_V_VQID_V	0x00000001
+#define SYS_VF_DIR_VQID_V_RSVD0		0xFFFFFFFE
+#define SYS_VF_DIR_VQID_V_VQID_V_LOC	0
+#define SYS_VF_DIR_VQID_V_RSVD0_LOC	1
+
+#define SYS_VF_DIR_VQID2QID(x) \
+	(0x10000f20 + (x) * 0x1000)
+#define SYS_VF_DIR_VQID2QID_RST 0x0
+
+#define SYS_VF_DIR_VQID2QID_QID		0x0000003F
+#define SYS_VF_DIR_VQID2QID_RSVD0	0xFFFFFFC0
+#define SYS_VF_DIR_VQID2QID_QID_LOC	0
+#define SYS_VF_DIR_VQID2QID_RSVD0_LOC	6
+
+#define SYS_LDB_VASQID_V(x) \
+	(0x10000f24 + (x) * 0x1000)
+#define SYS_LDB_VASQID_V_RST 0x0
+
+#define SYS_LDB_VASQID_V_VASQID_V	0x00000001
+#define SYS_LDB_VASQID_V_RSVD0		0xFFFFFFFE
+#define SYS_LDB_VASQID_V_VASQID_V_LOC	0
+#define SYS_LDB_VASQID_V_RSVD0_LOC	1
+
+#define SYS_DIR_VASQID_V(x) \
+	(0x10000f28 + (x) * 0x1000)
+#define SYS_DIR_VASQID_V_RST 0x0
+
+#define SYS_DIR_VASQID_V_VASQID_V	0x00000001
+#define SYS_DIR_VASQID_V_RSVD0		0xFFFFFFFE
+#define SYS_DIR_VASQID_V_VASQID_V_LOC	0
+#define SYS_DIR_VASQID_V_RSVD0_LOC	1
+
+#define SYS_ALARM_VF_SYND2(x) \
+	(0x10000f48 + (x) * 0x1000)
+#define SYS_ALARM_VF_SYND2_RST 0x0
+
+#define SYS_ALARM_VF_SYND2_LOCK_ID	0x0000FFFF
+#define SYS_ALARM_VF_SYND2_DEBUG	0x00FF0000
+#define SYS_ALARM_VF_SYND2_CQ_POP	0x01000000
+#define SYS_ALARM_VF_SYND2_QE_UHL	0x02000000
+#define SYS_ALARM_VF_SYND2_QE_ORSP	0x04000000
+#define SYS_ALARM_VF_SYND2_QE_VALID	0x08000000
+#define SYS_ALARM_VF_SYND2_ISZ		0x10000000
+#define SYS_ALARM_VF_SYND2_DSI_ERROR	0x20000000
+#define SYS_ALARM_VF_SYND2_DLBRSVD	0xC0000000
+#define SYS_ALARM_VF_SYND2_LOCK_ID_LOC		0
+#define SYS_ALARM_VF_SYND2_DEBUG_LOC		16
+#define SYS_ALARM_VF_SYND2_CQ_POP_LOC		24
+#define SYS_ALARM_VF_SYND2_QE_UHL_LOC		25
+#define SYS_ALARM_VF_SYND2_QE_ORSP_LOC		26
+#define SYS_ALARM_VF_SYND2_QE_VALID_LOC		27
+#define SYS_ALARM_VF_SYND2_ISZ_LOC		28
+#define SYS_ALARM_VF_SYND2_DSI_ERROR_LOC	29
+#define SYS_ALARM_VF_SYND2_DLBRSVD_LOC		30
+
+#define SYS_ALARM_VF_SYND1(x) \
+	(0x10000f44 + (x) * 0x1000)
+#define SYS_ALARM_VF_SYND1_RST 0x0
+
+#define SYS_ALARM_VF_SYND1_DSI		0x0000FFFF
+#define SYS_ALARM_VF_SYND1_QID		0x00FF0000
+#define SYS_ALARM_VF_SYND1_QTYPE	0x03000000
+#define SYS_ALARM_VF_SYND1_QPRI		0x1C000000
+#define SYS_ALARM_VF_SYND1_MSG_TYPE	0xE0000000
+#define SYS_ALARM_VF_SYND1_DSI_LOC	0
+#define SYS_ALARM_VF_SYND1_QID_LOC	16
+#define SYS_ALARM_VF_SYND1_QTYPE_LOC	24
+#define SYS_ALARM_VF_SYND1_QPRI_LOC	26
+#define SYS_ALARM_VF_SYND1_MSG_TYPE_LOC	29
+
+#define SYS_ALARM_VF_SYND0(x) \
+	(0x10000f40 + (x) * 0x1000)
+#define SYS_ALARM_VF_SYND0_RST 0x0
+
+#define SYS_ALARM_VF_SYND0_SYNDROME		0x000000FF
+#define SYS_ALARM_VF_SYND0_RTYPE		0x00000300
+#define SYS_ALARM_VF_SYND0_VF_SYND0_PARITY	0x00000400
+#define SYS_ALARM_VF_SYND0_VF_SYND1_PARITY	0x00000800
+#define SYS_ALARM_VF_SYND0_VF_SYND2_PARITY	0x00001000
+#define SYS_ALARM_VF_SYND0_IS_LDB		0x00002000
+#define SYS_ALARM_VF_SYND0_CLS			0x0000C000
+#define SYS_ALARM_VF_SYND0_AID			0x003F0000
+#define SYS_ALARM_VF_SYND0_UNIT			0x03C00000
+#define SYS_ALARM_VF_SYND0_SOURCE		0x3C000000
+#define SYS_ALARM_VF_SYND0_MORE			0x40000000
+#define SYS_ALARM_VF_SYND0_VALID		0x80000000
+#define SYS_ALARM_VF_SYND0_SYNDROME_LOC		0
+#define SYS_ALARM_VF_SYND0_RTYPE_LOC		8
+#define SYS_ALARM_VF_SYND0_VF_SYND0_PARITY_LOC	10
+#define SYS_ALARM_VF_SYND0_VF_SYND1_PARITY_LOC	11
+#define SYS_ALARM_VF_SYND0_VF_SYND2_PARITY_LOC	12
+#define SYS_ALARM_VF_SYND0_IS_LDB_LOC		13
+#define SYS_ALARM_VF_SYND0_CLS_LOC		14
+#define SYS_ALARM_VF_SYND0_AID_LOC		16
+#define SYS_ALARM_VF_SYND0_UNIT_LOC		22
+#define SYS_ALARM_VF_SYND0_SOURCE_LOC		26
+#define SYS_ALARM_VF_SYND0_MORE_LOC		30
+#define SYS_ALARM_VF_SYND0_VALID_LOC		31
+
+#define SYS_LDB_QID_CFG_V(x) \
+	(0x10000f58 + (x) * 0x1000)
+#define SYS_LDB_QID_CFG_V_RST 0x0
+
+#define SYS_LDB_QID_CFG_V_SN_CFG_V	0x00000001
+#define SYS_LDB_QID_CFG_V_FID_CFG_V	0x00000002
+#define SYS_LDB_QID_CFG_V_RSVD0		0xFFFFFFFC
+#define SYS_LDB_QID_CFG_V_SN_CFG_V_LOC	0
+#define SYS_LDB_QID_CFG_V_FID_CFG_V_LOC	1
+#define SYS_LDB_QID_CFG_V_RSVD0_LOC	2
+
+#define SYS_LDB_QID_ITS(x) \
+	(0x10000f54 + (x) * 0x1000)
+#define SYS_LDB_QID_ITS_RST 0x0
+
+#define SYS_LDB_QID_ITS_QID_ITS	0x00000001
+#define SYS_LDB_QID_ITS_RSVD0	0xFFFFFFFE
+#define SYS_LDB_QID_ITS_QID_ITS_LOC	0
+#define SYS_LDB_QID_ITS_RSVD0_LOC	1
+
+#define SYS_LDB_QID_V(x) \
+	(0x10000f50 + (x) * 0x1000)
+#define SYS_LDB_QID_V_RST 0x0
+
+#define SYS_LDB_QID_V_QID_V	0x00000001
+#define SYS_LDB_QID_V_RSVD0	0xFFFFFFFE
+#define SYS_LDB_QID_V_QID_V_LOC	0
+#define SYS_LDB_QID_V_RSVD0_LOC	1
+
+#define SYS_DIR_QID_ITS(x) \
+	(0x10000f64 + (x) * 0x1000)
+#define SYS_DIR_QID_ITS_RST 0x0
+
+#define SYS_DIR_QID_ITS_QID_ITS	0x00000001
+#define SYS_DIR_QID_ITS_RSVD0	0xFFFFFFFE
+#define SYS_DIR_QID_ITS_QID_ITS_LOC	0
+#define SYS_DIR_QID_ITS_RSVD0_LOC	1
+
+#define SYS_DIR_QID_V(x) \
+	(0x10000f60 + (x) * 0x1000)
+#define SYS_DIR_QID_V_RST 0x0
+
+#define SYS_DIR_QID_V_QID_V	0x00000001
+#define SYS_DIR_QID_V_RSVD0	0xFFFFFFFE
+#define SYS_DIR_QID_V_QID_V_LOC	0
+#define SYS_DIR_QID_V_RSVD0_LOC	1
+
+#define SYS_LDB_CQ_AI_DATA(x) \
+	(0x10000fa8 + (x) * 0x1000)
+#define SYS_LDB_CQ_AI_DATA_RST 0x0
+
+#define SYS_LDB_CQ_AI_DATA_CQ_AI_DATA	0xFFFFFFFF
+#define SYS_LDB_CQ_AI_DATA_CQ_AI_DATA_LOC	0
+
+#define SYS_LDB_CQ_AI_ADDR(x) \
+	(0x10000fa4 + (x) * 0x1000)
+#define SYS_LDB_CQ_AI_ADDR_RST 0x0
+
+#define SYS_LDB_CQ_AI_ADDR_RSVD1	0x00000003
+#define SYS_LDB_CQ_AI_ADDR_CQ_AI_ADDR	0x000FFFFC
+#define SYS_LDB_CQ_AI_ADDR_RSVD0	0xFFF00000
+#define SYS_LDB_CQ_AI_ADDR_RSVD1_LOC		0
+#define SYS_LDB_CQ_AI_ADDR_CQ_AI_ADDR_LOC	2
+#define SYS_LDB_CQ_AI_ADDR_RSVD0_LOC		20
+
+#define SYS_LDB_CQ_PASID(x) \
+	(0x10000fa0 + (x) * 0x1000)
+#define SYS_LDB_CQ_PASID_RST 0x0
+
+#define SYS_LDB_CQ_PASID_PASID		0x000FFFFF
+#define SYS_LDB_CQ_PASID_EXE_REQ	0x00100000
+#define SYS_LDB_CQ_PASID_PRIV_REQ	0x00200000
+#define SYS_LDB_CQ_PASID_FMT2		0x00400000
+#define SYS_LDB_CQ_PASID_RSVD0		0xFF800000
+#define SYS_LDB_CQ_PASID_PASID_LOC	0
+#define SYS_LDB_CQ_PASID_EXE_REQ_LOC	20
+#define SYS_LDB_CQ_PASID_PRIV_REQ_LOC	21
+#define SYS_LDB_CQ_PASID_FMT2_LOC	22
+#define SYS_LDB_CQ_PASID_RSVD0_LOC	23
+
+#define SYS_LDB_CQ_AT(x) \
+	(0x10000f9c + (x) * 0x1000)
+#define SYS_LDB_CQ_AT_RST 0x0
+
+#define SYS_LDB_CQ_AT_CQ_AT	0x00000003
+#define SYS_LDB_CQ_AT_RSVD0	0xFFFFFFFC
+#define SYS_LDB_CQ_AT_CQ_AT_LOC	0
+#define SYS_LDB_CQ_AT_RSVD0_LOC	2
+
+#define SYS_LDB_CQ_ISR(x) \
+	(0x10000f98 + (x) * 0x1000)
+#define SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB_CQ_ISR_MODE_DIS  0
+#define DLB_CQ_ISR_MODE_MSI  1
+#define DLB_CQ_ISR_MODE_MSIX 2
+#define DLB_CQ_ISR_MODE_ADI  3
+
+#define SYS_LDB_CQ_ISR_VECTOR	0x0000003F
+#define SYS_LDB_CQ_ISR_VF	0x000003C0
+#define SYS_LDB_CQ_ISR_EN_CODE	0x00000C00
+#define SYS_LDB_CQ_ISR_RSVD0	0xFFFFF000
+#define SYS_LDB_CQ_ISR_VECTOR_LOC	0
+#define SYS_LDB_CQ_ISR_VF_LOC		6
+#define SYS_LDB_CQ_ISR_EN_CODE_LOC	10
+#define SYS_LDB_CQ_ISR_RSVD0_LOC	12
+
+#define SYS_LDB_CQ2VF_PF_RO(x) \
+	(0x10000f94 + (x) * 0x1000)
+#define SYS_LDB_CQ2VF_PF_RO_RST 0x0
+
+#define SYS_LDB_CQ2VF_PF_RO_VF		0x0000000F
+#define SYS_LDB_CQ2VF_PF_RO_IS_PF	0x00000010
+#define SYS_LDB_CQ2VF_PF_RO_RO		0x00000020
+#define SYS_LDB_CQ2VF_PF_RO_RSVD0	0xFFFFFFC0
+#define SYS_LDB_CQ2VF_PF_RO_VF_LOC	0
+#define SYS_LDB_CQ2VF_PF_RO_IS_PF_LOC	4
+#define SYS_LDB_CQ2VF_PF_RO_RO_LOC	5
+#define SYS_LDB_CQ2VF_PF_RO_RSVD0_LOC	6
+
+#define SYS_LDB_PP_V(x) \
+	(0x10000f90 + (x) * 0x1000)
+#define SYS_LDB_PP_V_RST 0x0
+
+#define SYS_LDB_PP_V_PP_V	0x00000001
+#define SYS_LDB_PP_V_RSVD0	0xFFFFFFFE
+#define SYS_LDB_PP_V_PP_V_LOC	0
+#define SYS_LDB_PP_V_RSVD0_LOC	1
+
+#define SYS_LDB_PP2VDEV(x) \
+	(0x10000f8c + (x) * 0x1000)
+#define SYS_LDB_PP2VDEV_RST 0x0
+
+#define SYS_LDB_PP2VDEV_VDEV	0x0000000F
+#define SYS_LDB_PP2VDEV_RSVD0	0xFFFFFFF0
+#define SYS_LDB_PP2VDEV_VDEV_LOC	0
+#define SYS_LDB_PP2VDEV_RSVD0_LOC	4
+
+#define SYS_LDB_PP2VAS(x) \
+	(0x10000f88 + (x) * 0x1000)
+#define SYS_LDB_PP2VAS_RST 0x0
+
+#define SYS_LDB_PP2VAS_VAS	0x0000001F
+#define SYS_LDB_PP2VAS_RSVD0	0xFFFFFFE0
+#define SYS_LDB_PP2VAS_VAS_LOC		0
+#define SYS_LDB_PP2VAS_RSVD0_LOC	5
+
+#define SYS_LDB_CQ_ADDR_U(x) \
+	(0x10000f84 + (x) * 0x1000)
+#define SYS_LDB_CQ_ADDR_U_RST 0x0
+
+#define SYS_LDB_CQ_ADDR_U_ADDR_U	0xFFFFFFFF
+#define SYS_LDB_CQ_ADDR_U_ADDR_U_LOC	0
+
+#define SYS_LDB_CQ_ADDR_L(x) \
+	(0x10000f80 + (x) * 0x1000)
+#define SYS_LDB_CQ_ADDR_L_RST 0x0
+
+#define SYS_LDB_CQ_ADDR_L_RSVD0		0x0000003F
+#define SYS_LDB_CQ_ADDR_L_ADDR_L	0xFFFFFFC0
+#define SYS_LDB_CQ_ADDR_L_RSVD0_LOC	0
+#define SYS_LDB_CQ_ADDR_L_ADDR_L_LOC	6
+
+#define SYS_DIR_CQ_FMT(x) \
+	(0x10000fec + (x) * 0x1000)
+#define SYS_DIR_CQ_FMT_RST 0x0
+
+#define SYS_DIR_CQ_FMT_KEEP_PF_PPID	0x00000001
+#define SYS_DIR_CQ_FMT_RSVD0		0xFFFFFFFE
+#define SYS_DIR_CQ_FMT_KEEP_PF_PPID_LOC	0
+#define SYS_DIR_CQ_FMT_RSVD0_LOC	1
+
+#define SYS_DIR_CQ_AI_DATA(x) \
+	(0x10000fe8 + (x) * 0x1000)
+#define SYS_DIR_CQ_AI_DATA_RST 0x0
+
+#define SYS_DIR_CQ_AI_DATA_CQ_AI_DATA	0xFFFFFFFF
+#define SYS_DIR_CQ_AI_DATA_CQ_AI_DATA_LOC	0
+
+#define SYS_DIR_CQ_AI_ADDR(x) \
+	(0x10000fe4 + (x) * 0x1000)
+#define SYS_DIR_CQ_AI_ADDR_RST 0x0
+
+#define SYS_DIR_CQ_AI_ADDR_RSVD1	0x00000003
+#define SYS_DIR_CQ_AI_ADDR_CQ_AI_ADDR	0x000FFFFC
+#define SYS_DIR_CQ_AI_ADDR_RSVD0	0xFFF00000
+#define SYS_DIR_CQ_AI_ADDR_RSVD1_LOC		0
+#define SYS_DIR_CQ_AI_ADDR_CQ_AI_ADDR_LOC	2
+#define SYS_DIR_CQ_AI_ADDR_RSVD0_LOC		20
+
+#define SYS_DIR_CQ_PASID(x) \
+	(0x10000fe0 + (x) * 0x1000)
+#define SYS_DIR_CQ_PASID_RST 0x0
+
+#define SYS_DIR_CQ_PASID_PASID		0x000FFFFF
+#define SYS_DIR_CQ_PASID_EXE_REQ	0x00100000
+#define SYS_DIR_CQ_PASID_PRIV_REQ	0x00200000
+#define SYS_DIR_CQ_PASID_FMT2		0x00400000
+#define SYS_DIR_CQ_PASID_RSVD0		0xFF800000
+#define SYS_DIR_CQ_PASID_PASID_LOC	0
+#define SYS_DIR_CQ_PASID_EXE_REQ_LOC	20
+#define SYS_DIR_CQ_PASID_PRIV_REQ_LOC	21
+#define SYS_DIR_CQ_PASID_FMT2_LOC	22
+#define SYS_DIR_CQ_PASID_RSVD0_LOC	23
+
+#define SYS_DIR_CQ_AT(x) \
+	(0x10000fdc + (x) * 0x1000)
+#define SYS_DIR_CQ_AT_RST 0x0
+
+#define SYS_DIR_CQ_AT_CQ_AT	0x00000003
+#define SYS_DIR_CQ_AT_RSVD0	0xFFFFFFFC
+#define SYS_DIR_CQ_AT_CQ_AT_LOC	0
+#define SYS_DIR_CQ_AT_RSVD0_LOC	2
+
+#define SYS_DIR_CQ_ISR(x) \
+	(0x10000fd8 + (x) * 0x1000)
+#define SYS_DIR_CQ_ISR_RST 0x0
+
+#define SYS_DIR_CQ_ISR_VECTOR	0x0000003F
+#define SYS_DIR_CQ_ISR_VF	0x000003C0
+#define SYS_DIR_CQ_ISR_EN_CODE	0x00000C00
+#define SYS_DIR_CQ_ISR_RSVD0	0xFFFFF000
+#define SYS_DIR_CQ_ISR_VECTOR_LOC	0
+#define SYS_DIR_CQ_ISR_VF_LOC		6
+#define SYS_DIR_CQ_ISR_EN_CODE_LOC	10
+#define SYS_DIR_CQ_ISR_RSVD0_LOC	12
+
+#define SYS_DIR_CQ2VF_PF_RO(x) \
+	(0x10000fd4 + (x) * 0x1000)
+#define SYS_DIR_CQ2VF_PF_RO_RST 0x0
+
+#define SYS_DIR_CQ2VF_PF_RO_VF		0x0000000F
+#define SYS_DIR_CQ2VF_PF_RO_IS_PF	0x00000010
+#define SYS_DIR_CQ2VF_PF_RO_RO		0x00000020
+#define SYS_DIR_CQ2VF_PF_RO_RSVD0	0xFFFFFFC0
+#define SYS_DIR_CQ2VF_PF_RO_VF_LOC	0
+#define SYS_DIR_CQ2VF_PF_RO_IS_PF_LOC	4
+#define SYS_DIR_CQ2VF_PF_RO_RO_LOC	5
+#define SYS_DIR_CQ2VF_PF_RO_RSVD0_LOC	6
+
+#define SYS_DIR_PP_V(x) \
+	(0x10000fd0 + (x) * 0x1000)
+#define SYS_DIR_PP_V_RST 0x0
+
+#define SYS_DIR_PP_V_PP_V	0x00000001
+#define SYS_DIR_PP_V_RSVD0	0xFFFFFFFE
+#define SYS_DIR_PP_V_PP_V_LOC	0
+#define SYS_DIR_PP_V_RSVD0_LOC	1
+
+#define SYS_DIR_PP2VDEV(x) \
+	(0x10000fcc + (x) * 0x1000)
+#define SYS_DIR_PP2VDEV_RST 0x0
+
+#define SYS_DIR_PP2VDEV_VDEV	0x0000000F
+#define SYS_DIR_PP2VDEV_RSVD0	0xFFFFFFF0
+#define SYS_DIR_PP2VDEV_VDEV_LOC	0
+#define SYS_DIR_PP2VDEV_RSVD0_LOC	4
+
+#define SYS_DIR_PP2VAS(x) \
+	(0x10000fc8 + (x) * 0x1000)
+#define SYS_DIR_PP2VAS_RST 0x0
+
+#define SYS_DIR_PP2VAS_VAS	0x0000001F
+#define SYS_DIR_PP2VAS_RSVD0	0xFFFFFFE0
+#define SYS_DIR_PP2VAS_VAS_LOC		0
+#define SYS_DIR_PP2VAS_RSVD0_LOC	5
+
+#define SYS_DIR_CQ_ADDR_U(x) \
+	(0x10000fc4 + (x) * 0x1000)
+#define SYS_DIR_CQ_ADDR_U_RST 0x0
+
+#define SYS_DIR_CQ_ADDR_U_ADDR_U	0xFFFFFFFF
+#define SYS_DIR_CQ_ADDR_U_ADDR_U_LOC	0
+
+#define SYS_DIR_CQ_ADDR_L(x) \
+	(0x10000fc0 + (x) * 0x1000)
+#define SYS_DIR_CQ_ADDR_L_RST 0x0
+
+#define SYS_DIR_CQ_ADDR_L_RSVD0		0x0000003F
+#define SYS_DIR_CQ_ADDR_L_ADDR_L	0xFFFFFFC0
+#define SYS_DIR_CQ_ADDR_L_RSVD0_LOC	0
+#define SYS_DIR_CQ_ADDR_L_ADDR_L_LOC	6
+
+#define SYS_PM_SMON_COMP_MASK1 0x10003024
+#define SYS_PM_SMON_COMP_MASK1_RST 0xffffffff
+
+#define SYS_PM_SMON_COMP_MASK1_COMP_MASK1	0xFFFFFFFF
+#define SYS_PM_SMON_COMP_MASK1_COMP_MASK1_LOC	0
+
+#define SYS_PM_SMON_COMP_MASK0 0x10003020
+#define SYS_PM_SMON_COMP_MASK0_RST 0xffffffff
+
+#define SYS_PM_SMON_COMP_MASK0_COMP_MASK0	0xFFFFFFFF
+#define SYS_PM_SMON_COMP_MASK0_COMP_MASK0_LOC	0
+
+#define SYS_PM_SMON_MAX_TMR 0x1000301c
+#define SYS_PM_SMON_MAX_TMR_RST 0x0
+
+#define SYS_PM_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define SYS_PM_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define SYS_PM_SMON_TMR 0x10003018
+#define SYS_PM_SMON_TMR_RST 0x0
+
+#define SYS_PM_SMON_TMR_TIMER_VAL	0xFFFFFFFF
+#define SYS_PM_SMON_TMR_TIMER_VAL_LOC	0
+
+#define SYS_PM_SMON_ACTIVITYCNTR1 0x10003014
+#define SYS_PM_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define SYS_PM_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define SYS_PM_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define SYS_PM_SMON_ACTIVITYCNTR0 0x10003010
+#define SYS_PM_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define SYS_PM_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define SYS_PM_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define SYS_PM_SMON_COMPARE1 0x1000300c
+#define SYS_PM_SMON_COMPARE1_RST 0x0
+
+#define SYS_PM_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define SYS_PM_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define SYS_PM_SMON_COMPARE0 0x10003008
+#define SYS_PM_SMON_COMPARE0_RST 0x0
+
+#define SYS_PM_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define SYS_PM_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define SYS_PM_SMON_CFG1 0x10003004
+#define SYS_PM_SMON_CFG1_RST 0x0
+
+#define SYS_PM_SMON_CFG1_MODE0	0x000000FF
+#define SYS_PM_SMON_CFG1_MODE1	0x0000FF00
+#define SYS_PM_SMON_CFG1_RSVD	0xFFFF0000
+#define SYS_PM_SMON_CFG1_MODE0_LOC	0
+#define SYS_PM_SMON_CFG1_MODE1_LOC	8
+#define SYS_PM_SMON_CFG1_RSVD_LOC	16
+
+#define SYS_PM_SMON_CFG0 0x10003000
+#define SYS_PM_SMON_CFG0_RST 0x40000000
+
+#define SYS_PM_SMON_CFG0_SMON_ENABLE		0x00000001
+#define SYS_PM_SMON_CFG0_RSVD2			0x0000000E
+#define SYS_PM_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define SYS_PM_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define SYS_PM_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define SYS_PM_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define SYS_PM_SMON_CFG0_SMON_MODE		0x0000F000
+#define SYS_PM_SMON_CFG0_STOPCOUNTEROVFL	0x00010000
+#define SYS_PM_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define SYS_PM_SMON_CFG0_STATCOUNTER0OVFL	0x00040000
+#define SYS_PM_SMON_CFG0_STATCOUNTER1OVFL	0x00080000
+#define SYS_PM_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define SYS_PM_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define SYS_PM_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define SYS_PM_SMON_CFG0_RSVD1			0x00800000
+#define SYS_PM_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define SYS_PM_SMON_CFG0_RSVD0			0x20000000
+#define SYS_PM_SMON_CFG0_VERSION		0xC0000000
+#define SYS_PM_SMON_CFG0_SMON_ENABLE_LOC		0
+#define SYS_PM_SMON_CFG0_RSVD2_LOC			1
+#define SYS_PM_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define SYS_PM_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define SYS_PM_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define SYS_PM_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define SYS_PM_SMON_CFG0_SMON_MODE_LOC			12
+#define SYS_PM_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define SYS_PM_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define SYS_PM_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define SYS_PM_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define SYS_PM_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define SYS_PM_SMON_CFG0_INTTIMEROVFL_LOC		21
+#define SYS_PM_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define SYS_PM_SMON_CFG0_RSVD1_LOC			23
+#define SYS_PM_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define SYS_PM_SMON_CFG0_RSVD0_LOC			29
+#define SYS_PM_SMON_CFG0_VERSION_LOC			30
+
+#define SYS_SMON_COMP_MASK1(x) \
+	(0x18002024 + (x) * 0x40)
+#define SYS_SMON_COMP_MASK1_RST 0xffffffff
+
+#define SYS_SMON_COMP_MASK1_COMP_MASK1	0xFFFFFFFF
+#define SYS_SMON_COMP_MASK1_COMP_MASK1_LOC	0
+
+#define SYS_SMON_COMP_MASK0(x) \
+	(0x18002020 + (x) * 0x40)
+#define SYS_SMON_COMP_MASK0_RST 0xffffffff
+
+#define SYS_SMON_COMP_MASK0_COMP_MASK0	0xFFFFFFFF
+#define SYS_SMON_COMP_MASK0_COMP_MASK0_LOC	0
+
+#define SYS_SMON_MAX_TMR(x) \
+	(0x1800201c + (x) * 0x40)
+#define SYS_SMON_MAX_TMR_RST 0x0
+
+#define SYS_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define SYS_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define SYS_SMON_TMR(x) \
+	(0x18002018 + (x) * 0x40)
+#define SYS_SMON_TMR_RST 0x0
+
+#define SYS_SMON_TMR_TIMER_VAL	0xFFFFFFFF
+#define SYS_SMON_TMR_TIMER_VAL_LOC	0
+
+#define SYS_SMON_ACTIVITYCNTR1(x) \
+	(0x18002014 + (x) * 0x40)
+#define SYS_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define SYS_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define SYS_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define SYS_SMON_ACTIVITYCNTR0(x) \
+	(0x18002010 + (x) * 0x40)
+#define SYS_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define SYS_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define SYS_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define SYS_SMON_COMPARE1(x) \
+	(0x1800200c + (x) * 0x40)
+#define SYS_SMON_COMPARE1_RST 0x0
+
+#define SYS_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define SYS_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define SYS_SMON_COMPARE0(x) \
+	(0x18002008 + (x) * 0x40)
+#define SYS_SMON_COMPARE0_RST 0x0
+
+#define SYS_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define SYS_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define SYS_SMON_CFG1(x) \
+	(0x18002004 + (x) * 0x40)
+#define SYS_SMON_CFG1_RST 0x0
+
+#define SYS_SMON_CFG1_MODE0	0x000000FF
+#define SYS_SMON_CFG1_MODE1	0x0000FF00
+#define SYS_SMON_CFG1_RSVD	0xFFFF0000
+#define SYS_SMON_CFG1_MODE0_LOC	0
+#define SYS_SMON_CFG1_MODE1_LOC	8
+#define SYS_SMON_CFG1_RSVD_LOC	16
+
+#define SYS_SMON_CFG0(x) \
+	(0x18002000 + (x) * 0x40)
+#define SYS_SMON_CFG0_RST 0x40000000
+
+#define SYS_SMON_CFG0_SMON_ENABLE		0x00000001
+#define SYS_SMON_CFG0_RSVD2			0x0000000E
+#define SYS_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define SYS_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define SYS_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define SYS_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define SYS_SMON_CFG0_SMON_MODE			0x0000F000
+#define SYS_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define SYS_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define SYS_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define SYS_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define SYS_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define SYS_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define SYS_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define SYS_SMON_CFG0_RSVD1			0x00800000
+#define SYS_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define SYS_SMON_CFG0_RSVD0			0x20000000
+#define SYS_SMON_CFG0_VERSION			0xC0000000
+#define SYS_SMON_CFG0_SMON_ENABLE_LOC			0
+#define SYS_SMON_CFG0_RSVD2_LOC				1
+#define SYS_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define SYS_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define SYS_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define SYS_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define SYS_SMON_CFG0_SMON_MODE_LOC			12
+#define SYS_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define SYS_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define SYS_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define SYS_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define SYS_SMON_CFG0_STOPTIMEROVFL_LOC			20
+#define SYS_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define SYS_SMON_CFG0_STATTIMEROVFL_LOC			22
+#define SYS_SMON_CFG0_RSVD1_LOC				23
+#define SYS_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define SYS_SMON_CFG0_RSVD0_LOC				29
+#define SYS_SMON_CFG0_VERSION_LOC			30
+
+#define SYS_INGRESS_ALARM_ENBL 0x10000300
+#define SYS_INGRESS_ALARM_ENBL_RST 0x0
+
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_HCW		0x00000001
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_PP		0x00000002
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_PASID		0x00000004
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_QID		0x00000008
+#define SYS_INGRESS_ALARM_ENBL_DISABLED_QID		0x00000010
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_LDB_QID_CFG	0x00000020
+#define SYS_INGRESS_ALARM_ENBL_RSVD0			0xFFFFFFC0
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_HCW_LOC		0
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_PP_LOC		1
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_PASID_LOC	2
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_QID_LOC		3
+#define SYS_INGRESS_ALARM_ENBL_DISABLED_QID_LOC		4
+#define SYS_INGRESS_ALARM_ENBL_ILLEGAL_LDB_QID_CFG_LOC	5
+#define SYS_INGRESS_ALARM_ENBL_RSVD0_LOC		6
+
+#define SYS_MSIX_ACK 0x10000400
+#define SYS_MSIX_ACK_RST 0x0
+
+#define SYS_MSIX_ACK_MSIX_0_ACK	0x00000001
+#define SYS_MSIX_ACK_MSIX_1_ACK	0x00000002
+#define SYS_MSIX_ACK_RSVD0	0xFFFFFFFC
+#define SYS_MSIX_ACK_MSIX_0_ACK_LOC	0
+#define SYS_MSIX_ACK_MSIX_1_ACK_LOC	1
+#define SYS_MSIX_ACK_RSVD0_LOC		2
+
+#define SYS_MSIX_PASSTHRU 0x10000404
+#define SYS_MSIX_PASSTHRU_RST 0x0
+
+#define SYS_MSIX_PASSTHRU_MSIX_0_PASSTHRU	0x00000001
+#define SYS_MSIX_PASSTHRU_MSIX_1_PASSTHRU	0x00000002
+#define SYS_MSIX_PASSTHRU_RSVD0			0xFFFFFFFC
+#define SYS_MSIX_PASSTHRU_MSIX_0_PASSTHRU_LOC	0
+#define SYS_MSIX_PASSTHRU_MSIX_1_PASSTHRU_LOC	1
+#define SYS_MSIX_PASSTHRU_RSVD0_LOC		2
+
+#define SYS_MSIX_MODE 0x10000408
+#define SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB_MSIX_MODE_PACKED     0
+#define DLB_MSIX_MODE_COMPRESSED 1
+
+#define SYS_MSIX_MODE_MODE	0x00000001
+#define SYS_MSIX_MODE_POLL_MODE	0x00000002
+#define SYS_MSIX_MODE_POLL_MASK	0x00000004
+#define SYS_MSIX_MODE_POLL_LOCK	0x00000008
+#define SYS_MSIX_MODE_RSVD0	0xFFFFFFF0
+#define SYS_MSIX_MODE_MODE_LOC		0
+#define SYS_MSIX_MODE_POLL_MODE_LOC	1
+#define SYS_MSIX_MODE_POLL_MASK_LOC	2
+#define SYS_MSIX_MODE_POLL_LOCK_LOC	3
+#define SYS_MSIX_MODE_RSVD0_LOC		4
+
+#define SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_0_OCC_INT	0x00000001
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_1_OCC_INT	0x00000002
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_2_OCC_INT	0x00000004
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_3_OCC_INT	0x00000008
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_4_OCC_INT	0x00000010
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_5_OCC_INT	0x00000020
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_6_OCC_INT	0x00000040
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_7_OCC_INT	0x00000080
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_8_OCC_INT	0x00000100
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_9_OCC_INT	0x00000200
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_10_OCC_INT	0x00000400
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_11_OCC_INT	0x00000800
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_12_OCC_INT	0x00001000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_13_OCC_INT	0x00002000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_14_OCC_INT	0x00004000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_15_OCC_INT	0x00008000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_16_OCC_INT	0x00010000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_17_OCC_INT	0x00020000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_18_OCC_INT	0x00040000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_19_OCC_INT	0x00080000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_20_OCC_INT	0x00100000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_21_OCC_INT	0x00200000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_22_OCC_INT	0x00400000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_23_OCC_INT	0x00800000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_24_OCC_INT	0x01000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_25_OCC_INT	0x02000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_26_OCC_INT	0x04000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_27_OCC_INT	0x08000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_28_OCC_INT	0x10000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_29_OCC_INT	0x20000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_30_OCC_INT	0x40000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_31_OCC_INT	0x80000000
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_0_OCC_INT_LOC	0
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_1_OCC_INT_LOC	1
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_2_OCC_INT_LOC	2
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_3_OCC_INT_LOC	3
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_4_OCC_INT_LOC	4
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_5_OCC_INT_LOC	5
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_6_OCC_INT_LOC	6
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_7_OCC_INT_LOC	7
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_8_OCC_INT_LOC	8
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_9_OCC_INT_LOC	9
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_10_OCC_INT_LOC	10
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_11_OCC_INT_LOC	11
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_12_OCC_INT_LOC	12
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_13_OCC_INT_LOC	13
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_14_OCC_INT_LOC	14
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_15_OCC_INT_LOC	15
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_16_OCC_INT_LOC	16
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_17_OCC_INT_LOC	17
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_18_OCC_INT_LOC	18
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_19_OCC_INT_LOC	19
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_20_OCC_INT_LOC	20
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_21_OCC_INT_LOC	21
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_22_OCC_INT_LOC	22
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_23_OCC_INT_LOC	23
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_24_OCC_INT_LOC	24
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_25_OCC_INT_LOC	25
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_26_OCC_INT_LOC	26
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_27_OCC_INT_LOC	27
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_28_OCC_INT_LOC	28
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_29_OCC_INT_LOC	29
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_30_OCC_INT_LOC	30
+#define SYS_DIR_CQ_31_0_OCC_INT_STS_CQ_31_OCC_INT_LOC	31
+
+#define SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_32_OCC_INT	0x00000001
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_33_OCC_INT	0x00000002
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_34_OCC_INT	0x00000004
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_35_OCC_INT	0x00000008
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_36_OCC_INT	0x00000010
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_37_OCC_INT	0x00000020
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_38_OCC_INT	0x00000040
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_39_OCC_INT	0x00000080
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_40_OCC_INT	0x00000100
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_41_OCC_INT	0x00000200
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_42_OCC_INT	0x00000400
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_43_OCC_INT	0x00000800
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_44_OCC_INT	0x00001000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_45_OCC_INT	0x00002000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_46_OCC_INT	0x00004000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_47_OCC_INT	0x00008000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_48_OCC_INT	0x00010000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_49_OCC_INT	0x00020000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_50_OCC_INT	0x00040000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_51_OCC_INT	0x00080000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_52_OCC_INT	0x00100000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_53_OCC_INT	0x00200000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_54_OCC_INT	0x00400000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_55_OCC_INT	0x00800000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_56_OCC_INT	0x01000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_57_OCC_INT	0x02000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_58_OCC_INT	0x04000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_59_OCC_INT	0x08000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_60_OCC_INT	0x10000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_61_OCC_INT	0x20000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_62_OCC_INT	0x40000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_63_OCC_INT	0x80000000
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_32_OCC_INT_LOC	0
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_33_OCC_INT_LOC	1
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_34_OCC_INT_LOC	2
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_35_OCC_INT_LOC	3
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_36_OCC_INT_LOC	4
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_37_OCC_INT_LOC	5
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_38_OCC_INT_LOC	6
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_39_OCC_INT_LOC	7
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_40_OCC_INT_LOC	8
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_41_OCC_INT_LOC	9
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_42_OCC_INT_LOC	10
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_43_OCC_INT_LOC	11
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_44_OCC_INT_LOC	12
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_45_OCC_INT_LOC	13
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_46_OCC_INT_LOC	14
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_47_OCC_INT_LOC	15
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_48_OCC_INT_LOC	16
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_49_OCC_INT_LOC	17
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_50_OCC_INT_LOC	18
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_51_OCC_INT_LOC	19
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_52_OCC_INT_LOC	20
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_53_OCC_INT_LOC	21
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_54_OCC_INT_LOC	22
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_55_OCC_INT_LOC	23
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_56_OCC_INT_LOC	24
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_57_OCC_INT_LOC	25
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_58_OCC_INT_LOC	26
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_59_OCC_INT_LOC	27
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_60_OCC_INT_LOC	28
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_61_OCC_INT_LOC	29
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_62_OCC_INT_LOC	30
+#define SYS_DIR_CQ_63_32_OCC_INT_STS_CQ_63_OCC_INT_LOC	31
+
+#define SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_0_OCC_INT	0x00000001
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_1_OCC_INT	0x00000002
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_2_OCC_INT	0x00000004
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_3_OCC_INT	0x00000008
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_4_OCC_INT	0x00000010
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_5_OCC_INT	0x00000020
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_6_OCC_INT	0x00000040
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_7_OCC_INT	0x00000080
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_8_OCC_INT	0x00000100
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_9_OCC_INT	0x00000200
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_10_OCC_INT	0x00000400
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_11_OCC_INT	0x00000800
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_12_OCC_INT	0x00001000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_13_OCC_INT	0x00002000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_14_OCC_INT	0x00004000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_15_OCC_INT	0x00008000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_16_OCC_INT	0x00010000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_17_OCC_INT	0x00020000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_18_OCC_INT	0x00040000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_19_OCC_INT	0x00080000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_20_OCC_INT	0x00100000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_21_OCC_INT	0x00200000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_22_OCC_INT	0x00400000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_23_OCC_INT	0x00800000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_24_OCC_INT	0x01000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_25_OCC_INT	0x02000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_26_OCC_INT	0x04000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_27_OCC_INT	0x08000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_28_OCC_INT	0x10000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_29_OCC_INT	0x20000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_30_OCC_INT	0x40000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_31_OCC_INT	0x80000000
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_0_OCC_INT_LOC	0
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_1_OCC_INT_LOC	1
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_2_OCC_INT_LOC	2
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_3_OCC_INT_LOC	3
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_4_OCC_INT_LOC	4
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_5_OCC_INT_LOC	5
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_6_OCC_INT_LOC	6
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_7_OCC_INT_LOC	7
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_8_OCC_INT_LOC	8
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_9_OCC_INT_LOC	9
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_10_OCC_INT_LOC	10
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_11_OCC_INT_LOC	11
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_12_OCC_INT_LOC	12
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_13_OCC_INT_LOC	13
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_14_OCC_INT_LOC	14
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_15_OCC_INT_LOC	15
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_16_OCC_INT_LOC	16
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_17_OCC_INT_LOC	17
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_18_OCC_INT_LOC	18
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_19_OCC_INT_LOC	19
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_20_OCC_INT_LOC	20
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_21_OCC_INT_LOC	21
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_22_OCC_INT_LOC	22
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_23_OCC_INT_LOC	23
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_24_OCC_INT_LOC	24
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_25_OCC_INT_LOC	25
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_26_OCC_INT_LOC	26
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_27_OCC_INT_LOC	27
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_28_OCC_INT_LOC	28
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_29_OCC_INT_LOC	29
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_30_OCC_INT_LOC	30
+#define SYS_LDB_CQ_31_0_OCC_INT_STS_CQ_31_OCC_INT_LOC	31
+
+#define SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_32_OCC_INT	0x00000001
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_33_OCC_INT	0x00000002
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_34_OCC_INT	0x00000004
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_35_OCC_INT	0x00000008
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_36_OCC_INT	0x00000010
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_37_OCC_INT	0x00000020
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_38_OCC_INT	0x00000040
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_39_OCC_INT	0x00000080
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_40_OCC_INT	0x00000100
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_41_OCC_INT	0x00000200
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_42_OCC_INT	0x00000400
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_43_OCC_INT	0x00000800
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_44_OCC_INT	0x00001000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_45_OCC_INT	0x00002000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_46_OCC_INT	0x00004000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_47_OCC_INT	0x00008000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_48_OCC_INT	0x00010000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_49_OCC_INT	0x00020000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_50_OCC_INT	0x00040000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_51_OCC_INT	0x00080000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_52_OCC_INT	0x00100000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_53_OCC_INT	0x00200000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_54_OCC_INT	0x00400000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_55_OCC_INT	0x00800000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_56_OCC_INT	0x01000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_57_OCC_INT	0x02000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_58_OCC_INT	0x04000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_59_OCC_INT	0x08000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_60_OCC_INT	0x10000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_61_OCC_INT	0x20000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_62_OCC_INT	0x40000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_63_OCC_INT	0x80000000
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_32_OCC_INT_LOC	0
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_33_OCC_INT_LOC	1
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_34_OCC_INT_LOC	2
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_35_OCC_INT_LOC	3
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_36_OCC_INT_LOC	4
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_37_OCC_INT_LOC	5
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_38_OCC_INT_LOC	6
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_39_OCC_INT_LOC	7
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_40_OCC_INT_LOC	8
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_41_OCC_INT_LOC	9
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_42_OCC_INT_LOC	10
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_43_OCC_INT_LOC	11
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_44_OCC_INT_LOC	12
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_45_OCC_INT_LOC	13
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_46_OCC_INT_LOC	14
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_47_OCC_INT_LOC	15
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_48_OCC_INT_LOC	16
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_49_OCC_INT_LOC	17
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_50_OCC_INT_LOC	18
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_51_OCC_INT_LOC	19
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_52_OCC_INT_LOC	20
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_53_OCC_INT_LOC	21
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_54_OCC_INT_LOC	22
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_55_OCC_INT_LOC	23
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_56_OCC_INT_LOC	24
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_57_OCC_INT_LOC	25
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_58_OCC_INT_LOC	26
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_59_OCC_INT_LOC	27
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_60_OCC_INT_LOC	28
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_61_OCC_INT_LOC	29
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_62_OCC_INT_LOC	30
+#define SYS_LDB_CQ_63_32_OCC_INT_STS_CQ_63_OCC_INT_LOC	31
+
+#define SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define SYS_DIR_CQ_OPT_CLR_RST 0x0
+
+#define SYS_DIR_CQ_OPT_CLR_CQ		0x0000003F
+#define SYS_DIR_CQ_OPT_CLR_RSVD0	0xFFFFFFC0
+#define SYS_DIR_CQ_OPT_CLR_CQ_LOC	0
+#define SYS_DIR_CQ_OPT_CLR_RSVD0_LOC	6
+
+#define SYS_ALARM_HW_SYND 0x1000050c
+#define SYS_ALARM_HW_SYND_RST 0x0
+
+#define SYS_ALARM_HW_SYND_SYNDROME	0x000000FF
+#define SYS_ALARM_HW_SYND_RTYPE		0x00000300
+#define SYS_ALARM_HW_SYND_ALARM		0x00000400
+#define SYS_ALARM_HW_SYND_CWD		0x00000800
+#define SYS_ALARM_HW_SYND_VF_PF_MB	0x00001000
+#define SYS_ALARM_HW_SYND_RSVD0		0x00002000
+#define SYS_ALARM_HW_SYND_CLS		0x0000C000
+#define SYS_ALARM_HW_SYND_AID		0x003F0000
+#define SYS_ALARM_HW_SYND_UNIT		0x03C00000
+#define SYS_ALARM_HW_SYND_SOURCE	0x3C000000
+#define SYS_ALARM_HW_SYND_MORE		0x40000000
+#define SYS_ALARM_HW_SYND_VALID		0x80000000
+#define SYS_ALARM_HW_SYND_SYNDROME_LOC	0
+#define SYS_ALARM_HW_SYND_RTYPE_LOC	8
+#define SYS_ALARM_HW_SYND_ALARM_LOC	10
+#define SYS_ALARM_HW_SYND_CWD_LOC	11
+#define SYS_ALARM_HW_SYND_VF_PF_MB_LOC	12
+#define SYS_ALARM_HW_SYND_RSVD0_LOC	13
+#define SYS_ALARM_HW_SYND_CLS_LOC	14
+#define SYS_ALARM_HW_SYND_AID_LOC	16
+#define SYS_ALARM_HW_SYND_UNIT_LOC	22
+#define SYS_ALARM_HW_SYND_SOURCE_LOC	26
+#define SYS_ALARM_HW_SYND_MORE_LOC	30
+#define SYS_ALARM_HW_SYND_VALID_LOC	31
+
+#define AQED_QID_FID_LIM(x) \
+	(0x20000000 + (x) * 0x1000)
+#define AQED_QID_FID_LIM_RST 0x7ff
+
+#define AQED_QID_FID_LIM_QID_FID_LIMIT	0x00001FFF
+#define AQED_QID_FID_LIM_RSVD0		0xFFFFE000
+#define AQED_QID_FID_LIM_QID_FID_LIMIT_LOC	0
+#define AQED_QID_FID_LIM_RSVD0_LOC		13
+
+#define AQED_QID_HID_WIDTH(x) \
+	(0x20080000 + (x) * 0x1000)
+#define AQED_QID_HID_WIDTH_RST 0x0
+
+#define AQED_QID_HID_WIDTH_COMPRESS_CODE	0x00000007
+#define AQED_QID_HID_WIDTH_RSVD0		0xFFFFFFF8
+#define AQED_QID_HID_WIDTH_COMPRESS_CODE_LOC	0
+#define AQED_QID_HID_WIDTH_RSVD0_LOC		3
+
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI0	0x000000FF
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI1	0x0000FF00
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI2	0x00FF0000
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI3	0xFF000000
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI0_LOC	0
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI1_LOC	8
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI2_LOC	16
+#define AQED_CFG_ARB_WEIGHTS_TQPRI_ATM_0_PRI3_LOC	24
+
+#define AQED_SMON_ACTIVITYCNTR0 0x2c00004c
+#define AQED_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define AQED_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define AQED_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define AQED_SMON_ACTIVITYCNTR1 0x2c000050
+#define AQED_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define AQED_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define AQED_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define AQED_SMON_COMPARE0 0x2c000054
+#define AQED_SMON_COMPARE0_RST 0x0
+
+#define AQED_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define AQED_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define AQED_SMON_COMPARE1 0x2c000058
+#define AQED_SMON_COMPARE1_RST 0x0
+
+#define AQED_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define AQED_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define AQED_SMON_CFG0 0x2c00005c
+#define AQED_SMON_CFG0_RST 0x40000000
+
+#define AQED_SMON_CFG0_SMON_ENABLE		0x00000001
+#define AQED_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define AQED_SMON_CFG0_RSVZ0			0x0000000C
+#define AQED_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define AQED_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define AQED_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define AQED_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define AQED_SMON_CFG0_SMON_MODE		0x0000F000
+#define AQED_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define AQED_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define AQED_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define AQED_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define AQED_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define AQED_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define AQED_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define AQED_SMON_CFG0_RSVZ1			0x00800000
+#define AQED_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define AQED_SMON_CFG0_RSVZ2			0x20000000
+#define AQED_SMON_CFG0_VERSION			0xC0000000
+#define AQED_SMON_CFG0_SMON_ENABLE_LOC			0
+#define AQED_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define AQED_SMON_CFG0_RSVZ0_LOC			2
+#define AQED_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define AQED_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define AQED_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define AQED_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define AQED_SMON_CFG0_SMON_MODE_LOC			12
+#define AQED_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define AQED_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define AQED_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define AQED_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define AQED_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define AQED_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define AQED_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define AQED_SMON_CFG0_RSVZ1_LOC			23
+#define AQED_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define AQED_SMON_CFG0_RSVZ2_LOC			29
+#define AQED_SMON_CFG0_VERSION_LOC			30
+
+#define AQED_SMON_CFG1 0x2c000060
+#define AQED_SMON_CFG1_RST 0x0
+
+#define AQED_SMON_CFG1_MODE0	0x000000FF
+#define AQED_SMON_CFG1_MODE1	0x0000FF00
+#define AQED_SMON_CFG1_RSVZ0	0xFFFF0000
+#define AQED_SMON_CFG1_MODE0_LOC	0
+#define AQED_SMON_CFG1_MODE1_LOC	8
+#define AQED_SMON_CFG1_RSVZ0_LOC	16
+
+#define AQED_SMON_MAX_TMR 0x2c000064
+#define AQED_SMON_MAX_TMR_RST 0x0
+
+#define AQED_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define AQED_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define AQED_SMON_TMR 0x2c000068
+#define AQED_SMON_TMR_RST 0x0
+
+#define AQED_SMON_TMR_TIMER	0xFFFFFFFF
+#define AQED_SMON_TMR_TIMER_LOC	0
+
+#define ATM_QID2CQIDIX_00(x) \
+	(0x30080000 + (x) * 0x1000)
+#define ATM_QID2CQIDIX_00_RST 0x0
+#define ATM_QID2CQIDIX(x, y) \
+	(ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define ATM_QID2CQIDIX_NUM 16
+
+#define ATM_QID2CQIDIX_00_CQ_P0	0x000000FF
+#define ATM_QID2CQIDIX_00_CQ_P1	0x0000FF00
+#define ATM_QID2CQIDIX_00_CQ_P2	0x00FF0000
+#define ATM_QID2CQIDIX_00_CQ_P3	0xFF000000
+#define ATM_QID2CQIDIX_00_CQ_P0_LOC	0
+#define ATM_QID2CQIDIX_00_CQ_P1_LOC	8
+#define ATM_QID2CQIDIX_00_CQ_P2_LOC	16
+#define ATM_QID2CQIDIX_00_CQ_P3_LOC	24
+
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN0	0x000000FF
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN1	0x0000FF00
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN2	0x00FF0000
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN3	0xFF000000
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN0_LOC	0
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN1_LOC	8
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN2_LOC	16
+#define ATM_CFG_ARB_WEIGHTS_RDY_BIN_BIN3_LOC	24
+
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN0	0x000000FF
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN1	0x0000FF00
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN2	0x00FF0000
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN3	0xFF000000
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN0_LOC	0
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN1_LOC	8
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN2_LOC	16
+#define ATM_CFG_ARB_WEIGHTS_SCHED_BIN_BIN3_LOC	24
+
+#define ATM_SMON_ACTIVITYCNTR0 0x3c000050
+#define ATM_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define ATM_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define ATM_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define ATM_SMON_ACTIVITYCNTR1 0x3c000054
+#define ATM_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define ATM_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define ATM_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define ATM_SMON_COMPARE0 0x3c000058
+#define ATM_SMON_COMPARE0_RST 0x0
+
+#define ATM_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define ATM_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define ATM_SMON_COMPARE1 0x3c00005c
+#define ATM_SMON_COMPARE1_RST 0x0
+
+#define ATM_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define ATM_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define ATM_SMON_CFG0 0x3c000060
+#define ATM_SMON_CFG0_RST 0x40000000
+
+#define ATM_SMON_CFG0_SMON_ENABLE		0x00000001
+#define ATM_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define ATM_SMON_CFG0_RSVZ0			0x0000000C
+#define ATM_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define ATM_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define ATM_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define ATM_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define ATM_SMON_CFG0_SMON_MODE			0x0000F000
+#define ATM_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define ATM_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define ATM_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define ATM_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define ATM_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define ATM_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define ATM_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define ATM_SMON_CFG0_RSVZ1			0x00800000
+#define ATM_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define ATM_SMON_CFG0_RSVZ2			0x20000000
+#define ATM_SMON_CFG0_VERSION			0xC0000000
+#define ATM_SMON_CFG0_SMON_ENABLE_LOC			0
+#define ATM_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define ATM_SMON_CFG0_RSVZ0_LOC				2
+#define ATM_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define ATM_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define ATM_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define ATM_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define ATM_SMON_CFG0_SMON_MODE_LOC			12
+#define ATM_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define ATM_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define ATM_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define ATM_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define ATM_SMON_CFG0_STOPTIMEROVFL_LOC			20
+#define ATM_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define ATM_SMON_CFG0_STATTIMEROVFL_LOC			22
+#define ATM_SMON_CFG0_RSVZ1_LOC				23
+#define ATM_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define ATM_SMON_CFG0_RSVZ2_LOC				29
+#define ATM_SMON_CFG0_VERSION_LOC			30
+
+#define ATM_SMON_CFG1 0x3c000064
+#define ATM_SMON_CFG1_RST 0x0
+
+#define ATM_SMON_CFG1_MODE0	0x000000FF
+#define ATM_SMON_CFG1_MODE1	0x0000FF00
+#define ATM_SMON_CFG1_RSVZ0	0xFFFF0000
+#define ATM_SMON_CFG1_MODE0_LOC	0
+#define ATM_SMON_CFG1_MODE1_LOC	8
+#define ATM_SMON_CFG1_RSVZ0_LOC	16
+
+#define ATM_SMON_MAX_TMR 0x3c000068
+#define ATM_SMON_MAX_TMR_RST 0x0
+
+#define ATM_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define ATM_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define ATM_SMON_TMR 0x3c00006c
+#define ATM_SMON_TMR_RST 0x0
+
+#define ATM_SMON_TMR_TIMER	0xFFFFFFFF
+#define ATM_SMON_TMR_TIMER_LOC	0
 
 #define CHP_CFG_DIR_VAS_CRD(x) \
 	(0x40000000 + (x) * 0x1000)
 #define CHP_CFG_DIR_VAS_CRD_RST 0x0
 
-#define CHP_CFG_DIR_VAS_CRD_COUNT	0x00003FFF
-#define CHP_CFG_DIR_VAS_CRD_RSVD0	0xFFFFC000
-#define CHP_CFG_DIR_VAS_CRD_COUNT_LOC	0
-#define CHP_CFG_DIR_VAS_CRD_RSVD0_LOC	14
+#define CHP_CFG_DIR_VAS_CRD_COUNT	0x00003FFF
+#define CHP_CFG_DIR_VAS_CRD_RSVD0	0xFFFFC000
+#define CHP_CFG_DIR_VAS_CRD_COUNT_LOC	0
+#define CHP_CFG_DIR_VAS_CRD_RSVD0_LOC	14
+
+#define CHP_CFG_LDB_VAS_CRD(x) \
+	(0x40080000 + (x) * 0x1000)
+#define CHP_CFG_LDB_VAS_CRD_RST 0x0
+
+#define CHP_CFG_LDB_VAS_CRD_COUNT	0x00007FFF
+#define CHP_CFG_LDB_VAS_CRD_RSVD0	0xFFFF8000
+#define CHP_CFG_LDB_VAS_CRD_COUNT_LOC	0
+#define CHP_CFG_LDB_VAS_CRD_RSVD0_LOC	15
+
+#define CHP_ORD_QID_SN(x) \
+	(0x40100000 + (x) * 0x1000)
+#define CHP_ORD_QID_SN_RST 0x0
+
+#define CHP_ORD_QID_SN_SN	0x000003FF
+#define CHP_ORD_QID_SN_RSVD0	0xFFFFFC00
+#define CHP_ORD_QID_SN_SN_LOC		0
+#define CHP_ORD_QID_SN_RSVD0_LOC	10
+
+#define CHP_ORD_QID_SN_MAP(x) \
+	(0x40180000 + (x) * 0x1000)
+#define CHP_ORD_QID_SN_MAP_RST 0x0
+
+#define CHP_ORD_QID_SN_MAP_MODE		0x00000007
+#define CHP_ORD_QID_SN_MAP_SLOT		0x00000078
+#define CHP_ORD_QID_SN_MAP_RSVZ0	0x00000080
+#define CHP_ORD_QID_SN_MAP_GRP		0x00000100
+#define CHP_ORD_QID_SN_MAP_RSVZ1	0x00000200
+#define CHP_ORD_QID_SN_MAP_RSVD0	0xFFFFFC00
+#define CHP_ORD_QID_SN_MAP_MODE_LOC	0
+#define CHP_ORD_QID_SN_MAP_SLOT_LOC	3
+#define CHP_ORD_QID_SN_MAP_RSVZ0_LOC	7
+#define CHP_ORD_QID_SN_MAP_GRP_LOC	8
+#define CHP_ORD_QID_SN_MAP_RSVZ1_LOC	9
+#define CHP_ORD_QID_SN_MAP_RSVD0_LOC	10
+
+#define CHP_SN_CHK_ENBL(x) \
+	(0x40200000 + (x) * 0x1000)
+#define CHP_SN_CHK_ENBL_RST 0x0
+
+#define CHP_SN_CHK_ENBL_EN	0x00000001
+#define CHP_SN_CHK_ENBL_RSVD0	0xFFFFFFFE
+#define CHP_SN_CHK_ENBL_EN_LOC		0
+#define CHP_SN_CHK_ENBL_RSVD0_LOC	1
+
+#define CHP_DIR_CQ_DEPTH(x) \
+	(0x40280000 + (x) * 0x1000)
+#define CHP_DIR_CQ_DEPTH_RST 0x0
+
+#define CHP_DIR_CQ_DEPTH_DEPTH	0x00001FFF
+#define CHP_DIR_CQ_DEPTH_RSVD0	0xFFFFE000
+#define CHP_DIR_CQ_DEPTH_DEPTH_LOC	0
+#define CHP_DIR_CQ_DEPTH_RSVD0_LOC	13
+
+#define CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+	(0x40300000 + (x) * 0x1000)
+#define CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+
+#define CHP_DIR_CQ_INT_DEPTH_THRSH_DEPTH_THRESHOLD	0x00001FFF
+#define CHP_DIR_CQ_INT_DEPTH_THRSH_RSVD0		0xFFFFE000
+#define CHP_DIR_CQ_INT_DEPTH_THRSH_DEPTH_THRESHOLD_LOC	0
+#define CHP_DIR_CQ_INT_DEPTH_THRSH_RSVD0_LOC		13
+
+#define CHP_DIR_CQ_INT_ENB(x) \
+	(0x40380000 + (x) * 0x1000)
+#define CHP_DIR_CQ_INT_ENB_RST 0x0
+
+#define CHP_DIR_CQ_INT_ENB_EN_TIM	0x00000001
+#define CHP_DIR_CQ_INT_ENB_EN_DEPTH	0x00000002
+#define CHP_DIR_CQ_INT_ENB_RSVD0	0xFFFFFFFC
+#define CHP_DIR_CQ_INT_ENB_EN_TIM_LOC	0
+#define CHP_DIR_CQ_INT_ENB_EN_DEPTH_LOC	1
+#define CHP_DIR_CQ_INT_ENB_RSVD0_LOC	2
+
+#define CHP_DIR_CQ_TMR_THRSH(x) \
+	(0x40480000 + (x) * 0x1000)
+#define CHP_DIR_CQ_TMR_THRSH_RST 0x1
+
+#define CHP_DIR_CQ_TMR_THRSH_THRSH_0	0x00000001
+#define CHP_DIR_CQ_TMR_THRSH_THRSH_13_1	0x00003FFE
+#define CHP_DIR_CQ_TMR_THRSH_RSVD0	0xFFFFC000
+#define CHP_DIR_CQ_TMR_THRSH_THRSH_0_LOC	0
+#define CHP_DIR_CQ_TMR_THRSH_THRSH_13_1_LOC	1
+#define CHP_DIR_CQ_TMR_THRSH_RSVD0_LOC		14
+
+#define CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+	(0x40500000 + (x) * 0x1000)
+#define CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+
+#define CHP_DIR_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT	0x0000000F
+#define CHP_DIR_CQ_TKN_DEPTH_SEL_RSVD0			0xFFFFFFF0
+#define CHP_DIR_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT_LOC	0
+#define CHP_DIR_CQ_TKN_DEPTH_SEL_RSVD0_LOC		4
+
+#define CHP_DIR_CQ_WD_ENB(x) \
+	(0x40580000 + (x) * 0x1000)
+#define CHP_DIR_CQ_WD_ENB_RST 0x0
+
+#define CHP_DIR_CQ_WD_ENB_WD_ENABLE	0x00000001
+#define CHP_DIR_CQ_WD_ENB_RSVD0		0xFFFFFFFE
+#define CHP_DIR_CQ_WD_ENB_WD_ENABLE_LOC	0
+#define CHP_DIR_CQ_WD_ENB_RSVD0_LOC	1
+
+#define CHP_DIR_CQ_WPTR(x) \
+	(0x40600000 + (x) * 0x1000)
+#define CHP_DIR_CQ_WPTR_RST 0x0
+
+#define CHP_DIR_CQ_WPTR_WRITE_POINTER	0x00001FFF
+#define CHP_DIR_CQ_WPTR_RSVD0		0xFFFFE000
+#define CHP_DIR_CQ_WPTR_WRITE_POINTER_LOC	0
+#define CHP_DIR_CQ_WPTR_RSVD0_LOC		13
+
+#define CHP_DIR_CQ2VAS(x) \
+	(0x40680000 + (x) * 0x1000)
+#define CHP_DIR_CQ2VAS_RST 0x0
+
+#define CHP_DIR_CQ2VAS_CQ2VAS	0x0000001F
+#define CHP_DIR_CQ2VAS_RSVD0	0xFFFFFFE0
+#define CHP_DIR_CQ2VAS_CQ2VAS_LOC	0
+#define CHP_DIR_CQ2VAS_RSVD0_LOC	5
+
+#define CHP_HIST_LIST_BASE(x) \
+	(0x40700000 + (x) * 0x1000)
+#define CHP_HIST_LIST_BASE_RST 0x0
+
+#define CHP_HIST_LIST_BASE_BASE		0x00001FFF
+#define CHP_HIST_LIST_BASE_RSVD0	0xFFFFE000
+#define CHP_HIST_LIST_BASE_BASE_LOC	0
+#define CHP_HIST_LIST_BASE_RSVD0_LOC	13
+
+#define CHP_HIST_LIST_LIM(x) \
+	(0x40780000 + (x) * 0x1000)
+#define CHP_HIST_LIST_LIM_RST 0x0
+
+#define CHP_HIST_LIST_LIM_LIMIT	0x00001FFF
+#define CHP_HIST_LIST_LIM_RSVD0	0xFFFFE000
+#define CHP_HIST_LIST_LIM_LIMIT_LOC	0
+#define CHP_HIST_LIST_LIM_RSVD0_LOC	13
+
+#define CHP_HIST_LIST_POP_PTR(x) \
+	(0x40800000 + (x) * 0x1000)
+#define CHP_HIST_LIST_POP_PTR_RST 0x0
+
+#define CHP_HIST_LIST_POP_PTR_POP_PTR		0x00001FFF
+#define CHP_HIST_LIST_POP_PTR_GENERATION	0x00002000
+#define CHP_HIST_LIST_POP_PTR_RSVD0		0xFFFFC000
+#define CHP_HIST_LIST_POP_PTR_POP_PTR_LOC	0
+#define CHP_HIST_LIST_POP_PTR_GENERATION_LOC	13
+#define CHP_HIST_LIST_POP_PTR_RSVD0_LOC		14
+
+#define CHP_HIST_LIST_PUSH_PTR(x) \
+	(0x40880000 + (x) * 0x1000)
+#define CHP_HIST_LIST_PUSH_PTR_RST 0x0
+
+#define CHP_HIST_LIST_PUSH_PTR_PUSH_PTR		0x00001FFF
+#define CHP_HIST_LIST_PUSH_PTR_GENERATION	0x00002000
+#define CHP_HIST_LIST_PUSH_PTR_RSVD0		0xFFFFC000
+#define CHP_HIST_LIST_PUSH_PTR_PUSH_PTR_LOC	0
+#define CHP_HIST_LIST_PUSH_PTR_GENERATION_LOC	13
+#define CHP_HIST_LIST_PUSH_PTR_RSVD0_LOC	14
+
+#define CHP_LDB_CQ_DEPTH(x) \
+	(0x40900000 + (x) * 0x1000)
+#define CHP_LDB_CQ_DEPTH_RST 0x0
+
+#define CHP_LDB_CQ_DEPTH_DEPTH	0x000007FF
+#define CHP_LDB_CQ_DEPTH_RSVD0	0xFFFFF800
+#define CHP_LDB_CQ_DEPTH_DEPTH_LOC	0
+#define CHP_LDB_CQ_DEPTH_RSVD0_LOC	11
+
+#define CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+	(0x40980000 + (x) * 0x1000)
+#define CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+
+#define CHP_LDB_CQ_INT_DEPTH_THRSH_DEPTH_THRESHOLD	0x000007FF
+#define CHP_LDB_CQ_INT_DEPTH_THRSH_RSVD0		0xFFFFF800
+#define CHP_LDB_CQ_INT_DEPTH_THRSH_DEPTH_THRESHOLD_LOC	0
+#define CHP_LDB_CQ_INT_DEPTH_THRSH_RSVD0_LOC		11
+
+#define CHP_LDB_CQ_INT_ENB(x) \
+	(0x40a00000 + (x) * 0x1000)
+#define CHP_LDB_CQ_INT_ENB_RST 0x0
+
+#define CHP_LDB_CQ_INT_ENB_EN_TIM	0x00000001
+#define CHP_LDB_CQ_INT_ENB_EN_DEPTH	0x00000002
+#define CHP_LDB_CQ_INT_ENB_RSVD0	0xFFFFFFFC
+#define CHP_LDB_CQ_INT_ENB_EN_TIM_LOC	0
+#define CHP_LDB_CQ_INT_ENB_EN_DEPTH_LOC	1
+#define CHP_LDB_CQ_INT_ENB_RSVD0_LOC	2
+
+#define CHP_LDB_CQ_TMR_THRSH(x) \
+	(0x40b00000 + (x) * 0x1000)
+#define CHP_LDB_CQ_TMR_THRSH_RST 0x1
+
+#define CHP_LDB_CQ_TMR_THRSH_THRSH_0	0x00000001
+#define CHP_LDB_CQ_TMR_THRSH_THRSH_13_1	0x00003FFE
+#define CHP_LDB_CQ_TMR_THRSH_RSVD0	0xFFFFC000
+#define CHP_LDB_CQ_TMR_THRSH_THRSH_0_LOC	0
+#define CHP_LDB_CQ_TMR_THRSH_THRSH_13_1_LOC	1
+#define CHP_LDB_CQ_TMR_THRSH_RSVD0_LOC		14
+
+#define CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+	(0x40b80000 + (x) * 0x1000)
+#define CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+
+#define CHP_LDB_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT	0x0000000F
+#define CHP_LDB_CQ_TKN_DEPTH_SEL_RSVD0			0xFFFFFFF0
+#define CHP_LDB_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT_LOC	0
+#define CHP_LDB_CQ_TKN_DEPTH_SEL_RSVD0_LOC		4
+
+#define CHP_LDB_CQ_WD_ENB(x) \
+	(0x40c00000 + (x) * 0x1000)
+#define CHP_LDB_CQ_WD_ENB_RST 0x0
+
+#define CHP_LDB_CQ_WD_ENB_WD_ENABLE	0x00000001
+#define CHP_LDB_CQ_WD_ENB_RSVD0		0xFFFFFFFE
+#define CHP_LDB_CQ_WD_ENB_WD_ENABLE_LOC	0
+#define CHP_LDB_CQ_WD_ENB_RSVD0_LOC	1
+
+#define CHP_LDB_CQ_WPTR(x) \
+	(0x40c80000 + (x) * 0x1000)
+#define CHP_LDB_CQ_WPTR_RST 0x0
+
+#define CHP_LDB_CQ_WPTR_WRITE_POINTER	0x000007FF
+#define CHP_LDB_CQ_WPTR_RSVD0		0xFFFFF800
+#define CHP_LDB_CQ_WPTR_WRITE_POINTER_LOC	0
+#define CHP_LDB_CQ_WPTR_RSVD0_LOC		11
+
+#define CHP_LDB_CQ2VAS(x) \
+	(0x40d00000 + (x) * 0x1000)
+#define CHP_LDB_CQ2VAS_RST 0x0
+
+#define CHP_LDB_CQ2VAS_CQ2VAS	0x0000001F
+#define CHP_LDB_CQ2VAS_RSVD0	0xFFFFFFE0
+#define CHP_LDB_CQ2VAS_CQ2VAS_LOC	0
+#define CHP_LDB_CQ2VAS_RSVD0_LOC	5
+
+#define CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+
+#define CHP_CFG_CHP_CSR_CTRL_INT_COR_ALARM_DIS		0x00000001
+#define CHP_CFG_CHP_CSR_CTRL_INT_COR_SYND_DIS		0x00000002
+#define CHP_CFG_CHP_CSR_CTRL_INT_UNCR_ALARM_DIS		0x00000004
+#define CHP_CFG_CHP_CSR_CTRL_INT_UNC_SYND_DIS		0x00000008
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF0_ALARM_DIS		0x00000010
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF0_SYND_DIS		0x00000020
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF1_ALARM_DIS		0x00000040
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF1_SYND_DIS		0x00000080
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF2_ALARM_DIS		0x00000100
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF2_SYND_DIS		0x00000200
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF3_ALARM_DIS		0x00000400
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF3_SYND_DIS		0x00000800
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF4_ALARM_DIS		0x00001000
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF4_SYND_DIS		0x00002000
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF5_ALARM_DIS		0x00004000
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF5_SYND_DIS		0x00008000
+#define CHP_CFG_CHP_CSR_CTRL_DLB_COR_ALARM_ENABLE	0x00010000
+#define CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_LDB_CQ_MODE	0x00020000
+#define CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_DIR_CQ_MODE	0x00040000
+#define CHP_CFG_CHP_CSR_CTRL_PAD_WRITE_LDB		0x00080000
+#define CHP_CFG_CHP_CSR_CTRL_PAD_WRITE_DIR		0x00100000
+#define CHP_CFG_CHP_CSR_CTRL_PAD_FIRST_WRITE_LDB	0x00200000
+#define CHP_CFG_CHP_CSR_CTRL_PAD_FIRST_WRITE_DIR	0x00400000
+#define CHP_CFG_CHP_CSR_CTRL_RSVZ0			0xFF800000
+#define CHP_CFG_CHP_CSR_CTRL_INT_COR_ALARM_DIS_LOC		0
+#define CHP_CFG_CHP_CSR_CTRL_INT_COR_SYND_DIS_LOC		1
+#define CHP_CFG_CHP_CSR_CTRL_INT_UNCR_ALARM_DIS_LOC		2
+#define CHP_CFG_CHP_CSR_CTRL_INT_UNC_SYND_DIS_LOC		3
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF0_ALARM_DIS_LOC		4
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF0_SYND_DIS_LOC		5
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF1_ALARM_DIS_LOC		6
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF1_SYND_DIS_LOC		7
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF2_ALARM_DIS_LOC		8
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF2_SYND_DIS_LOC		9
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF3_ALARM_DIS_LOC		10
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF3_SYND_DIS_LOC		11
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF4_ALARM_DIS_LOC		12
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF4_SYND_DIS_LOC		13
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF5_ALARM_DIS_LOC		14
+#define CHP_CFG_CHP_CSR_CTRL_INT_INF5_SYND_DIS_LOC		15
+#define CHP_CFG_CHP_CSR_CTRL_DLB_COR_ALARM_ENABLE_LOC		16
+#define CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_LDB_CQ_MODE_LOC	17
+#define CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_DIR_CQ_MODE_LOC	18
+#define CHP_CFG_CHP_CSR_CTRL_PAD_WRITE_LDB_LOC			19
+#define CHP_CFG_CHP_CSR_CTRL_PAD_WRITE_DIR_LOC			20
+#define CHP_CFG_CHP_CSR_CTRL_PAD_FIRST_WRITE_LDB_LOC		21
+#define CHP_CFG_CHP_CSR_CTRL_PAD_FIRST_WRITE_DIR_LOC		22
+#define CHP_CFG_CHP_CSR_CTRL_RSVZ0_LOC				23
+
+#define CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+
+#define CHP_DIR_CQ_INTR_ARMED0_ARMED	0xFFFFFFFF
+#define CHP_DIR_CQ_INTR_ARMED0_ARMED_LOC	0
+
+#define CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+
+#define CHP_DIR_CQ_INTR_ARMED1_ARMED	0xFFFFFFFF
+#define CHP_DIR_CQ_INTR_ARMED1_ARMED_LOC	0
+
+#define CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+
+#define CHP_CFG_DIR_CQ_TIMER_CTL_SAMPLE_INTERVAL	0x000000FF
+#define CHP_CFG_DIR_CQ_TIMER_CTL_ENB			0x00000100
+#define CHP_CFG_DIR_CQ_TIMER_CTL_RSVZ0			0xFFFFFE00
+#define CHP_CFG_DIR_CQ_TIMER_CTL_SAMPLE_INTERVAL_LOC	0
+#define CHP_CFG_DIR_CQ_TIMER_CTL_ENB_LOC		8
+#define CHP_CFG_DIR_CQ_TIMER_CTL_RSVZ0_LOC		9
+
+#define CHP_CFG_DIR_WDTO_0 0x44000088
+#define CHP_CFG_DIR_WDTO_0_RST 0x0
+
+#define CHP_CFG_DIR_WDTO_0_WDTO	0xFFFFFFFF
+#define CHP_CFG_DIR_WDTO_0_WDTO_LOC	0
+
+#define CHP_CFG_DIR_WDTO_1 0x4400008c
+#define CHP_CFG_DIR_WDTO_1_RST 0x0
+
+#define CHP_CFG_DIR_WDTO_1_WDTO	0xFFFFFFFF
+#define CHP_CFG_DIR_WDTO_1_WDTO_LOC	0
+
+#define CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+
+#define CHP_CFG_DIR_WD_DISABLE0_WD_DISABLE	0xFFFFFFFF
+#define CHP_CFG_DIR_WD_DISABLE0_WD_DISABLE_LOC	0
+
+#define CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+
+#define CHP_CFG_DIR_WD_DISABLE1_WD_DISABLE	0xFFFFFFFF
+#define CHP_CFG_DIR_WD_DISABLE1_WD_DISABLE_LOC	0
+
+#define CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_SAMPLE_INTERVAL	0x0FFFFFFF
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_ENB			0x10000000
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_RSVZ0		0xE0000000
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_SAMPLE_INTERVAL_LOC	0
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_ENB_LOC		28
+#define CHP_CFG_DIR_WD_ENB_INTERVAL_RSVZ0_LOC		29
+
+#define CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+
+#define CHP_CFG_DIR_WD_THRESHOLD_WD_THRESHOLD	0x000000FF
+#define CHP_CFG_DIR_WD_THRESHOLD_RSVZ0		0xFFFFFF00
+#define CHP_CFG_DIR_WD_THRESHOLD_WD_THRESHOLD_LOC	0
+#define CHP_CFG_DIR_WD_THRESHOLD_RSVZ0_LOC		8
+
+#define CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+
+#define CHP_LDB_CQ_INTR_ARMED0_ARMED	0xFFFFFFFF
+#define CHP_LDB_CQ_INTR_ARMED0_ARMED_LOC	0
+
+#define CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+
+#define CHP_LDB_CQ_INTR_ARMED1_ARMED	0xFFFFFFFF
+#define CHP_LDB_CQ_INTR_ARMED1_ARMED_LOC	0
+
+#define CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+
+#define CHP_CFG_LDB_CQ_TIMER_CTL_SAMPLE_INTERVAL	0x000000FF
+#define CHP_CFG_LDB_CQ_TIMER_CTL_ENB			0x00000100
+#define CHP_CFG_LDB_CQ_TIMER_CTL_RSVZ0			0xFFFFFE00
+#define CHP_CFG_LDB_CQ_TIMER_CTL_SAMPLE_INTERVAL_LOC	0
+#define CHP_CFG_LDB_CQ_TIMER_CTL_ENB_LOC		8
+#define CHP_CFG_LDB_CQ_TIMER_CTL_RSVZ0_LOC		9
+
+#define CHP_CFG_LDB_WDTO_0 0x440000dc
+#define CHP_CFG_LDB_WDTO_0_RST 0x0
+
+#define CHP_CFG_LDB_WDTO_0_WDTO	0xFFFFFFFF
+#define CHP_CFG_LDB_WDTO_0_WDTO_LOC	0
+
+#define CHP_CFG_LDB_WDTO_1 0x440000e0
+#define CHP_CFG_LDB_WDTO_1_RST 0x0
+
+#define CHP_CFG_LDB_WDTO_1_WDTO	0xFFFFFFFF
+#define CHP_CFG_LDB_WDTO_1_WDTO_LOC	0
+
+#define CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+
+#define CHP_CFG_LDB_WD_DISABLE0_WD_DISABLE	0xFFFFFFFF
+#define CHP_CFG_LDB_WD_DISABLE0_WD_DISABLE_LOC	0
+
+#define CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+
+#define CHP_CFG_LDB_WD_DISABLE1_WD_DISABLE	0xFFFFFFFF
+#define CHP_CFG_LDB_WD_DISABLE1_WD_DISABLE_LOC	0
+
+#define CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_SAMPLE_INTERVAL	0x0FFFFFFF
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_ENB			0x10000000
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_RSVZ0		0xE0000000
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_SAMPLE_INTERVAL_LOC	0
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_ENB_LOC		28
+#define CHP_CFG_LDB_WD_ENB_INTERVAL_RSVZ0_LOC		29
+
+#define CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+
+#define CHP_CFG_LDB_WD_THRESHOLD_WD_THRESHOLD	0x000000FF
+#define CHP_CFG_LDB_WD_THRESHOLD_RSVZ0		0xFFFFFF00
+#define CHP_CFG_LDB_WD_THRESHOLD_WD_THRESHOLD_LOC	0
+#define CHP_CFG_LDB_WD_THRESHOLD_RSVZ0_LOC		8
+
+#define CHP_SMON_COMPARE0 0x4c000000
+#define CHP_SMON_COMPARE0_RST 0x0
+
+#define CHP_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define CHP_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define CHP_SMON_COMPARE1 0x4c000004
+#define CHP_SMON_COMPARE1_RST 0x0
+
+#define CHP_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define CHP_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define CHP_SMON_CFG0 0x4c000008
+#define CHP_SMON_CFG0_RST 0x40000000
+
+#define CHP_SMON_CFG0_SMON_ENABLE		0x00000001
+#define CHP_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define CHP_SMON_CFG0_RSVZ0			0x0000000C
+#define CHP_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define CHP_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define CHP_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define CHP_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define CHP_SMON_CFG0_SMON_MODE			0x0000F000
+#define CHP_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define CHP_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define CHP_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define CHP_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define CHP_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define CHP_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define CHP_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define CHP_SMON_CFG0_RSVZ1			0x00800000
+#define CHP_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define CHP_SMON_CFG0_RSVZ2			0x20000000
+#define CHP_SMON_CFG0_VERSION			0xC0000000
+#define CHP_SMON_CFG0_SMON_ENABLE_LOC			0
+#define CHP_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define CHP_SMON_CFG0_RSVZ0_LOC				2
+#define CHP_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define CHP_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define CHP_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define CHP_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define CHP_SMON_CFG0_SMON_MODE_LOC			12
+#define CHP_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define CHP_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define CHP_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define CHP_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define CHP_SMON_CFG0_STOPTIMEROVFL_LOC			20
+#define CHP_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define CHP_SMON_CFG0_STATTIMEROVFL_LOC			22
+#define CHP_SMON_CFG0_RSVZ1_LOC				23
+#define CHP_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define CHP_SMON_CFG0_RSVZ2_LOC				29
+#define CHP_SMON_CFG0_VERSION_LOC			30
+
+#define CHP_SMON_CFG1 0x4c00000c
+#define CHP_SMON_CFG1_RST 0x0
+
+#define CHP_SMON_CFG1_MODE0	0x000000FF
+#define CHP_SMON_CFG1_MODE1	0x0000FF00
+#define CHP_SMON_CFG1_RSVZ0	0xFFFF0000
+#define CHP_SMON_CFG1_MODE0_LOC	0
+#define CHP_SMON_CFG1_MODE1_LOC	8
+#define CHP_SMON_CFG1_RSVZ0_LOC	16
+
+#define CHP_SMON_ACTIVITYCNTR0 0x4c000010
+#define CHP_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define CHP_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define CHP_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define CHP_SMON_ACTIVITYCNTR1 0x4c000014
+#define CHP_SMON_ACTIVITYCNTR1_RST 0x0
 
-#define CHP_CFG_LDB_VAS_CRD(x) \
-	(0x40080000 + (x) * 0x1000)
-#define CHP_CFG_LDB_VAS_CRD_RST 0x0
+#define CHP_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define CHP_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
 
-#define CHP_CFG_LDB_VAS_CRD_COUNT	0x00007FFF
-#define CHP_CFG_LDB_VAS_CRD_RSVD0	0xFFFF8000
-#define CHP_CFG_LDB_VAS_CRD_COUNT_LOC	0
-#define CHP_CFG_LDB_VAS_CRD_RSVD0_LOC	15
+#define CHP_SMON_MAX_TMR 0x4c000018
+#define CHP_SMON_MAX_TMR_RST 0x0
+
+#define CHP_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define CHP_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define CHP_SMON_TMR 0x4c00001c
+#define CHP_SMON_TMR_RST 0x0
+
+#define CHP_SMON_TMR_TIMER	0xFFFFFFFF
+#define CHP_SMON_TMR_TIMER_LOC	0
+
+#define CHP_CTRL_DIAG_02 0x4c000028
+#define CHP_CTRL_DIAG_02_RST 0x1555
+
+#define CHP_CTRL_DIAG_02_EGRESS_CREDIT_STATUS_EMPTY			0x00000001
+#define CHP_CTRL_DIAG_02_EGRESS_CREDIT_STATUS_AFULL			0x00000002
+#define CHP_CTRL_DIAG_02_CHP_OUTBOUND_HCW_PIPE_CREDIT_STATUS_EMPTY	0x00000004
+#define CHP_CTRL_DIAG_02_CHP_OUTBOUND_HCW_PIPE_CREDIT_STATUS_AFULL	0x00000008
+#define CHP_CTRL_DIAG_02_CHP_LSP_AP_CMP_PIPE_CREDIT_STATUS_EMPTY	0x00000010
+#define CHP_CTRL_DIAG_02_CHP_LSP_AP_CMP_PIPE_CREDIT_STATUS_AFULL	0x00000020
+#define CHP_CTRL_DIAG_02_CHP_LSP_TOK_PIPE_CREDIT_STATUS_EMPTY		0x00000040
+#define CHP_CTRL_DIAG_02_CHP_LSP_TOK_PIPE_CREDIT_STATUS_AFULL		0x00000080
+#define CHP_CTRL_DIAG_02_CHP_ROP_PIPE_CREDIT_STATUS_EMPTY		0x00000100
+#define CHP_CTRL_DIAG_02_CHP_ROP_PIPE_CREDIT_STATUS_AFULL		0x00000200
+#define CHP_CTRL_DIAG_02_QED_TO_CQ_PIPE_CREDIT_STATUS_EMPTY		0x00000400
+#define CHP_CTRL_DIAG_02_QED_TO_CQ_PIPE_CREDIT_STATUS_AFULL		0x00000800
+#define CHP_CTRL_DIAG_02_EGRESS_LSP_TOKEN_CREDIT_STATUS_EMPTY		0x00001000
+#define CHP_CTRL_DIAG_02_EGRESS_LSP_TOKEN_CREDIT_STATUS_AFULL		0x00002000
+#define CHP_CTRL_DIAG_02_RSVD0						0xFFFFC000
+#define CHP_CTRL_DIAG_02_EGRESS_CREDIT_STATUS_EMPTY_LOC			0
+#define CHP_CTRL_DIAG_02_EGRESS_CREDIT_STATUS_AFULL_LOC			1
+#define CHP_CTRL_DIAG_02_CHP_OUTBOUND_HCW_PIPE_CREDIT_STATUS_EMPTY_LOC	2
+#define CHP_CTRL_DIAG_02_CHP_OUTBOUND_HCW_PIPE_CREDIT_STATUS_AFULL_LOC	3
+#define CHP_CTRL_DIAG_02_CHP_LSP_AP_CMP_PIPE_CREDIT_STATUS_EMPTY_LOC	4
+#define CHP_CTRL_DIAG_02_CHP_LSP_AP_CMP_PIPE_CREDIT_STATUS_AFULL_LOC	5
+#define CHP_CTRL_DIAG_02_CHP_LSP_TOK_PIPE_CREDIT_STATUS_EMPTY_LOC	6
+#define CHP_CTRL_DIAG_02_CHP_LSP_TOK_PIPE_CREDIT_STATUS_AFULL_LOC	7
+#define CHP_CTRL_DIAG_02_CHP_ROP_PIPE_CREDIT_STATUS_EMPTY_LOC		8
+#define CHP_CTRL_DIAG_02_CHP_ROP_PIPE_CREDIT_STATUS_AFULL_LOC		9
+#define CHP_CTRL_DIAG_02_QED_TO_CQ_PIPE_CREDIT_STATUS_EMPTY_LOC		10
+#define CHP_CTRL_DIAG_02_QED_TO_CQ_PIPE_CREDIT_STATUS_AFULL_LOC		11
+#define CHP_CTRL_DIAG_02_EGRESS_LSP_TOKEN_CREDIT_STATUS_EMPTY_LOC	12
+#define CHP_CTRL_DIAG_02_EGRESS_LSP_TOKEN_CREDIT_STATUS_AFULL_LOC	13
+#define CHP_CTRL_DIAG_02_RSVD0_LOC					14
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI0	0x000000FF
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI1	0x0000FF00
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI2	0x00FF0000
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI3	0xFF000000
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI0_LOC	0
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI1_LOC	8
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI2_LOC	16
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_PRI3_LOC	24
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RSVZ0	0xFFFFFFFF
+#define DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RSVZ0_LOC	0
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI0	0x000000FF
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI1	0x0000FF00
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI2	0x00FF0000
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI3	0xFF000000
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI0_LOC	0
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI1_LOC	8
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI2_LOC	16
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI3_LOC	24
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RSVZ0	0xFFFFFFFF
+#define DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RSVZ0_LOC	0
+
+#define DP_DIR_CSR_CTRL 0x54000010
+#define DP_DIR_CSR_CTRL_RST 0x0
+
+#define DP_DIR_CSR_CTRL_INT_COR_ALARM_DIS	0x00000001
+#define DP_DIR_CSR_CTRL_INT_COR_SYND_DIS	0x00000002
+#define DP_DIR_CSR_CTRL_INT_UNCR_ALARM_DIS	0x00000004
+#define DP_DIR_CSR_CTRL_INT_UNC_SYND_DIS	0x00000008
+#define DP_DIR_CSR_CTRL_INT_INF0_ALARM_DIS	0x00000010
+#define DP_DIR_CSR_CTRL_INT_INF0_SYND_DIS	0x00000020
+#define DP_DIR_CSR_CTRL_INT_INF1_ALARM_DIS	0x00000040
+#define DP_DIR_CSR_CTRL_INT_INF1_SYND_DIS	0x00000080
+#define DP_DIR_CSR_CTRL_INT_INF2_ALARM_DIS	0x00000100
+#define DP_DIR_CSR_CTRL_INT_INF2_SYND_DIS	0x00000200
+#define DP_DIR_CSR_CTRL_INT_INF3_ALARM_DIS	0x00000400
+#define DP_DIR_CSR_CTRL_INT_INF3_SYND_DIS	0x00000800
+#define DP_DIR_CSR_CTRL_INT_INF4_ALARM_DIS	0x00001000
+#define DP_DIR_CSR_CTRL_INT_INF4_SYND_DIS	0x00002000
+#define DP_DIR_CSR_CTRL_INT_INF5_ALARM_DIS	0x00004000
+#define DP_DIR_CSR_CTRL_INT_INF5_SYND_DIS	0x00008000
+#define DP_DIR_CSR_CTRL_RSVZ0			0xFFFF0000
+#define DP_DIR_CSR_CTRL_INT_COR_ALARM_DIS_LOC	0
+#define DP_DIR_CSR_CTRL_INT_COR_SYND_DIS_LOC	1
+#define DP_DIR_CSR_CTRL_INT_UNCR_ALARM_DIS_LOC	2
+#define DP_DIR_CSR_CTRL_INT_UNC_SYND_DIS_LOC	3
+#define DP_DIR_CSR_CTRL_INT_INF0_ALARM_DIS_LOC	4
+#define DP_DIR_CSR_CTRL_INT_INF0_SYND_DIS_LOC	5
+#define DP_DIR_CSR_CTRL_INT_INF1_ALARM_DIS_LOC	6
+#define DP_DIR_CSR_CTRL_INT_INF1_SYND_DIS_LOC	7
+#define DP_DIR_CSR_CTRL_INT_INF2_ALARM_DIS_LOC	8
+#define DP_DIR_CSR_CTRL_INT_INF2_SYND_DIS_LOC	9
+#define DP_DIR_CSR_CTRL_INT_INF3_ALARM_DIS_LOC	10
+#define DP_DIR_CSR_CTRL_INT_INF3_SYND_DIS_LOC	11
+#define DP_DIR_CSR_CTRL_INT_INF4_ALARM_DIS_LOC	12
+#define DP_DIR_CSR_CTRL_INT_INF4_SYND_DIS_LOC	13
+#define DP_DIR_CSR_CTRL_INT_INF5_ALARM_DIS_LOC	14
+#define DP_DIR_CSR_CTRL_INT_INF5_SYND_DIS_LOC	15
+#define DP_DIR_CSR_CTRL_RSVZ0_LOC		16
+
+#define DP_SMON_ACTIVITYCNTR0 0x5c000058
+#define DP_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define DP_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define DP_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define DP_SMON_ACTIVITYCNTR1 0x5c00005c
+#define DP_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define DP_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define DP_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define DP_SMON_COMPARE0 0x5c000060
+#define DP_SMON_COMPARE0_RST 0x0
+
+#define DP_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define DP_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define DP_SMON_COMPARE1 0x5c000064
+#define DP_SMON_COMPARE1_RST 0x0
+
+#define DP_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define DP_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define DP_SMON_CFG0 0x5c000068
+#define DP_SMON_CFG0_RST 0x40000000
+
+#define DP_SMON_CFG0_SMON_ENABLE		0x00000001
+#define DP_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define DP_SMON_CFG0_RSVZ0			0x0000000C
+#define DP_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define DP_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define DP_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define DP_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define DP_SMON_CFG0_SMON_MODE			0x0000F000
+#define DP_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define DP_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define DP_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define DP_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define DP_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define DP_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define DP_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define DP_SMON_CFG0_RSVZ1			0x00800000
+#define DP_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define DP_SMON_CFG0_RSVZ2			0x20000000
+#define DP_SMON_CFG0_VERSION			0xC0000000
+#define DP_SMON_CFG0_SMON_ENABLE_LOC		0
+#define DP_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC	1
+#define DP_SMON_CFG0_RSVZ0_LOC			2
+#define DP_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define DP_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define DP_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define DP_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define DP_SMON_CFG0_SMON_MODE_LOC		12
+#define DP_SMON_CFG0_STOPCOUNTEROVFL_LOC	16
+#define DP_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define DP_SMON_CFG0_STATCOUNTER0OVFL_LOC	18
+#define DP_SMON_CFG0_STATCOUNTER1OVFL_LOC	19
+#define DP_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define DP_SMON_CFG0_INTTIMEROVFL_LOC		21
+#define DP_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define DP_SMON_CFG0_RSVZ1_LOC			23
+#define DP_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define DP_SMON_CFG0_RSVZ2_LOC			29
+#define DP_SMON_CFG0_VERSION_LOC		30
+
+#define DP_SMON_CFG1 0x5c00006c
+#define DP_SMON_CFG1_RST 0x0
+
+#define DP_SMON_CFG1_MODE0	0x000000FF
+#define DP_SMON_CFG1_MODE1	0x0000FF00
+#define DP_SMON_CFG1_RSVZ0	0xFFFF0000
+#define DP_SMON_CFG1_MODE0_LOC	0
+#define DP_SMON_CFG1_MODE1_LOC	8
+#define DP_SMON_CFG1_RSVZ0_LOC	16
+
+#define DP_SMON_MAX_TMR 0x5c000070
+#define DP_SMON_MAX_TMR_RST 0x0
+
+#define DP_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define DP_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define DP_SMON_TMR 0x5c000074
+#define DP_SMON_TMR_RST 0x0
+
+#define DP_SMON_TMR_TIMER	0xFFFFFFFF
+#define DP_SMON_TMR_TIMER_LOC	0
+
+#define DQED_SMON_ACTIVITYCNTR0 0x6c000024
+#define DQED_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define DQED_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define DQED_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define DQED_SMON_ACTIVITYCNTR1 0x6c000028
+#define DQED_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define DQED_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define DQED_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define DQED_SMON_COMPARE0 0x6c00002c
+#define DQED_SMON_COMPARE0_RST 0x0
+
+#define DQED_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define DQED_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define DQED_SMON_COMPARE1 0x6c000030
+#define DQED_SMON_COMPARE1_RST 0x0
+
+#define DQED_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define DQED_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define DQED_SMON_CFG0 0x6c000034
+#define DQED_SMON_CFG0_RST 0x40000000
+
+#define DQED_SMON_CFG0_SMON_ENABLE		0x00000001
+#define DQED_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define DQED_SMON_CFG0_RSVZ0			0x0000000C
+#define DQED_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define DQED_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define DQED_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define DQED_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define DQED_SMON_CFG0_SMON_MODE		0x0000F000
+#define DQED_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define DQED_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define DQED_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define DQED_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define DQED_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define DQED_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define DQED_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define DQED_SMON_CFG0_RSVZ1			0x00800000
+#define DQED_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define DQED_SMON_CFG0_RSVZ2			0x20000000
+#define DQED_SMON_CFG0_VERSION			0xC0000000
+#define DQED_SMON_CFG0_SMON_ENABLE_LOC			0
+#define DQED_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define DQED_SMON_CFG0_RSVZ0_LOC			2
+#define DQED_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define DQED_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define DQED_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define DQED_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define DQED_SMON_CFG0_SMON_MODE_LOC			12
+#define DQED_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define DQED_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define DQED_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define DQED_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define DQED_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define DQED_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define DQED_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define DQED_SMON_CFG0_RSVZ1_LOC			23
+#define DQED_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define DQED_SMON_CFG0_RSVZ2_LOC			29
+#define DQED_SMON_CFG0_VERSION_LOC			30
+
+#define DQED_SMON_CFG1 0x6c000038
+#define DQED_SMON_CFG1_RST 0x0
+
+#define DQED_SMON_CFG1_MODE0	0x000000FF
+#define DQED_SMON_CFG1_MODE1	0x0000FF00
+#define DQED_SMON_CFG1_RSVZ0	0xFFFF0000
+#define DQED_SMON_CFG1_MODE0_LOC	0
+#define DQED_SMON_CFG1_MODE1_LOC	8
+#define DQED_SMON_CFG1_RSVZ0_LOC	16
+
+#define DQED_SMON_MAX_TMR 0x6c00003c
+#define DQED_SMON_MAX_TMR_RST 0x0
+
+#define DQED_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define DQED_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define DQED_SMON_TMR 0x6c000040
+#define DQED_SMON_TMR_RST 0x0
+
+#define DQED_SMON_TMR_TIMER	0xFFFFFFFF
+#define DQED_SMON_TMR_TIMER_LOC	0
+
+#define QED_SMON_ACTIVITYCNTR0 0x7c000024
+#define QED_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define QED_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define QED_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define QED_SMON_ACTIVITYCNTR1 0x7c000028
+#define QED_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define QED_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define QED_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define QED_SMON_COMPARE0 0x7c00002c
+#define QED_SMON_COMPARE0_RST 0x0
+
+#define QED_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define QED_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define QED_SMON_COMPARE1 0x7c000030
+#define QED_SMON_COMPARE1_RST 0x0
+
+#define QED_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define QED_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define QED_SMON_CFG0 0x7c000034
+#define QED_SMON_CFG0_RST 0x40000000
+
+#define QED_SMON_CFG0_SMON_ENABLE		0x00000001
+#define QED_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define QED_SMON_CFG0_RSVZ0			0x0000000C
+#define QED_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define QED_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define QED_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define QED_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define QED_SMON_CFG0_SMON_MODE			0x0000F000
+#define QED_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define QED_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define QED_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define QED_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define QED_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define QED_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define QED_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define QED_SMON_CFG0_RSVZ1			0x00800000
+#define QED_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define QED_SMON_CFG0_RSVZ2			0x20000000
+#define QED_SMON_CFG0_VERSION			0xC0000000
+#define QED_SMON_CFG0_SMON_ENABLE_LOC			0
+#define QED_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define QED_SMON_CFG0_RSVZ0_LOC				2
+#define QED_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define QED_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define QED_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define QED_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define QED_SMON_CFG0_SMON_MODE_LOC			12
+#define QED_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define QED_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define QED_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define QED_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define QED_SMON_CFG0_STOPTIMEROVFL_LOC			20
+#define QED_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define QED_SMON_CFG0_STATTIMEROVFL_LOC			22
+#define QED_SMON_CFG0_RSVZ1_LOC				23
+#define QED_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define QED_SMON_CFG0_RSVZ2_LOC				29
+#define QED_SMON_CFG0_VERSION_LOC			30
+
+#define QED_SMON_CFG1 0x7c000038
+#define QED_SMON_CFG1_RST 0x0
+
+#define QED_SMON_CFG1_MODE0	0x000000FF
+#define QED_SMON_CFG1_MODE1	0x0000FF00
+#define QED_SMON_CFG1_RSVZ0	0xFFFF0000
+#define QED_SMON_CFG1_MODE0_LOC	0
+#define QED_SMON_CFG1_MODE1_LOC	8
+#define QED_SMON_CFG1_RSVZ0_LOC	16
+
+#define QED_SMON_MAX_TMR 0x7c00003c
+#define QED_SMON_MAX_TMR_RST 0x0
+
+#define QED_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define QED_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define QED_SMON_TMR 0x7c000040
+#define QED_SMON_TMR_RST 0x0
+
+#define QED_SMON_TMR_TIMER	0xFFFFFFFF
+#define QED_SMON_TMR_TIMER_LOC	0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI0	0x000000FF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI1	0x0000FF00
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI2	0x00FF0000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI3	0xFF000000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI0_LOC	0
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI1_LOC	8
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI2_LOC	16
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_PRI3_LOC	24
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RSVZ0	0xFFFFFFFF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RSVZ0_LOC	0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI0	0x000000FF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI1	0x0000FF00
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI2	0x00FF0000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI3	0xFF000000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI0_LOC	0
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI1_LOC	8
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI2_LOC	16
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_0_PRI3_LOC	24
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RSVZ0	0xFFFFFFFF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RSVZ0_LOC	0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI0	0x000000FF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI1	0x0000FF00
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI2	0x00FF0000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI3	0xFF000000
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI0_LOC	0
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI1_LOC	8
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI2_LOC	16
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_PRI3_LOC	24
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RSVZ0	0xFFFFFFFF
+#define NALB_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RSVZ0_LOC	0
+
+#define NALB_SMON_ACTIVITYCNTR0 0x8c000064
+#define NALB_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define NALB_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define NALB_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define NALB_SMON_ACTIVITYCNTR1 0x8c000068
+#define NALB_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define NALB_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define NALB_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define NALB_SMON_COMPARE0 0x8c00006c
+#define NALB_SMON_COMPARE0_RST 0x0
+
+#define NALB_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define NALB_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define NALB_SMON_COMPARE1 0x8c000070
+#define NALB_SMON_COMPARE1_RST 0x0
+
+#define NALB_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define NALB_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define NALB_SMON_CFG0 0x8c000074
+#define NALB_SMON_CFG0_RST 0x40000000
+
+#define NALB_SMON_CFG0_SMON_ENABLE		0x00000001
+#define NALB_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define NALB_SMON_CFG0_RSVZ0			0x0000000C
+#define NALB_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define NALB_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define NALB_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define NALB_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define NALB_SMON_CFG0_SMON_MODE		0x0000F000
+#define NALB_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define NALB_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define NALB_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define NALB_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define NALB_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define NALB_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define NALB_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define NALB_SMON_CFG0_RSVZ1			0x00800000
+#define NALB_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define NALB_SMON_CFG0_RSVZ2			0x20000000
+#define NALB_SMON_CFG0_VERSION			0xC0000000
+#define NALB_SMON_CFG0_SMON_ENABLE_LOC			0
+#define NALB_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define NALB_SMON_CFG0_RSVZ0_LOC			2
+#define NALB_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define NALB_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define NALB_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define NALB_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define NALB_SMON_CFG0_SMON_MODE_LOC			12
+#define NALB_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define NALB_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define NALB_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define NALB_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define NALB_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define NALB_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define NALB_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define NALB_SMON_CFG0_RSVZ1_LOC			23
+#define NALB_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define NALB_SMON_CFG0_RSVZ2_LOC			29
+#define NALB_SMON_CFG0_VERSION_LOC			30
+
+#define NALB_SMON_CFG1 0x8c000078
+#define NALB_SMON_CFG1_RST 0x0
+
+#define NALB_SMON_CFG1_MODE0	0x000000FF
+#define NALB_SMON_CFG1_MODE1	0x0000FF00
+#define NALB_SMON_CFG1_RSVZ0	0xFFFF0000
+#define NALB_SMON_CFG1_MODE0_LOC	0
+#define NALB_SMON_CFG1_MODE1_LOC	8
+#define NALB_SMON_CFG1_RSVZ0_LOC	16
+
+#define NALB_SMON_MAX_TMR 0x8c00007c
+#define NALB_SMON_MAX_TMR_RST 0x0
+
+#define NALB_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define NALB_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define NALB_SMON_TMR 0x8c000080
+#define NALB_SMON_TMR_RST 0x0
+
+#define NALB_SMON_TMR_TIMER	0xFFFFFFFF
+#define NALB_SMON_TMR_TIMER_LOC	0
+
+#define RO_GRP_0_SLT_SHFT(x) \
+	(0x96000000 + (x) * 0x4)
+#define RO_GRP_0_SLT_SHFT_RST 0x0
+
+#define RO_GRP_0_SLT_SHFT_CHANGE	0x000003FF
+#define RO_GRP_0_SLT_SHFT_RSVD0		0xFFFFFC00
+#define RO_GRP_0_SLT_SHFT_CHANGE_LOC	0
+#define RO_GRP_0_SLT_SHFT_RSVD0_LOC	10
+
+#define RO_GRP_1_SLT_SHFT(x) \
+	(0x96010000 + (x) * 0x4)
+#define RO_GRP_1_SLT_SHFT_RST 0x0
+
+#define RO_GRP_1_SLT_SHFT_CHANGE	0x000003FF
+#define RO_GRP_1_SLT_SHFT_RSVD0		0xFFFFFC00
+#define RO_GRP_1_SLT_SHFT_CHANGE_LOC	0
+#define RO_GRP_1_SLT_SHFT_RSVD0_LOC	10
+
+#define RO_GRP_SN_MODE 0x94000000
+#define RO_GRP_SN_MODE_RST 0x0
+
+#define RO_GRP_SN_MODE_SN_MODE_0	0x00000007
+#define RO_GRP_SN_MODE_RSZV0		0x000000F8
+#define RO_GRP_SN_MODE_SN_MODE_1	0x00000700
+#define RO_GRP_SN_MODE_RSZV1		0xFFFFF800
+#define RO_GRP_SN_MODE_SN_MODE_0_LOC	0
+#define RO_GRP_SN_MODE_RSZV0_LOC	3
+#define RO_GRP_SN_MODE_SN_MODE_1_LOC	8
+#define RO_GRP_SN_MODE_RSZV1_LOC	11
+
+#define RO_CFG_CTRL_GENERAL_0 0x9c000000
+#define RO_CFG_CTRL_GENERAL_0_RST 0x0
+
+#define RO_CFG_CTRL_GENERAL_0_UNIT_SINGLE_STEP_MODE	0x00000001
+#define RO_CFG_CTRL_GENERAL_0_RR_EN			0x00000002
+#define RO_CFG_CTRL_GENERAL_0_RSZV0			0xFFFFFFFC
+#define RO_CFG_CTRL_GENERAL_0_UNIT_SINGLE_STEP_MODE_LOC	0
+#define RO_CFG_CTRL_GENERAL_0_RR_EN_LOC			1
+#define RO_CFG_CTRL_GENERAL_0_RSZV0_LOC			2
+
+#define RO_SMON_ACTIVITYCNTR0 0x9c000030
+#define RO_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define RO_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define RO_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define RO_SMON_ACTIVITYCNTR1 0x9c000034
+#define RO_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define RO_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define RO_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define RO_SMON_COMPARE0 0x9c000038
+#define RO_SMON_COMPARE0_RST 0x0
+
+#define RO_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define RO_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define RO_SMON_COMPARE1 0x9c00003c
+#define RO_SMON_COMPARE1_RST 0x0
+
+#define RO_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define RO_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define RO_SMON_CFG0 0x9c000040
+#define RO_SMON_CFG0_RST 0x40000000
+
+#define RO_SMON_CFG0_SMON_ENABLE		0x00000001
+#define RO_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define RO_SMON_CFG0_RSVZ0			0x0000000C
+#define RO_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define RO_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define RO_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define RO_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define RO_SMON_CFG0_SMON_MODE			0x0000F000
+#define RO_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define RO_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define RO_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define RO_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define RO_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define RO_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define RO_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define RO_SMON_CFG0_RSVZ1			0x00800000
+#define RO_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define RO_SMON_CFG0_RSVZ2			0x20000000
+#define RO_SMON_CFG0_VERSION			0xC0000000
+#define RO_SMON_CFG0_SMON_ENABLE_LOC		0
+#define RO_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC	1
+#define RO_SMON_CFG0_RSVZ0_LOC			2
+#define RO_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define RO_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define RO_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define RO_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define RO_SMON_CFG0_SMON_MODE_LOC		12
+#define RO_SMON_CFG0_STOPCOUNTEROVFL_LOC	16
+#define RO_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define RO_SMON_CFG0_STATCOUNTER0OVFL_LOC	18
+#define RO_SMON_CFG0_STATCOUNTER1OVFL_LOC	19
+#define RO_SMON_CFG0_STOPTIMEROVFL_LOC		20
+#define RO_SMON_CFG0_INTTIMEROVFL_LOC		21
+#define RO_SMON_CFG0_STATTIMEROVFL_LOC		22
+#define RO_SMON_CFG0_RSVZ1_LOC			23
+#define RO_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define RO_SMON_CFG0_RSVZ2_LOC			29
+#define RO_SMON_CFG0_VERSION_LOC		30
+
+#define RO_SMON_CFG1 0x9c000044
+#define RO_SMON_CFG1_RST 0x0
+
+#define RO_SMON_CFG1_MODE0	0x000000FF
+#define RO_SMON_CFG1_MODE1	0x0000FF00
+#define RO_SMON_CFG1_RSVZ0	0xFFFF0000
+#define RO_SMON_CFG1_MODE0_LOC	0
+#define RO_SMON_CFG1_MODE1_LOC	8
+#define RO_SMON_CFG1_RSVZ0_LOC	16
+
+#define RO_SMON_MAX_TMR 0x9c000048
+#define RO_SMON_MAX_TMR_RST 0x0
+
+#define RO_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define RO_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define RO_SMON_TMR 0x9c00004c
+#define RO_SMON_TMR_RST 0x0
+
+#define RO_SMON_TMR_TIMER	0xFFFFFFFF
+#define RO_SMON_TMR_TIMER_LOC	0
+
+#define LSP_CQ2PRIOV(x) \
+	(0xa0000000 + (x) * 0x1000)
+#define LSP_CQ2PRIOV_RST 0x0
+
+#define LSP_CQ2PRIOV_PRIO	0x00FFFFFF
+#define LSP_CQ2PRIOV_V		0xFF000000
+#define LSP_CQ2PRIOV_PRIO_LOC	0
+#define LSP_CQ2PRIOV_V_LOC	24
+
+#define LSP_CQ2QID0(x) \
+	(0xa0080000 + (x) * 0x1000)
+#define LSP_CQ2QID0_RST 0x0
+
+#define LSP_CQ2QID0_QID_P0	0x0000007F
+#define LSP_CQ2QID0_RSVD3	0x00000080
+#define LSP_CQ2QID0_QID_P1	0x00007F00
+#define LSP_CQ2QID0_RSVD2	0x00008000
+#define LSP_CQ2QID0_QID_P2	0x007F0000
+#define LSP_CQ2QID0_RSVD1	0x00800000
+#define LSP_CQ2QID0_QID_P3	0x7F000000
+#define LSP_CQ2QID0_RSVD0	0x80000000
+#define LSP_CQ2QID0_QID_P0_LOC	0
+#define LSP_CQ2QID0_RSVD3_LOC	7
+#define LSP_CQ2QID0_QID_P1_LOC	8
+#define LSP_CQ2QID0_RSVD2_LOC	15
+#define LSP_CQ2QID0_QID_P2_LOC	16
+#define LSP_CQ2QID0_RSVD1_LOC	23
+#define LSP_CQ2QID0_QID_P3_LOC	24
+#define LSP_CQ2QID0_RSVD0_LOC	31
+
+#define LSP_CQ2QID1(x) \
+	(0xa0100000 + (x) * 0x1000)
+#define LSP_CQ2QID1_RST 0x0
+
+#define LSP_CQ2QID1_QID_P4	0x0000007F
+#define LSP_CQ2QID1_RSVD3	0x00000080
+#define LSP_CQ2QID1_QID_P5	0x00007F00
+#define LSP_CQ2QID1_RSVD2	0x00008000
+#define LSP_CQ2QID1_QID_P6	0x007F0000
+#define LSP_CQ2QID1_RSVD1	0x00800000
+#define LSP_CQ2QID1_QID_P7	0x7F000000
+#define LSP_CQ2QID1_RSVD0	0x80000000
+#define LSP_CQ2QID1_QID_P4_LOC	0
+#define LSP_CQ2QID1_RSVD3_LOC	7
+#define LSP_CQ2QID1_QID_P5_LOC	8
+#define LSP_CQ2QID1_RSVD2_LOC	15
+#define LSP_CQ2QID1_QID_P6_LOC	16
+#define LSP_CQ2QID1_RSVD1_LOC	23
+#define LSP_CQ2QID1_QID_P7_LOC	24
+#define LSP_CQ2QID1_RSVD0_LOC	31
+
+#define LSP_CQ_DIR_DSBL(x) \
+	(0xa0180000 + (x) * 0x1000)
+#define LSP_CQ_DIR_DSBL_RST 0x1
+
+#define LSP_CQ_DIR_DSBL_DISABLED	0x00000001
+#define LSP_CQ_DIR_DSBL_RSVD0		0xFFFFFFFE
+#define LSP_CQ_DIR_DSBL_DISABLED_LOC	0
+#define LSP_CQ_DIR_DSBL_RSVD0_LOC	1
+
+#define LSP_CQ_DIR_TKN_CNT(x) \
+	(0xa0200000 + (x) * 0x1000)
+#define LSP_CQ_DIR_TKN_CNT_RST 0x0
+
+#define LSP_CQ_DIR_TKN_CNT_COUNT	0x00001FFF
+#define LSP_CQ_DIR_TKN_CNT_RSVD0	0xFFFFE000
+#define LSP_CQ_DIR_TKN_CNT_COUNT_LOC	0
+#define LSP_CQ_DIR_TKN_CNT_RSVD0_LOC	13
+
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+	(0xa0280000 + (x) * 0x1000)
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_TOKEN_DEPTH_SELECT	0x0000000F
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_DISABLE_WB_OPT	0x00000010
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_IGNORE_DEPTH	0x00000020
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RSVD0		0xFFFFFFC0
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_TOKEN_DEPTH_SELECT_LOC	0
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_DISABLE_WB_OPT_LOC		4
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_IGNORE_DEPTH_LOC		5
+#define LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RSVD0_LOC			6
+
+#define LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+	(0xa0300000 + (x) * 0x1000)
+#define LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+
+#define LSP_CQ_DIR_TOT_SCH_CNTL_COUNT	0xFFFFFFFF
+#define LSP_CQ_DIR_TOT_SCH_CNTL_COUNT_LOC	0
+
+#define LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+	(0xa0380000 + (x) * 0x1000)
+#define LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+
+#define LSP_CQ_DIR_TOT_SCH_CNTH_COUNT	0xFFFFFFFF
+#define LSP_CQ_DIR_TOT_SCH_CNTH_COUNT_LOC	0
+
+#define LSP_CQ_LDB_DSBL(x) \
+	(0xa0400000 + (x) * 0x1000)
+#define LSP_CQ_LDB_DSBL_RST 0x1
+
+#define LSP_CQ_LDB_DSBL_DISABLED	0x00000001
+#define LSP_CQ_LDB_DSBL_RSVD0		0xFFFFFFFE
+#define LSP_CQ_LDB_DSBL_DISABLED_LOC	0
+#define LSP_CQ_LDB_DSBL_RSVD0_LOC	1
+
+#define LSP_CQ_LDB_INFL_CNT(x) \
+	(0xa0480000 + (x) * 0x1000)
+#define LSP_CQ_LDB_INFL_CNT_RST 0x0
+
+#define LSP_CQ_LDB_INFL_CNT_COUNT	0x00000FFF
+#define LSP_CQ_LDB_INFL_CNT_RSVD0	0xFFFFF000
+#define LSP_CQ_LDB_INFL_CNT_COUNT_LOC	0
+#define LSP_CQ_LDB_INFL_CNT_RSVD0_LOC	12
+
+#define LSP_CQ_LDB_INFL_LIM(x) \
+	(0xa0500000 + (x) * 0x1000)
+#define LSP_CQ_LDB_INFL_LIM_RST 0x0
+
+#define LSP_CQ_LDB_INFL_LIM_LIMIT	0x00000FFF
+#define LSP_CQ_LDB_INFL_LIM_RSVD0	0xFFFFF000
+#define LSP_CQ_LDB_INFL_LIM_LIMIT_LOC	0
+#define LSP_CQ_LDB_INFL_LIM_RSVD0_LOC	12
+
+#define LSP_CQ_LDB_TKN_CNT(x) \
+	(0xa0580000 + (x) * 0x1000)
+#define LSP_CQ_LDB_TKN_CNT_RST 0x0
+
+#define LSP_CQ_LDB_TKN_CNT_TOKEN_COUNT	0x000007FF
+#define LSP_CQ_LDB_TKN_CNT_RSVD0	0xFFFFF800
+#define LSP_CQ_LDB_TKN_CNT_TOKEN_COUNT_LOC	0
+#define LSP_CQ_LDB_TKN_CNT_RSVD0_LOC		11
+
+#define LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+	(0xa0600000 + (x) * 0x1000)
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT	0x0000000F
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_IGNORE_DEPTH		0x00000010
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_RSVD0			0xFFFFFFE0
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT_LOC	0
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_IGNORE_DEPTH_LOC	4
+#define LSP_CQ_LDB_TKN_DEPTH_SEL_RSVD0_LOC		5
+
+#define LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+	(0xa0680000 + (x) * 0x1000)
+#define LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+
+#define LSP_CQ_LDB_TOT_SCH_CNTL_COUNT	0xFFFFFFFF
+#define LSP_CQ_LDB_TOT_SCH_CNTL_COUNT_LOC	0
+
+#define LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+	(0xa0700000 + (x) * 0x1000)
+#define LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+
+#define LSP_CQ_LDB_TOT_SCH_CNTH_COUNT	0xFFFFFFFF
+#define LSP_CQ_LDB_TOT_SCH_CNTH_COUNT_LOC	0
+
+#define LSP_QID_DIR_MAX_DEPTH(x) \
+	(0xa0780000 + (x) * 0x1000)
+#define LSP_QID_DIR_MAX_DEPTH_RST 0x0
+
+#define LSP_QID_DIR_MAX_DEPTH_DEPTH	0x00001FFF
+#define LSP_QID_DIR_MAX_DEPTH_RSVD0	0xFFFFE000
+#define LSP_QID_DIR_MAX_DEPTH_DEPTH_LOC	0
+#define LSP_QID_DIR_MAX_DEPTH_RSVD0_LOC	13
+
+#define LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+	(0xa0800000 + (x) * 0x1000)
+#define LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+
+#define LSP_QID_DIR_TOT_ENQ_CNTL_COUNT	0xFFFFFFFF
+#define LSP_QID_DIR_TOT_ENQ_CNTL_COUNT_LOC	0
+
+#define LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+	(0xa0880000 + (x) * 0x1000)
+#define LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+
+#define LSP_QID_DIR_TOT_ENQ_CNTH_COUNT	0xFFFFFFFF
+#define LSP_QID_DIR_TOT_ENQ_CNTH_COUNT_LOC	0
+
+#define LSP_QID_DIR_ENQUEUE_CNT(x) \
+	(0xa0900000 + (x) * 0x1000)
+#define LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+
+#define LSP_QID_DIR_ENQUEUE_CNT_COUNT	0x00001FFF
+#define LSP_QID_DIR_ENQUEUE_CNT_RSVD0	0xFFFFE000
+#define LSP_QID_DIR_ENQUEUE_CNT_COUNT_LOC	0
+#define LSP_QID_DIR_ENQUEUE_CNT_RSVD0_LOC	13
+
+#define LSP_QID_DIR_DEPTH_THRSH(x) \
+	(0xa0980000 + (x) * 0x1000)
+#define LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+
+#define LSP_QID_DIR_DEPTH_THRSH_THRESH	0x00001FFF
+#define LSP_QID_DIR_DEPTH_THRSH_RSVD0	0xFFFFE000
+#define LSP_QID_DIR_DEPTH_THRSH_THRESH_LOC	0
+#define LSP_QID_DIR_DEPTH_THRSH_RSVD0_LOC	13
+
+#define LSP_QID_AQED_ACTIVE_CNT(x) \
+	(0xa0a00000 + (x) * 0x1000)
+#define LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+
+#define LSP_QID_AQED_ACTIVE_CNT_COUNT	0x00000FFF
+#define LSP_QID_AQED_ACTIVE_CNT_RSVD0	0xFFFFF000
+#define LSP_QID_AQED_ACTIVE_CNT_COUNT_LOC	0
+#define LSP_QID_AQED_ACTIVE_CNT_RSVD0_LOC	12
+
+#define LSP_QID_AQED_ACTIVE_LIM(x) \
+	(0xa0a80000 + (x) * 0x1000)
+#define LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+
+#define LSP_QID_AQED_ACTIVE_LIM_LIMIT	0x00000FFF
+#define LSP_QID_AQED_ACTIVE_LIM_RSVD0	0xFFFFF000
+#define LSP_QID_AQED_ACTIVE_LIM_LIMIT_LOC	0
+#define LSP_QID_AQED_ACTIVE_LIM_RSVD0_LOC	12
+
+#define LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+	(0xa0b00000 + (x) * 0x1000)
+#define LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+
+#define LSP_QID_ATM_TOT_ENQ_CNTL_COUNT	0xFFFFFFFF
+#define LSP_QID_ATM_TOT_ENQ_CNTL_COUNT_LOC	0
+
+#define LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+	(0xa0b80000 + (x) * 0x1000)
+#define LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+
+#define LSP_QID_ATM_TOT_ENQ_CNTH_COUNT	0xFFFFFFFF
+#define LSP_QID_ATM_TOT_ENQ_CNTH_COUNT_LOC	0
+
+#define LSP_QID_ATQ_ENQUEUE_CNT(x) \
+	(0xa0c00000 + (x) * 0x1000)
+#define LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+
+#define LSP_QID_ATQ_ENQUEUE_CNT_COUNT	0x00003FFF
+#define LSP_QID_ATQ_ENQUEUE_CNT_RSVD0	0xFFFFC000
+#define LSP_QID_ATQ_ENQUEUE_CNT_COUNT_LOC	0
+#define LSP_QID_ATQ_ENQUEUE_CNT_RSVD0_LOC	14
+
+#define LSP_QID_LDB_ENQUEUE_CNT(x) \
+	(0xa0c80000 + (x) * 0x1000)
+#define LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+
+#define LSP_QID_LDB_ENQUEUE_CNT_COUNT	0x00003FFF
+#define LSP_QID_LDB_ENQUEUE_CNT_RSVD0	0xFFFFC000
+#define LSP_QID_LDB_ENQUEUE_CNT_COUNT_LOC	0
+#define LSP_QID_LDB_ENQUEUE_CNT_RSVD0_LOC	14
+
+#define LSP_QID_LDB_INFL_CNT(x) \
+	(0xa0d00000 + (x) * 0x1000)
+#define LSP_QID_LDB_INFL_CNT_RST 0x0
+
+#define LSP_QID_LDB_INFL_CNT_COUNT	0x00000FFF
+#define LSP_QID_LDB_INFL_CNT_RSVD0	0xFFFFF000
+#define LSP_QID_LDB_INFL_CNT_COUNT_LOC	0
+#define LSP_QID_LDB_INFL_CNT_RSVD0_LOC	12
+
+#define LSP_QID_LDB_INFL_LIM(x) \
+	(0xa0d80000 + (x) * 0x1000)
+#define LSP_QID_LDB_INFL_LIM_RST 0x0
+
+#define LSP_QID_LDB_INFL_LIM_LIMIT	0x00000FFF
+#define LSP_QID_LDB_INFL_LIM_RSVD0	0xFFFFF000
+#define LSP_QID_LDB_INFL_LIM_LIMIT_LOC	0
+#define LSP_QID_LDB_INFL_LIM_RSVD0_LOC	12
+
+#define LSP_QID2CQIDIX_00(x) \
+	(0xa0e00000 + (x) * 0x1000)
+#define LSP_QID2CQIDIX_00_RST 0x0
+#define LSP_QID2CQIDIX(x, y) \
+	(LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define LSP_QID2CQIDIX_NUM 16
+
+#define LSP_QID2CQIDIX_00_CQ_P0	0x000000FF
+#define LSP_QID2CQIDIX_00_CQ_P1	0x0000FF00
+#define LSP_QID2CQIDIX_00_CQ_P2	0x00FF0000
+#define LSP_QID2CQIDIX_00_CQ_P3	0xFF000000
+#define LSP_QID2CQIDIX_00_CQ_P0_LOC	0
+#define LSP_QID2CQIDIX_00_CQ_P1_LOC	8
+#define LSP_QID2CQIDIX_00_CQ_P2_LOC	16
+#define LSP_QID2CQIDIX_00_CQ_P3_LOC	24
+
+#define LSP_QID2CQIDIX2_00(x) \
+	(0xa1600000 + (x) * 0x1000)
+#define LSP_QID2CQIDIX2_00_RST 0x0
+#define LSP_QID2CQIDIX2(x, y) \
+	(LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define LSP_QID2CQIDIX2_NUM 16
+
+#define LSP_QID2CQIDIX2_00_CQ_P0	0x000000FF
+#define LSP_QID2CQIDIX2_00_CQ_P1	0x0000FF00
+#define LSP_QID2CQIDIX2_00_CQ_P2	0x00FF0000
+#define LSP_QID2CQIDIX2_00_CQ_P3	0xFF000000
+#define LSP_QID2CQIDIX2_00_CQ_P0_LOC	0
+#define LSP_QID2CQIDIX2_00_CQ_P1_LOC	8
+#define LSP_QID2CQIDIX2_00_CQ_P2_LOC	16
+#define LSP_QID2CQIDIX2_00_CQ_P3_LOC	24
+
+#define LSP_QID_LDB_REPLAY_CNT(x) \
+	(0xa1e00000 + (x) * 0x1000)
+#define LSP_QID_LDB_REPLAY_CNT_RST 0x0
+
+#define LSP_QID_LDB_REPLAY_CNT_COUNT	0x00003FFF
+#define LSP_QID_LDB_REPLAY_CNT_RSVD0	0xFFFFC000
+#define LSP_QID_LDB_REPLAY_CNT_COUNT_LOC	0
+#define LSP_QID_LDB_REPLAY_CNT_RSVD0_LOC	14
+
+#define LSP_QID_NALDB_MAX_DEPTH(x) \
+	(0xa1f00000 + (x) * 0x1000)
+#define LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+
+#define LSP_QID_NALDB_MAX_DEPTH_DEPTH	0x00003FFF
+#define LSP_QID_NALDB_MAX_DEPTH_RSVD0	0xFFFFC000
+#define LSP_QID_NALDB_MAX_DEPTH_DEPTH_LOC	0
+#define LSP_QID_NALDB_MAX_DEPTH_RSVD0_LOC	14
+
+#define LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+	(0xa1f80000 + (x) * 0x1000)
+#define LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+
+#define LSP_QID_NALDB_TOT_ENQ_CNTL_COUNT	0xFFFFFFFF
+#define LSP_QID_NALDB_TOT_ENQ_CNTL_COUNT_LOC	0
+
+#define LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+	(0xa2000000 + (x) * 0x1000)
+#define LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+
+#define LSP_QID_NALDB_TOT_ENQ_CNTH_COUNT	0xFFFFFFFF
+#define LSP_QID_NALDB_TOT_ENQ_CNTH_COUNT_LOC	0
+
+#define LSP_QID_ATM_DEPTH_THRSH(x) \
+	(0xa2080000 + (x) * 0x1000)
+#define LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+
+#define LSP_QID_ATM_DEPTH_THRSH_THRESH	0x00003FFF
+#define LSP_QID_ATM_DEPTH_THRSH_RSVD0	0xFFFFC000
+#define LSP_QID_ATM_DEPTH_THRSH_THRESH_LOC	0
+#define LSP_QID_ATM_DEPTH_THRSH_RSVD0_LOC	14
+
+#define LSP_QID_NALDB_DEPTH_THRSH(x) \
+	(0xa2100000 + (x) * 0x1000)
+#define LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+
+#define LSP_QID_NALDB_DEPTH_THRSH_THRESH	0x00003FFF
+#define LSP_QID_NALDB_DEPTH_THRSH_RSVD0		0xFFFFC000
+#define LSP_QID_NALDB_DEPTH_THRSH_THRESH_LOC	0
+#define LSP_QID_NALDB_DEPTH_THRSH_RSVD0_LOC	14
+
+#define LSP_QID_ATM_ACTIVE(x) \
+	(0xa2180000 + (x) * 0x1000)
+#define LSP_QID_ATM_ACTIVE_RST 0x0
+
+#define LSP_QID_ATM_ACTIVE_COUNT	0x00003FFF
+#define LSP_QID_ATM_ACTIVE_RSVD0	0xFFFFC000
+#define LSP_QID_ATM_ACTIVE_COUNT_LOC	0
+#define LSP_QID_ATM_ACTIVE_RSVD0_LOC	14
+
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI0_WEIGHT	0x000000FF
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI1_WEIGHT	0x0000FF00
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI2_WEIGHT	0x00FF0000
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI3_WEIGHT	0xFF000000
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI0_WEIGHT_LOC	0
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI1_WEIGHT_LOC	8
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI2_WEIGHT_LOC	16
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_PRI3_WEIGHT_LOC	24
+
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RSVZ0	0xFFFFFFFF
+#define LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RSVZ0_LOC	0
+
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI0_WEIGHT	0x000000FF
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI1_WEIGHT	0x0000FF00
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI2_WEIGHT	0x00FF0000
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI3_WEIGHT	0xFF000000
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI0_WEIGHT_LOC	0
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI1_WEIGHT_LOC	8
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI2_WEIGHT_LOC	16
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_0_PRI3_WEIGHT_LOC	24
+
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_1_RSVZ0	0xFFFFFFFF
+#define LSP_CFG_ARB_WEIGHT_LDB_QID_1_RSVZ0_LOC	0
+
+#define LSP_LDB_SCHED_CTRL 0xa400002c
+#define LSP_LDB_SCHED_CTRL_RST 0x0
+
+#define LSP_LDB_SCHED_CTRL_CQ			0x000000FF
+#define LSP_LDB_SCHED_CTRL_QIDIX		0x00000700
+#define LSP_LDB_SCHED_CTRL_VALUE		0x00000800
+#define LSP_LDB_SCHED_CTRL_NALB_HASWORK_V	0x00001000
+#define LSP_LDB_SCHED_CTRL_RLIST_HASWORK_V	0x00002000
+#define LSP_LDB_SCHED_CTRL_SLIST_HASWORK_V	0x00004000
+#define LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V	0x00008000
+#define LSP_LDB_SCHED_CTRL_AQED_NFULL_V		0x00010000
+#define LSP_LDB_SCHED_CTRL_RSVZ0		0xFFFE0000
+#define LSP_LDB_SCHED_CTRL_CQ_LOC		0
+#define LSP_LDB_SCHED_CTRL_QIDIX_LOC		8
+#define LSP_LDB_SCHED_CTRL_VALUE_LOC		11
+#define LSP_LDB_SCHED_CTRL_NALB_HASWORK_V_LOC	12
+#define LSP_LDB_SCHED_CTRL_RLIST_HASWORK_V_LOC	13
+#define LSP_LDB_SCHED_CTRL_SLIST_HASWORK_V_LOC	14
+#define LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V_LOC	15
+#define LSP_LDB_SCHED_CTRL_AQED_NFULL_V_LOC	16
+#define LSP_LDB_SCHED_CTRL_RSVZ0_LOC		17
+
+#define LSP_DIR_SCH_CNT_L 0xa4000034
+#define LSP_DIR_SCH_CNT_L_RST 0x0
+
+#define LSP_DIR_SCH_CNT_L_COUNT	0xFFFFFFFF
+#define LSP_DIR_SCH_CNT_L_COUNT_LOC	0
+
+#define LSP_DIR_SCH_CNT_H 0xa4000038
+#define LSP_DIR_SCH_CNT_H_RST 0x0
+
+#define LSP_DIR_SCH_CNT_H_COUNT	0xFFFFFFFF
+#define LSP_DIR_SCH_CNT_H_COUNT_LOC	0
+
+#define LSP_LDB_SCH_CNT_L 0xa400003c
+#define LSP_LDB_SCH_CNT_L_RST 0x0
+
+#define LSP_LDB_SCH_CNT_L_COUNT	0xFFFFFFFF
+#define LSP_LDB_SCH_CNT_L_COUNT_LOC	0
+
+#define LSP_LDB_SCH_CNT_H 0xa4000040
+#define LSP_LDB_SCH_CNT_H_RST 0x0
+
+#define LSP_LDB_SCH_CNT_H_COUNT	0xFFFFFFFF
+#define LSP_LDB_SCH_CNT_H_COUNT_LOC	0
+
+#define LSP_CFG_SHDW_CTRL 0xa4000070
+#define LSP_CFG_SHDW_CTRL_RST 0x0
+
+#define LSP_CFG_SHDW_CTRL_TRANSFER	0x00000001
+#define LSP_CFG_SHDW_CTRL_RSVD0		0xFFFFFFFE
+#define LSP_CFG_SHDW_CTRL_TRANSFER_LOC	0
+#define LSP_CFG_SHDW_CTRL_RSVD0_LOC	1
+
+#define LSP_CFG_SHDW_RANGE_COS(x) \
+	(0xa4000074 + (x) * 4)
+#define LSP_CFG_SHDW_RANGE_COS_RST 0x40
+
+#define LSP_CFG_SHDW_RANGE_COS_BW_RANGE		0x000001FF
+#define LSP_CFG_SHDW_RANGE_COS_RSVZ0		0x7FFFFE00
+#define LSP_CFG_SHDW_RANGE_COS_NO_EXTRA_CREDIT	0x80000000
+#define LSP_CFG_SHDW_RANGE_COS_BW_RANGE_LOC		0
+#define LSP_CFG_SHDW_RANGE_COS_RSVZ0_LOC		9
+#define LSP_CFG_SHDW_RANGE_COS_NO_EXTRA_CREDIT_LOC	31
+
+#define LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define LSP_CFG_CTRL_GENERAL_0_RST 0x0
+
+#define LSP_CFG_CTRL_GENERAL_0_DISAB_ATQ_EMPTY_ARB	0x00000001
+#define LSP_CFG_CTRL_GENERAL_0_INC_TOK_UNIT_IDLE	0x00000002
+#define LSP_CFG_CTRL_GENERAL_0_DISAB_RLIST_PRI		0x00000004
+#define LSP_CFG_CTRL_GENERAL_0_INC_CMP_UNIT_IDLE	0x00000008
+#define LSP_CFG_CTRL_GENERAL_0_RSVZ0			0x00000030
+#define LSP_CFG_CTRL_GENERAL_0_DIR_SINGLE_OP		0x00000040
+#define LSP_CFG_CTRL_GENERAL_0_DIR_HALF_BW		0x00000080
+#define LSP_CFG_CTRL_GENERAL_0_DIR_SINGLE_OUT		0x00000100
+#define LSP_CFG_CTRL_GENERAL_0_DIR_DISAB_MULTI		0x00000200
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_SINGLE_OP		0x00000400
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_HALF_BW		0x00000800
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_SINGLE_OUT		0x00001000
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_DISAB_MULTI		0x00002000
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_SINGLE_OP		0x00004000
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_HALF_BW		0x00008000
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_SINGLE_OUT	0x00010000
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_SINGLE_OP		0x00020000
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_HALF_BW		0x00040000
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_SINGLE_OUT		0x00080000
+#define LSP_CFG_CTRL_GENERAL_0_LDB_SINGLE_OP		0x00100000
+#define LSP_CFG_CTRL_GENERAL_0_LDB_HALF_BW		0x00200000
+#define LSP_CFG_CTRL_GENERAL_0_LDB_DISAB_MULTI		0x00400000
+#define LSP_CFG_CTRL_GENERAL_0_ATM_SINGLE_SCH		0x00800000
+#define LSP_CFG_CTRL_GENERAL_0_ATM_SINGLE_CMP		0x01000000
+#define LSP_CFG_CTRL_GENERAL_0_LDB_CE_TOG_ARB		0x02000000
+#define LSP_CFG_CTRL_GENERAL_0_RSVZ1			0x04000000
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_VALID_SEL		0x18000000
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_VALUE_SEL		0x20000000
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_COMPARE_SEL	0xC0000000
+#define LSP_CFG_CTRL_GENERAL_0_DISAB_ATQ_EMPTY_ARB_LOC	0
+#define LSP_CFG_CTRL_GENERAL_0_INC_TOK_UNIT_IDLE_LOC	1
+#define LSP_CFG_CTRL_GENERAL_0_DISAB_RLIST_PRI_LOC	2
+#define LSP_CFG_CTRL_GENERAL_0_INC_CMP_UNIT_IDLE_LOC	3
+#define LSP_CFG_CTRL_GENERAL_0_RSVZ0_LOC		4
+#define LSP_CFG_CTRL_GENERAL_0_DIR_SINGLE_OP_LOC	6
+#define LSP_CFG_CTRL_GENERAL_0_DIR_HALF_BW_LOC		7
+#define LSP_CFG_CTRL_GENERAL_0_DIR_SINGLE_OUT_LOC	8
+#define LSP_CFG_CTRL_GENERAL_0_DIR_DISAB_MULTI_LOC	9
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_SINGLE_OP_LOC	10
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_HALF_BW_LOC		11
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_SINGLE_OUT_LOC	12
+#define LSP_CFG_CTRL_GENERAL_0_ATQ_DISAB_MULTI_LOC	13
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_SINGLE_OP_LOC	14
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_HALF_BW_LOC	15
+#define LSP_CFG_CTRL_GENERAL_0_DIRRPL_SINGLE_OUT_LOC	16
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_SINGLE_OP_LOC	17
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_HALF_BW_LOC	18
+#define LSP_CFG_CTRL_GENERAL_0_LBRPL_SINGLE_OUT_LOC	19
+#define LSP_CFG_CTRL_GENERAL_0_LDB_SINGLE_OP_LOC	20
+#define LSP_CFG_CTRL_GENERAL_0_LDB_HALF_BW_LOC		21
+#define LSP_CFG_CTRL_GENERAL_0_LDB_DISAB_MULTI_LOC	22
+#define LSP_CFG_CTRL_GENERAL_0_ATM_SINGLE_SCH_LOC	23
+#define LSP_CFG_CTRL_GENERAL_0_ATM_SINGLE_CMP_LOC	24
+#define LSP_CFG_CTRL_GENERAL_0_LDB_CE_TOG_ARB_LOC	25
+#define LSP_CFG_CTRL_GENERAL_0_RSVZ1_LOC		26
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_VALID_SEL_LOC	27
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_VALUE_SEL_LOC	29
+#define LSP_CFG_CTRL_GENERAL_0_SMON0_COMPARE_SEL_LOC	30
+
+#define LSP_SMON_COMPARE0 0xac000048
+#define LSP_SMON_COMPARE0_RST 0x0
+
+#define LSP_SMON_COMPARE0_COMPARE0	0xFFFFFFFF
+#define LSP_SMON_COMPARE0_COMPARE0_LOC	0
+
+#define LSP_SMON_COMPARE1 0xac00004c
+#define LSP_SMON_COMPARE1_RST 0x0
+
+#define LSP_SMON_COMPARE1_COMPARE1	0xFFFFFFFF
+#define LSP_SMON_COMPARE1_COMPARE1_LOC	0
+
+#define LSP_SMON_CFG0 0xac000050
+#define LSP_SMON_CFG0_RST 0x40000000
+
+#define LSP_SMON_CFG0_SMON_ENABLE		0x00000001
+#define LSP_SMON_CFG0_SMON_0TRIGGER_ENABLE	0x00000002
+#define LSP_SMON_CFG0_RSVZ0			0x0000000C
+#define LSP_SMON_CFG0_SMON0_FUNCTION		0x00000070
+#define LSP_SMON_CFG0_SMON0_FUNCTION_COMPARE	0x00000080
+#define LSP_SMON_CFG0_SMON1_FUNCTION		0x00000700
+#define LSP_SMON_CFG0_SMON1_FUNCTION_COMPARE	0x00000800
+#define LSP_SMON_CFG0_SMON_MODE			0x0000F000
+#define LSP_SMON_CFG0_STOPCOUNTEROVFL		0x00010000
+#define LSP_SMON_CFG0_INTCOUNTEROVFL		0x00020000
+#define LSP_SMON_CFG0_STATCOUNTER0OVFL		0x00040000
+#define LSP_SMON_CFG0_STATCOUNTER1OVFL		0x00080000
+#define LSP_SMON_CFG0_STOPTIMEROVFL		0x00100000
+#define LSP_SMON_CFG0_INTTIMEROVFL		0x00200000
+#define LSP_SMON_CFG0_STATTIMEROVFL		0x00400000
+#define LSP_SMON_CFG0_RSVZ1			0x00800000
+#define LSP_SMON_CFG0_TIMER_PRESCALE		0x1F000000
+#define LSP_SMON_CFG0_RSVZ2			0x20000000
+#define LSP_SMON_CFG0_VERSION			0xC0000000
+#define LSP_SMON_CFG0_SMON_ENABLE_LOC			0
+#define LSP_SMON_CFG0_SMON_0TRIGGER_ENABLE_LOC		1
+#define LSP_SMON_CFG0_RSVZ0_LOC				2
+#define LSP_SMON_CFG0_SMON0_FUNCTION_LOC		4
+#define LSP_SMON_CFG0_SMON0_FUNCTION_COMPARE_LOC	7
+#define LSP_SMON_CFG0_SMON1_FUNCTION_LOC		8
+#define LSP_SMON_CFG0_SMON1_FUNCTION_COMPARE_LOC	11
+#define LSP_SMON_CFG0_SMON_MODE_LOC			12
+#define LSP_SMON_CFG0_STOPCOUNTEROVFL_LOC		16
+#define LSP_SMON_CFG0_INTCOUNTEROVFL_LOC		17
+#define LSP_SMON_CFG0_STATCOUNTER0OVFL_LOC		18
+#define LSP_SMON_CFG0_STATCOUNTER1OVFL_LOC		19
+#define LSP_SMON_CFG0_STOPTIMEROVFL_LOC			20
+#define LSP_SMON_CFG0_INTTIMEROVFL_LOC			21
+#define LSP_SMON_CFG0_STATTIMEROVFL_LOC			22
+#define LSP_SMON_CFG0_RSVZ1_LOC				23
+#define LSP_SMON_CFG0_TIMER_PRESCALE_LOC		24
+#define LSP_SMON_CFG0_RSVZ2_LOC				29
+#define LSP_SMON_CFG0_VERSION_LOC			30
+
+#define LSP_SMON_CFG1 0xac000054
+#define LSP_SMON_CFG1_RST 0x0
+
+#define LSP_SMON_CFG1_MODE0	0x000000FF
+#define LSP_SMON_CFG1_MODE1	0x0000FF00
+#define LSP_SMON_CFG1_RSVZ0	0xFFFF0000
+#define LSP_SMON_CFG1_MODE0_LOC	0
+#define LSP_SMON_CFG1_MODE1_LOC	8
+#define LSP_SMON_CFG1_RSVZ0_LOC	16
+
+#define LSP_SMON_ACTIVITYCNTR0 0xac000058
+#define LSP_SMON_ACTIVITYCNTR0_RST 0x0
+
+#define LSP_SMON_ACTIVITYCNTR0_COUNTER0	0xFFFFFFFF
+#define LSP_SMON_ACTIVITYCNTR0_COUNTER0_LOC	0
+
+#define LSP_SMON_ACTIVITYCNTR1 0xac00005c
+#define LSP_SMON_ACTIVITYCNTR1_RST 0x0
+
+#define LSP_SMON_ACTIVITYCNTR1_COUNTER1	0xFFFFFFFF
+#define LSP_SMON_ACTIVITYCNTR1_COUNTER1_LOC	0
+
+#define LSP_SMON_MAX_TMR 0xac000060
+#define LSP_SMON_MAX_TMR_RST 0x0
+
+#define LSP_SMON_MAX_TMR_MAXVALUE	0xFFFFFFFF
+#define LSP_SMON_MAX_TMR_MAXVALUE_LOC	0
+
+#define LSP_SMON_TMR 0xac000064
+#define LSP_SMON_TMR_RST 0x0
+
+#define LSP_SMON_TMR_TIMER	0xFFFFFFFF
+#define LSP_SMON_TMR_TIMER_LOC	0
+
+#define CM_DIAG_RESET_STS 0xb4000000
+#define CM_DIAG_RESET_STS_RST 0x80000bff
+
+#define CM_DIAG_RESET_STS_CHP_PF_RESET_DONE	0x00000001
+#define CM_DIAG_RESET_STS_ROP_PF_RESET_DONE	0x00000002
+#define CM_DIAG_RESET_STS_LSP_PF_RESET_DONE	0x00000004
+#define CM_DIAG_RESET_STS_NALB_PF_RESET_DONE	0x00000008
+#define CM_DIAG_RESET_STS_AP_PF_RESET_DONE	0x00000010
+#define CM_DIAG_RESET_STS_DP_PF_RESET_DONE	0x00000020
+#define CM_DIAG_RESET_STS_QED_PF_RESET_DONE	0x00000040
+#define CM_DIAG_RESET_STS_DQED_PF_RESET_DONE	0x00000080
+#define CM_DIAG_RESET_STS_AQED_PF_RESET_DONE	0x00000100
+#define CM_DIAG_RESET_STS_SYS_PF_RESET_DONE	0x00000200
+#define CM_DIAG_RESET_STS_PF_RESET_ACTIVE	0x00000400
+#define CM_DIAG_RESET_STS_FLRSM_STATE		0x0003F800
+#define CM_DIAG_RESET_STS_RSVD0			0x7FFC0000
+#define CM_DIAG_RESET_STS_DLB_PROC_RESET_DONE	0x80000000
+#define CM_DIAG_RESET_STS_CHP_PF_RESET_DONE_LOC		0
+#define CM_DIAG_RESET_STS_ROP_PF_RESET_DONE_LOC		1
+#define CM_DIAG_RESET_STS_LSP_PF_RESET_DONE_LOC		2
+#define CM_DIAG_RESET_STS_NALB_PF_RESET_DONE_LOC	3
+#define CM_DIAG_RESET_STS_AP_PF_RESET_DONE_LOC		4
+#define CM_DIAG_RESET_STS_DP_PF_RESET_DONE_LOC		5
+#define CM_DIAG_RESET_STS_QED_PF_RESET_DONE_LOC		6
+#define CM_DIAG_RESET_STS_DQED_PF_RESET_DONE_LOC	7
+#define CM_DIAG_RESET_STS_AQED_PF_RESET_DONE_LOC	8
+#define CM_DIAG_RESET_STS_SYS_PF_RESET_DONE_LOC		9
+#define CM_DIAG_RESET_STS_PF_RESET_ACTIVE_LOC		10
+#define CM_DIAG_RESET_STS_FLRSM_STATE_LOC		11
+#define CM_DIAG_RESET_STS_RSVD0_LOC			18
+#define CM_DIAG_RESET_STS_DLB_PROC_RESET_DONE_LOC	31
 
 #define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
 #define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
@@ -134,4 +3584,57 @@
 #define CM_CFG_PM_PMCSR_DISABLE_DISABLE_LOC	0
 #define CM_CFG_PM_PMCSR_DISABLE_RSVZ0_LOC	1
 
+#define VF_VF2PF_MAILBOX_BYTES 256
+#define VF_VF2PF_MAILBOX(x) \
+	(0x1000 + (x) * 0x4)
+#define VF_VF2PF_MAILBOX_RST 0x0
+
+#define VF_VF2PF_MAILBOX_MSG	0xFFFFFFFF
+#define VF_VF2PF_MAILBOX_MSG_LOC	0
+
+#define VF_VF2PF_MAILBOX_ISR 0x1f00
+#define VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define VF_SIOV_MBOX_ISR_TRIGGER 0x8000
+
+#define VF_VF2PF_MAILBOX_ISR_ISR	0x00000001
+#define VF_VF2PF_MAILBOX_ISR_RSVD0	0xFFFFFFFE
+#define VF_VF2PF_MAILBOX_ISR_ISR_LOC	0
+#define VF_VF2PF_MAILBOX_ISR_RSVD0_LOC	1
+
+#define VF_PF2VF_MAILBOX_BYTES 64
+#define VF_PF2VF_MAILBOX(x) \
+	(0x2000 + (x) * 0x4)
+#define VF_PF2VF_MAILBOX_RST 0x0
+
+#define VF_PF2VF_MAILBOX_MSG	0xFFFFFFFF
+#define VF_PF2VF_MAILBOX_MSG_LOC	0
+
+#define VF_PF2VF_MAILBOX_ISR 0x2f00
+#define VF_PF2VF_MAILBOX_ISR_RST 0x0
+
+#define VF_PF2VF_MAILBOX_ISR_PF_ISR	0x00000001
+#define VF_PF2VF_MAILBOX_ISR_RSVD0	0xFFFFFFFE
+#define VF_PF2VF_MAILBOX_ISR_PF_ISR_LOC	0
+#define VF_PF2VF_MAILBOX_ISR_RSVD0_LOC	1
+
+#define VF_VF_MSI_ISR_PEND 0x2f10
+#define VF_VF_MSI_ISR_PEND_RST 0x0
+
+#define VF_VF_MSI_ISR_PEND_ISR_PEND	0xFFFFFFFF
+#define VF_VF_MSI_ISR_PEND_ISR_PEND_LOC	0
+
+#define VF_VF_RESET_IN_PROGRESS 0x3000
+#define VF_VF_RESET_IN_PROGRESS_RST 0x1
+
+#define VF_VF_RESET_IN_PROGRESS_RESET_IN_PROGRESS	0x00000001
+#define VF_VF_RESET_IN_PROGRESS_RSVD0			0xFFFFFFFE
+#define VF_VF_RESET_IN_PROGRESS_RESET_IN_PROGRESS_LOC	0
+#define VF_VF_RESET_IN_PROGRESS_RSVD0_LOC		1
+
+#define VF_VF_MSI_ISR 0x4000
+#define VF_VF_MSI_ISR_RST 0x0
+
+#define VF_VF_MSI_ISR_VF_MSI_ISR	0xFFFFFFFF
+#define VF_VF_MSI_ISR_VF_MSI_ISR_LOC	0
+
 #endif /* __DLB_REGS_H */
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 6d73c2479819..26b7d9ea94e3 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -973,6 +973,390 @@ static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 	return 0;
 }
 
+static void __dlb_domain_reset_ldb_port_registers(struct dlb_hw *hw,
+						  struct dlb_ldb_port *port)
+{
+	DLB_CSR_WR(hw,
+		   SYS_LDB_PP2VAS(port->id.phys_id),
+		   SYS_LDB_PP2VAS_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ2VAS(port->id.phys_id),
+		   CHP_LDB_CQ2VAS_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_PP2VDEV(port->id.phys_id),
+		   SYS_LDB_PP2VDEV_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_PP_V(port->id.phys_id),
+		   SYS_LDB_PP_V_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_DSBL(port->id.phys_id),
+		   LSP_CQ_LDB_DSBL_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_DEPTH(port->id.phys_id),
+		   CHP_LDB_CQ_DEPTH_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+		   LSP_CQ_LDB_INFL_LIM_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_HIST_LIST_LIM(port->id.phys_id),
+		   CHP_HIST_LIST_LIM_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_HIST_LIST_BASE(port->id.phys_id),
+		   CHP_HIST_LIST_BASE_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+		   CHP_HIST_LIST_POP_PTR_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+		   CHP_HIST_LIST_PUSH_PTR_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		   CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+		   CHP_LDB_CQ_TMR_THRSH_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+		   CHP_LDB_CQ_INT_ENB_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ_ISR(port->id.phys_id),
+		   SYS_LDB_CQ_ISR_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+		   LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		   CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_WPTR(port->id.phys_id),
+		   CHP_LDB_CQ_WPTR_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+		   LSP_CQ_LDB_TKN_CNT_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+		   SYS_LDB_CQ_ADDR_L_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+		   SYS_LDB_CQ_ADDR_U_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ_AT(port->id.phys_id),
+		   SYS_LDB_CQ_AT_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ_PASID(port->id.phys_id),
+		   SYS_LDB_CQ_PASID_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+		   SYS_LDB_CQ2VF_PF_RO_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+		   LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+		   LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ2QID0(port->id.phys_id),
+		   LSP_CQ2QID0_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ2QID1(port->id.phys_id),
+		   LSP_CQ2QID1_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ2PRIOV(port->id.phys_id),
+		   LSP_CQ2PRIOV_RST);
+}
+
+static void dlb_domain_reset_ldb_port_registers(struct dlb_hw *hw,
+						struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list)
+			__dlb_domain_reset_ldb_port_registers(hw, port);
+	}
+}
+
+static void
+__dlb_domain_reset_dir_port_registers(struct dlb_hw *hw,
+				      struct dlb_dir_pq_pair *port)
+{
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ2VAS(port->id.phys_id),
+		   CHP_DIR_CQ2VAS_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_DIR_DSBL(port->id.phys_id),
+		   LSP_CQ_DIR_DSBL_RST);
+
+	DLB_CSR_WR(hw, SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_DEPTH(port->id.phys_id),
+		   CHP_DIR_CQ_DEPTH_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+		   CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+		   CHP_DIR_CQ_TMR_THRSH_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+		   CHP_DIR_CQ_INT_ENB_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_ISR(port->id.phys_id),
+		   SYS_DIR_CQ_ISR_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+		   LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+		   CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_WPTR(port->id.phys_id),
+		   CHP_DIR_CQ_WPTR_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+		   LSP_CQ_DIR_TKN_CNT_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+		   SYS_DIR_CQ_ADDR_L_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+		   SYS_DIR_CQ_ADDR_U_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_AT(port->id.phys_id),
+		   SYS_DIR_CQ_AT_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_PASID(port->id.phys_id),
+		   SYS_DIR_CQ_PASID_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ_FMT(port->id.phys_id),
+		   SYS_DIR_CQ_FMT_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+		   SYS_DIR_CQ2VF_PF_RO_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+		   LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+	DLB_CSR_WR(hw,
+		   LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+		   LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_PP2VAS(port->id.phys_id),
+		   SYS_DIR_PP2VAS_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ2VAS(port->id.phys_id),
+		   CHP_DIR_CQ2VAS_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_PP2VDEV(port->id.phys_id),
+		   SYS_DIR_PP2VDEV_RST);
+
+	DLB_CSR_WR(hw,
+		   SYS_DIR_PP_V(port->id.phys_id),
+		   SYS_DIR_PP_V_RST);
+}
+
+static void dlb_domain_reset_dir_port_registers(struct dlb_hw *hw,
+						struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *port;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list)
+		__dlb_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb_domain_reset_ldb_queue_registers(struct dlb_hw *hw,
+						 struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_queue *queue;
+
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		unsigned int queue_id = queue->id.phys_id;
+		int i;
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+			   LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+			   LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+			   LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+			   LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_NALDB_MAX_DEPTH(queue_id),
+			   LSP_QID_NALDB_MAX_DEPTH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_LDB_INFL_LIM(queue_id),
+			   LSP_QID_LDB_INFL_LIM_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_AQED_ACTIVE_LIM(queue_id),
+			   LSP_QID_AQED_ACTIVE_LIM_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_ATM_DEPTH_THRSH(queue_id),
+			   LSP_QID_ATM_DEPTH_THRSH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+			   LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+		DLB_CSR_WR(hw,
+			   SYS_LDB_QID_ITS(queue_id),
+			   SYS_LDB_QID_ITS_RST);
+
+		DLB_CSR_WR(hw,
+			   CHP_ORD_QID_SN(queue_id),
+			   CHP_ORD_QID_SN_RST);
+
+		DLB_CSR_WR(hw,
+			   CHP_ORD_QID_SN_MAP(queue_id),
+			   CHP_ORD_QID_SN_MAP_RST);
+
+		DLB_CSR_WR(hw,
+			   SYS_LDB_QID_V(queue_id),
+			   SYS_LDB_QID_V_RST);
+
+		DLB_CSR_WR(hw,
+			   SYS_LDB_QID_CFG_V(queue_id),
+			   SYS_LDB_QID_CFG_V_RST);
+
+		if (queue->sn_cfg_valid) {
+			u32 offs[2];
+
+			offs[0] = RO_GRP_0_SLT_SHFT(queue->sn_slot);
+			offs[1] = RO_GRP_1_SLT_SHFT(queue->sn_slot);
+
+			DLB_CSR_WR(hw,
+				   offs[queue->sn_group],
+				   RO_GRP_0_SLT_SHFT_RST);
+		}
+
+		for (i = 0; i < LSP_QID2CQIDIX_NUM; i++) {
+			DLB_CSR_WR(hw,
+				   LSP_QID2CQIDIX(queue_id, i),
+				   LSP_QID2CQIDIX_00_RST);
+
+			DLB_CSR_WR(hw,
+				   LSP_QID2CQIDIX2(queue_id, i),
+				   LSP_QID2CQIDIX2_00_RST);
+
+			DLB_CSR_WR(hw,
+				   ATM_QID2CQIDIX(queue_id, i),
+				   ATM_QID2CQIDIX_00_RST);
+		}
+	}
+}
+
+static void dlb_domain_reset_dir_queue_registers(struct dlb_hw *hw,
+						 struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *queue;
+
+	list_for_each_entry(queue, &domain->used_dir_pq_pairs, domain_list) {
+		DLB_CSR_WR(hw,
+			   LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+			   LSP_QID_DIR_MAX_DEPTH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+			   LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+			   LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+		DLB_CSR_WR(hw,
+			   LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+			   LSP_QID_DIR_DEPTH_THRSH_RST);
+
+		DLB_CSR_WR(hw,
+			   SYS_DIR_QID_ITS(queue->id.phys_id),
+			   SYS_DIR_QID_ITS_RST);
+
+		DLB_CSR_WR(hw,
+			   SYS_DIR_QID_V(queue->id.phys_id),
+			   SYS_DIR_QID_V_RST);
+	}
+}
+
+static void dlb_domain_reset_registers(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain)
+{
+	dlb_domain_reset_ldb_port_registers(hw, domain);
+
+	dlb_domain_reset_dir_port_registers(hw, domain);
+
+	dlb_domain_reset_ldb_queue_registers(hw, domain);
+
+	dlb_domain_reset_dir_queue_registers(hw, domain);
+
+	DLB_CSR_WR(hw,
+		   CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+		   CHP_CFG_LDB_VAS_CRD_RST);
+
+	DLB_CSR_WR(hw,
+		   CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+		   CHP_CFG_DIR_VAS_CRD_RST);
+}
+
 static void dlb_log_reset_domain(struct dlb_hw *hw, u32 domain_id,
 				 bool vdev_req, unsigned int vdev_id)
 {
@@ -1019,6 +1403,9 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	if (!domain || !domain->configured)
 		return -EINVAL;
 
+	/* Reset the QID and port state. */
+	dlb_domain_reset_registers(hw, domain);
+
 	return dlb_domain_reset_software_state(hw, domain);
 }
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 08/20] dlb: add runtime power-management support
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (6 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 07/20] dlb: add low-level register reset operations Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 09/20] dlb: add queue create, reset, get-depth ioctls Mike Ximing Chen
                   ` (12 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Implemented a power-management policy of putting the device in D0 when in
use (when there are any open device files or memory mappings, or there are
any virtual devices), and D3Hot otherwise.

Add resume/suspend callbacks; when the device resumes, reset the hardware
to a known good state.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_main.c   | 101 +++++++++++++++++++++++++++++++++-
 drivers/misc/dlb/dlb_pf_ops.c |   8 +++
 2 files changed, 108 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 70030d779033..5484d9aee02c 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -8,6 +8,7 @@
 #include <linux/init.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pm_runtime.h>
 #include <linux/uaccess.h>
 
 #include "dlb_main.h"
@@ -73,12 +74,35 @@ static int dlb_open(struct inode *i, struct file *f)
 	f->private_data = dlb;
 	dlb->f = f;
 
+	/*
+	 * Increment the device's usage count and immediately wake it
+	 * if it was suspended.
+	 */
+	pm_runtime_get_sync(&dlb->pdev->dev);
+
+	return 0;
+}
+
+static int dlb_close(struct inode *i, struct file *f)
+{
+	struct dlb *dlb = f->private_data;
+
+	mutex_lock(&dlb->resource_mutex);
+	/*
+	 * Decrement the device's usage count and suspend it when
+	 * the application stops using it.
+	 */
+	pm_runtime_put_sync_suspend(&dlb->pdev->dev);
+
+	mutex_unlock(&dlb->resource_mutex);
+
 	return 0;
 }
 
 static const struct file_operations dlb_fops = {
 	.owner   = THIS_MODULE,
 	.open    = dlb_open,
+	.release = dlb_close,
 	.unlocked_ioctl = dlb_ioctl,
 };
 
@@ -97,6 +121,12 @@ int dlb_init_domain(struct dlb *dlb, u32 domain_id)
 
 	dlb->sched_domains[domain_id] = domain;
 
+	/*
+	 * The matching put is in dlb_free_domain, executed when the domain's
+	 * refcnt reaches zero.
+	 */
+	pm_runtime_get_sync(&dlb->pdev->dev);
+
 	return 0;
 }
 
@@ -121,7 +151,21 @@ static int __dlb_free_domain(struct dlb_domain *domain)
 
 void dlb_free_domain(struct kref *kref)
 {
-	__dlb_free_domain(container_of(kref, struct dlb_domain, refcnt));
+	struct dlb_domain *domain;
+	struct dlb *dlb;
+
+	domain = container_of(kref, struct dlb_domain, refcnt);
+
+	dlb = domain->dlb;
+
+	__dlb_free_domain(domain);
+
+	/*
+	 * Decrement the device's usage count and suspend it when
+	 * the last application stops using it. The matching get is in
+	 * dlb_init_domain.
+	 */
+	pm_runtime_put_sync_suspend(&dlb->pdev->dev);
 }
 
 static int dlb_domain_close(struct inode *i, struct file *f)
@@ -230,6 +274,14 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret)
 		goto init_driver_state_fail;
 
+	/*
+	 * Undo the 'get' operation by the PCI layer during probe and
+	 * (if PF) immediately suspend the device. Since the device is only
+	 * enabled when an application requests it, an autosuspend delay is
+	 * likely not beneficial.
+	 */
+	pm_runtime_put_sync_suspend(&pdev->dev);
+
 	return 0;
 
 init_driver_state_fail:
@@ -253,6 +305,9 @@ static void dlb_remove(struct pci_dev *pdev)
 {
 	struct dlb *dlb = pci_get_drvdata(pdev);
 
+	/* Undo the PM operation in dlb_probe(). */
+	pm_runtime_get_noresume(&pdev->dev);
+
 	dlb_resource_free(&dlb->hw);
 
 	device_destroy(dlb_class, dlb->dev_number);
@@ -264,17 +319,61 @@ static void dlb_remove(struct pci_dev *pdev)
 	spin_unlock(&dlb_ids_lock);
 }
 
+#ifdef CONFIG_PM
+static void dlb_reset_hardware_state(struct dlb *dlb)
+{
+	dlb_reset_device(dlb->pdev);
+}
+
+static int dlb_runtime_suspend(struct device *dev)
+{
+	/* Return and let the PCI subsystem put the device in D3hot. */
+
+	return 0;
+}
+
+static int dlb_runtime_resume(struct device *dev)
+{
+	struct pci_dev *pdev = container_of(dev, struct pci_dev, dev);
+	struct dlb *dlb = pci_get_drvdata(pdev);
+	int ret;
+
+	/*
+	 * The PCI subsystem put the device in D0, but the device may not have
+	 * completed powering up. Wait until the device is ready before
+	 * proceeding.
+	 */
+	ret = dlb->ops->wait_for_device_ready(dlb, pdev);
+	if (ret)
+		return ret;
+
+	/* Now reinitialize the device state. */
+	dlb_reset_hardware_state(dlb);
+
+	return 0;
+}
+#endif
+
 static struct pci_device_id dlb_id_table[] = {
 	{ PCI_DEVICE_DATA(INTEL, DLB_PF, DLB_PF) },
 	{ 0 }
 };
 MODULE_DEVICE_TABLE(pci, dlb_id_table);
 
+#ifdef CONFIG_PM
+static const struct dev_pm_ops dlb_pm_ops = {
+	SET_RUNTIME_PM_OPS(dlb_runtime_suspend, dlb_runtime_resume, NULL)
+};
+#endif
+
 static struct pci_driver dlb_pci_driver = {
 	.name		 = "dlb",
 	.id_table	 = dlb_id_table,
 	.probe		 = dlb_probe,
 	.remove		 = dlb_remove,
+#ifdef CONFIG_PM
+	.driver.pm	 = &dlb_pm_ops,
+#endif
 };
 
 static int __init dlb_init_module(void)
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 5dea0037d14b..494a482368f6 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -2,6 +2,7 @@
 /* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
 
 #include <linux/delay.h>
+#include <linux/pm_runtime.h>
 
 #include "dlb_main.h"
 #include "dlb_regs.h"
@@ -50,6 +51,13 @@ static int dlb_pf_init_driver_state(struct dlb *dlb)
 {
 	mutex_init(&dlb->resource_mutex);
 
+	/*
+	 * Allow PF runtime power-management (forbidden by default by the PCI
+	 * layer during scan). The driver puts the device into D3hot while
+	 * there are no scheduling domains to service.
+	 */
+	pm_runtime_allow(&dlb->pdev->dev);
+
 	return 0;
 }
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 09/20] dlb: add queue create, reset, get-depth ioctls
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (7 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 08/20] dlb: add runtime power-management support Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 10/20] dlb: add register operations for queue management Mike Ximing Chen
                   ` (11 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add ioctl commands to create DLB queues and query their depth, and the
corresponding scheduling domain reset code to drain the queues when they
are no longer in use.

When a CPU enqueues a queue entry (QE) to DLB, the QE entry is sent to
a DLB queue. These queues hold queue entries (QEs) that have not yet
been scheduled to a destination port. The queue's depth is the number of
QEs residing in a queue.

Each queue supports multiple priority levels, and while a directed queue
has a 1:1 mapping with a directed port, load-balanced queues can be
configured with a set of load-balanced ports that software desires the
queue's QEs to be scheduled to.

For ease of review, this commit is limited to higher-level code including
the ioctl commands, request verification, and debug log messages. All
register access/configuration code will be included in a subsequent commit.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_hw_types.h |  27 ++
 drivers/misc/dlb/dlb_ioctl.c    |  63 ++++
 drivers/misc/dlb/dlb_main.c     |   2 +
 drivers/misc/dlb/dlb_main.h     |  17 +
 drivers/misc/dlb/dlb_pf_ops.c   |  36 ++
 drivers/misc/dlb/dlb_resource.c | 589 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |  20 ++
 include/uapi/linux/dlb.h        | 151 ++++++++
 8 files changed, 905 insertions(+)

diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index c486ea344292..d382c414e2b0 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -150,6 +150,33 @@ struct dlb_sn_group {
 	u32 id;
 };
 
+static inline bool dlb_sn_group_full(struct dlb_sn_group *group)
+{
+	const u32 mask[] = {
+		0x0000ffff,  /* 64 SNs per queue */
+		0x000000ff,  /* 128 SNs per queue */
+		0x0000000f,  /* 256 SNs per queue */
+		0x00000003,  /* 512 SNs per queue */
+		0x00000001}; /* 1024 SNs per queue */
+
+	return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb_sn_group_alloc_slot(struct dlb_sn_group *group)
+{
+	const u32 bound[] = {16, 8, 4, 2, 1};
+	u32 i;
+
+	for (i = 0; i < bound[group->mode]; i++) {
+		if (!(group->slot_use_bitmap & BIT(i))) {
+			group->slot_use_bitmap |= BIT(i);
+			return i;
+		}
+	}
+
+	return -1;
+}
+
 static inline void
 dlb_sn_group_free_slot(struct dlb_sn_group *group, int slot)
 {
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 75892966f061..0fc20b32f0cf 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -8,6 +8,69 @@
 
 #include "dlb_main.h"
 
+/*
+ * The DLB domain ioctl callback template minimizes replication of boilerplate
+ * code to copy arguments, acquire and release the resource lock, and execute
+ * the command.  The arguments and response structure name should have the
+ * format dlb_<lower_name>_args.
+ */
+#define DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(lower_name)			   \
+static int dlb_domain_ioctl_##lower_name(struct dlb *dlb,		   \
+					 struct dlb_domain *domain,	   \
+					 unsigned long user_arg)	   \
+{									   \
+	struct dlb_##lower_name##_args __user *uarg;			   \
+	struct dlb_cmd_response response = {0};				   \
+	struct dlb_##lower_name##_args arg;				   \
+	int ret;							   \
+									   \
+	uarg = (void __user *)user_arg;					   \
+	if (copy_from_user(&arg, uarg, sizeof(arg)))			   \
+		return -EFAULT;						   \
+									   \
+	mutex_lock(&dlb->resource_mutex);				   \
+									   \
+	ret = dlb->ops->lower_name(&dlb->hw,				   \
+				    domain->id,				   \
+				    &arg,				   \
+				    &response);				   \
+									   \
+	mutex_unlock(&dlb->resource_mutex);				   \
+									   \
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);		   \
+									   \
+	if (copy_to_user((void __user *)&uarg->response,		   \
+			 &response,					   \
+			 sizeof(response)))				   \
+		return -EFAULT;						   \
+									   \
+	return ret;							   \
+}
+
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
+
+long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+	struct dlb_domain *dom = f->private_data;
+	struct dlb *dlb = dom->dlb;
+
+	switch (cmd) {
+	case DLB_IOC_CREATE_LDB_QUEUE:
+		return dlb_domain_ioctl_create_ldb_queue(dlb, dom, arg);
+	case DLB_IOC_CREATE_DIR_QUEUE:
+		return dlb_domain_ioctl_create_dir_queue(dlb, dom, arg);
+	case DLB_IOC_GET_LDB_QUEUE_DEPTH:
+		return dlb_domain_ioctl_get_ldb_queue_depth(dlb, dom, arg);
+	case DLB_IOC_GET_DIR_QUEUE_DEPTH:
+		return dlb_domain_ioctl_get_dir_queue_depth(dlb, dom, arg);
+	default:
+		return -ENOTTY;
+	}
+}
+
 /* [7:0]: device revision, [15:8]: device version */
 #define DLB_SET_DEVICE_VERSION(ver, rev) (((ver) << 8) | (rev))
 
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 5484d9aee02c..64915824ca03 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -185,6 +185,8 @@ static int dlb_domain_close(struct inode *i, struct file *f)
 const struct file_operations dlb_domain_fops = {
 	.owner   = THIS_MODULE,
 	.release = dlb_domain_close,
+	.unlocked_ioctl = dlb_domain_ioctl,
+	.compat_ioctl = compat_ptr_ioctl,
 };
 
 /**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index ecfda11b297b..227630adf8ac 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -42,9 +42,25 @@ struct dlb_device_ops {
 	int (*create_sched_domain)(struct dlb_hw *hw,
 				   struct dlb_create_sched_domain_args *args,
 				   struct dlb_cmd_response *resp);
+	int (*create_ldb_queue)(struct dlb_hw *hw,
+				u32 domain_id,
+				struct dlb_create_ldb_queue_args *args,
+				struct dlb_cmd_response *resp);
+	int (*create_dir_queue)(struct dlb_hw *hw,
+				u32 domain_id,
+				struct dlb_create_dir_queue_args *args,
+				struct dlb_cmd_response *resp);
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
 	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
+	int (*get_ldb_queue_depth)(struct dlb_hw *hw,
+				   u32 domain_id,
+				   struct dlb_get_ldb_queue_depth_args *args,
+				   struct dlb_cmd_response *resp);
+	int (*get_dir_queue_depth)(struct dlb_hw *hw,
+				   u32 domain_id,
+				   struct dlb_get_dir_queue_depth_args *args,
+				   struct dlb_cmd_response *resp);
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
@@ -76,6 +92,7 @@ struct dlb {
 
 /* Prototypes for dlb_ioctl.c */
 long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
 
 int dlb_init_domain(struct dlb *dlb, u32 domain_id);
 void dlb_free_domain(struct kref *kref);
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 494a482368f6..32991c5f3366 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -114,6 +114,22 @@ static int dlb_pf_create_sched_domain(struct dlb_hw *hw,
 	return dlb_hw_create_sched_domain(hw, args, resp, false, 0);
 }
 
+static int
+dlb_pf_create_ldb_queue(struct dlb_hw *hw, u32 id,
+			struct dlb_create_ldb_queue_args *args,
+			struct dlb_cmd_response *resp)
+{
+	return dlb_hw_create_ldb_queue(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb_pf_create_dir_queue(struct dlb_hw *hw, u32 id,
+			struct dlb_create_dir_queue_args *args,
+			struct dlb_cmd_response *resp)
+{
+	return dlb_hw_create_dir_queue(hw, id, args, resp, false, 0);
+}
+
 static int dlb_pf_get_num_resources(struct dlb_hw *hw,
 				    struct dlb_get_num_resources_args *args)
 {
@@ -126,6 +142,22 @@ dlb_pf_reset_domain(struct dlb_hw *hw, u32 id)
 	return dlb_reset_domain(hw, id, false, 0);
 }
 
+static int
+dlb_pf_get_ldb_queue_depth(struct dlb_hw *hw, u32 id,
+			   struct dlb_get_ldb_queue_depth_args *args,
+			   struct dlb_cmd_response *resp)
+{
+	return dlb_hw_get_ldb_queue_depth(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb_pf_get_dir_queue_depth(struct dlb_hw *hw, u32 id,
+			   struct dlb_get_dir_queue_depth_args *args,
+			   struct dlb_cmd_response *resp)
+{
+	return dlb_hw_get_dir_queue_depth(hw, id, args, resp, false, 0);
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -137,6 +169,10 @@ struct dlb_device_ops dlb_pf_ops = {
 	.enable_pm = dlb_pf_enable_pm,
 	.wait_for_device_ready = dlb_pf_wait_for_device_ready,
 	.create_sched_domain = dlb_pf_create_sched_domain,
+	.create_ldb_queue = dlb_pf_create_ldb_queue,
+	.create_dir_queue = dlb_pf_create_dir_queue,
 	.get_num_resources = dlb_pf_get_num_resources,
 	.reset_domain = dlb_pf_reset_domain,
+	.get_ldb_queue_depth = dlb_pf_get_ldb_queue_depth,
+	.get_dir_queue_depth = dlb_pf_get_dir_queue_depth,
 };
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 26b7d9ea94e3..b36f14a661fa 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -224,6 +224,40 @@ static struct dlb_hw_domain *dlb_get_domain_from_id(struct dlb_hw *hw, u32 id,
 	return NULL;
 }
 
+static struct dlb_dir_pq_pair *
+dlb_get_domain_used_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *port;
+
+	if (id >= DLB_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+	}
+
+	return NULL;
+}
+
+static struct dlb_ldb_queue *
+dlb_get_domain_ldb_queue(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_queue *queue;
+
+	if (id >= DLB_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		if ((!vdev_req && queue->id.phys_id == id) ||
+		    (vdev_req && queue->id.virt_id == id))
+			return queue;
+	}
+
+	return NULL;
+}
+
 static int dlb_attach_ldb_queues(struct dlb_hw *hw,
 				 struct dlb_function_resources *rsrcs,
 				 struct dlb_hw_domain *domain, u32 num_queues,
@@ -660,6 +694,156 @@ dlb_verify_create_sched_dom_args(struct dlb_function_resources *rsrcs,
 	return 0;
 }
 
+static int
+dlb_verify_create_ldb_queue_args(struct dlb_hw *hw, u32 domain_id,
+				 struct dlb_create_ldb_queue_args *args,
+				 struct dlb_cmd_response *resp,
+				 bool vdev_req, unsigned int vdev_id,
+				 struct dlb_hw_domain **out_domain,
+				 struct dlb_ldb_queue **out_queue)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	int i;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	queue = list_first_entry_or_null(&domain->avail_ldb_queues,
+					 typeof(*queue), domain_list);
+	if (!queue) {
+		resp->status = DLB_ST_LDB_QUEUES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_sequence_numbers) {
+		for (i = 0; i < DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+			struct dlb_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+			if (group->sequence_numbers_per_queue ==
+			    args->num_sequence_numbers &&
+			    !dlb_sn_group_full(group))
+				break;
+		}
+
+		if (i == DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+			resp->status = DLB_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	if (args->num_qid_inflights > 4096) {
+		resp->status = DLB_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	/* Inflights must be <= number of sequence numbers if ordered */
+	if (args->num_sequence_numbers != 0 &&
+	    args->num_qid_inflights > args->num_sequence_numbers) {
+		resp->status = DLB_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+		return -EINVAL;
+	}
+
+	if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+		resp->status = DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	if (args->num_atomic_inflights &&
+	    args->lock_id_comp_level != 0 &&
+	    args->lock_id_comp_level != 64 &&
+	    args->lock_id_comp_level != 128 &&
+	    args->lock_id_comp_level != 256 &&
+	    args->lock_id_comp_level != 512 &&
+	    args->lock_id_comp_level != 1024 &&
+	    args->lock_id_comp_level != 2048 &&
+	    args->lock_id_comp_level != 4096 &&
+	    args->lock_id_comp_level != 65536) {
+		resp->status = DLB_ST_INVALID_LOCK_ID_COMP_LEVEL;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+	*out_queue = queue;
+
+	return 0;
+}
+
+static int
+dlb_verify_create_dir_queue_args(struct dlb_hw *hw, u32 domain_id,
+				 struct dlb_create_dir_queue_args *args,
+				 struct dlb_cmd_response *resp,
+				 bool vdev_req, unsigned int vdev_id,
+				 struct dlb_hw_domain **out_domain,
+				 struct dlb_dir_pq_pair **out_queue)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_dir_pq_pair *pq;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	/*
+	 * If the user claims the port is already configured, validate the port
+	 * ID, its domain, and whether the port is configured.
+	 */
+	if (args->port_id != -1) {
+		pq = dlb_get_domain_used_dir_pq(args->port_id,
+						vdev_req,
+						domain);
+
+		if (!pq || pq->domain_id.phys_id != domain->id.phys_id ||
+		    !pq->port_configured) {
+			resp->status = DLB_ST_INVALID_PORT_ID;
+			return -EINVAL;
+		}
+	} else {
+		/*
+		 * If the queue's port is not configured, validate that a free
+		 * port-queue pair is available.
+		 */
+		pq = list_first_entry_or_null(&domain->avail_dir_pq_pairs,
+					      typeof(*pq), domain_list);
+		if (!pq) {
+			resp->status = DLB_ST_DIR_QUEUES_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	*out_domain = domain;
+	*out_queue = pq;
+
+	return 0;
+}
+
 static void dlb_configure_domain_credits(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
 {
@@ -725,6 +909,68 @@ dlb_domain_attach_resources(struct dlb_hw *hw,
 	return 0;
 }
 
+static int
+dlb_ldb_queue_attach_to_sn_group(struct dlb_hw *hw,
+				 struct dlb_ldb_queue *queue,
+				 struct dlb_create_ldb_queue_args *args)
+{
+	int slot = -1;
+	int i;
+
+	queue->sn_cfg_valid = false;
+
+	if (args->num_sequence_numbers == 0)
+		return 0;
+
+	for (i = 0; i < DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+		struct dlb_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+		if (group->sequence_numbers_per_queue ==
+		    args->num_sequence_numbers &&
+		    !dlb_sn_group_full(group)) {
+			slot = dlb_sn_group_alloc_slot(group);
+			if (slot >= 0)
+				break;
+		}
+	}
+
+	if (slot == -1) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: no sequence number slots available\n",
+			   __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	queue->sn_cfg_valid = true;
+	queue->sn_group = i;
+	queue->sn_slot = slot;
+	return 0;
+}
+
+static int
+dlb_ldb_queue_attach_resources(struct dlb_hw *hw,
+			       struct dlb_hw_domain *domain,
+			       struct dlb_ldb_queue *queue,
+			       struct dlb_create_ldb_queue_args *args)
+{
+	int ret;
+
+	ret = dlb_ldb_queue_attach_to_sn_group(hw, queue, args);
+	if (ret)
+		return ret;
+
+	/* Attach QID inflights */
+	queue->num_qid_inflights = args->num_qid_inflights;
+
+	/* Attach atomic inflights */
+	queue->aqed_limit = args->num_atomic_inflights;
+
+	domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+	domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+	return 0;
+}
+
 static void
 dlb_log_create_sched_domain_args(struct dlb_hw *hw,
 				 struct dlb_create_sched_domain_args *args,
@@ -828,6 +1074,172 @@ int dlb_hw_create_sched_domain(struct dlb_hw *hw,
 	return 0;
 }
 
+static void
+dlb_log_create_ldb_queue_args(struct dlb_hw *hw, u32 domain_id,
+			      struct dlb_create_ldb_queue_args *args,
+			      bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB create load-balanced queue arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID:                  %d\n",
+		   domain_id);
+	DLB_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+		   args->num_sequence_numbers);
+	DLB_HW_DBG(hw, "\tNumber of QID inflights:    %d\n",
+		   args->num_qid_inflights);
+	DLB_HW_DBG(hw, "\tNumber of ATM inflights:    %d\n",
+		   args->num_atomic_inflights);
+}
+
+/**
+ * dlb_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    the domain has already been started, or the requested queue name is
+ *	    already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_ldb_queue(struct dlb_hw *hw, u32 domain_id,
+			    struct dlb_create_ldb_queue_args *args,
+			    struct dlb_cmd_response *resp,
+			    bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	int ret;
+
+	dlb_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_create_ldb_queue_args(hw, domain_id, args, resp,
+					       vdev_req, vdev_id, &domain, &queue);
+	if (ret)
+		return ret;
+
+	ret = dlb_ldb_queue_attach_resources(hw, domain, queue, args);
+	if (ret) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+			   __func__, __LINE__);
+		return ret;
+	}
+
+	queue->num_mappings = 0;
+
+	queue->configured = true;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	list_del(&queue->domain_list);
+
+	list_add(&queue->domain_list, &domain->used_ldb_queues);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb_log_create_dir_queue_args(struct dlb_hw *hw, u32 domain_id,
+			      struct dlb_create_dir_queue_args *args,
+			      bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB create directed queue arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB_HW_DBG(hw, "\tPort ID:   %d\n", args->port_id);
+}
+
+/**
+ * dlb_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ *	    or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
+			    struct dlb_create_dir_queue_args *args,
+			    struct dlb_cmd_response *resp,
+			    bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_dir_pq_pair *queue;
+	struct dlb_hw_domain *domain;
+	int ret;
+
+	dlb_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_create_dir_queue_args(hw, domain_id, args, resp,
+					       vdev_req, vdev_id, &domain, &queue);
+	if (ret)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->port_id == -1) {
+		list_del(&queue->domain_list);
+
+		list_add(&queue->domain_list, &domain->used_dir_pq_pairs);
+	}
+
+	resp->status = 0;
+
+	resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+	return 0;
+}
+
 static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 					   struct dlb_hw_domain *domain)
 {
@@ -973,6 +1385,151 @@ static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 	return 0;
 }
 
+static void dlb_log_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
+					u32 queue_id, bool vdev_req,
+					unsigned int vf_id)
+{
+	DLB_HW_DBG(hw, "DLB get directed queue depth:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+/**
+ * dlb_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_get_dir_queue_depth_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_dir_pq_pair *queue;
+	struct dlb_hw_domain *domain;
+	int id;
+
+	id = domain_id;
+
+	dlb_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+				    vdev_req, vdev_id);
+
+	domain = dlb_get_domain_from_id(hw, id, vdev_req, vdev_id);
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	id = args->queue_id;
+
+	queue = dlb_get_domain_used_dir_pq(id, vdev_req, domain);
+	if (!queue) {
+		resp->status = DLB_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = 0;
+
+	return 0;
+}
+
+static u32 dlb_ldb_queue_depth(struct dlb_hw *hw, struct dlb_ldb_queue *queue)
+{
+	u32 aqed, ldb, atm;
+
+	aqed = DLB_CSR_RD(hw, LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+	ldb = DLB_CSR_RD(hw, LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+	atm = DLB_CSR_RD(hw, LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+	return BITS_GET(aqed, LSP_QID_AQED_ACTIVE_CNT_COUNT)
+	       + BITS_GET(ldb, LSP_QID_LDB_ENQUEUE_CNT_COUNT)
+	       + BITS_GET(atm, LSP_QID_ATM_ACTIVE_COUNT);
+}
+
+static bool dlb_ldb_queue_is_empty(struct dlb_hw *hw, struct dlb_ldb_queue *queue)
+{
+	return dlb_ldb_queue_depth(hw, queue) == 0;
+}
+
+static void dlb_log_get_ldb_queue_depth(struct dlb_hw *hw, u32 domain_id,
+					u32 queue_id, bool vdev_req,
+					unsigned int vf_id)
+{
+	DLB_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+	DLB_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+/**
+ * dlb_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb_hw_get_ldb_queue_depth(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_get_ldb_queue_depth_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+
+	dlb_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+				    vdev_req, vdev_id);
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+	if (!queue) {
+		resp->status = DLB_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	resp->id = 0;
+
+	return 0;
+}
+
 static void __dlb_domain_reset_ldb_port_registers(struct dlb_hw *hw,
 						  struct dlb_ldb_port *port)
 {
@@ -1337,6 +1894,27 @@ static void dlb_domain_reset_dir_queue_registers(struct dlb_hw *hw,
 	}
 }
 
+static int dlb_domain_verify_reset_success(struct dlb_hw *hw,
+					   struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_queue *queue;
+
+	/*
+	 * Confirm that all the domain's queue's inflight counts and AQED
+	 * active counts are 0.
+	 */
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		if (!dlb_ldb_queue_is_empty(hw, queue)) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: failed to empty ldb queue %d\n",
+				   __func__, queue->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
+	return 0;
+}
+
 static void dlb_domain_reset_registers(struct dlb_hw *hw,
 				       struct dlb_hw_domain *domain)
 {
@@ -1395,6 +1973,7 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id)
 {
 	struct dlb_hw_domain *domain;
+	int ret;
 
 	dlb_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
 
@@ -1403,6 +1982,16 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	if (!domain || !domain->configured)
 		return -EINVAL;
 
+	/*
+	 * For each queue owned by this domain, disable its write permissions to
+	 * cause any traffic sent to it to be dropped. Well-behaved software
+	 * should not be sending QEs at this point.
+	 */
+
+	ret = dlb_domain_verify_reset_success(hw, domain);
+	if (ret)
+		return ret;
+
 	/* Reset the QID and port state. */
 	dlb_domain_reset_registers(hw, domain);
 
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 8c50f449cb9b..50e674e46dbb 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -19,6 +19,16 @@ int dlb_hw_create_sched_domain(struct dlb_hw *hw,
 			       struct dlb_cmd_response *resp,
 			       bool vdev_req, unsigned int vdev_id);
 
+int dlb_hw_create_ldb_queue(struct dlb_hw *hw, u32 domain_id,
+			    struct dlb_create_ldb_queue_args *args,
+			    struct dlb_cmd_response *resp,
+			    bool vdev_req, unsigned int vdev_id);
+
+int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
+			    struct dlb_create_dir_queue_args *args,
+			    struct dlb_cmd_response *resp,
+			    bool vdev_req, unsigned int vdev_id);
+
 int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id);
 
@@ -28,4 +38,14 @@ int dlb_hw_get_num_resources(struct dlb_hw *hw,
 
 void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
 
+int dlb_hw_get_ldb_queue_depth(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_get_ldb_queue_depth_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id);
+
+int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_get_dir_queue_depth_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id);
+
 #endif /* __DLB_RESOURCE_H */
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 0513116072a7..0c09a9bbc9d3 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -19,6 +19,15 @@ enum dlb_error {
 	DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
 	DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
 	DLB_ST_DOMAIN_RESET_FAILED,
+	DLB_ST_INVALID_DOMAIN_ID,
+	DLB_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+	DLB_ST_INVALID_LDB_QUEUE_ID,
+	DLB_ST_DOMAIN_NOT_CONFIGURED,
+	DLB_ST_INVALID_QID,
+	DLB_ST_DOMAIN_STARTED,
+	DLB_ST_DIR_QUEUES_UNAVAILABLE,
+	DLB_ST_INVALID_PORT_ID,
+	DLB_ST_INVALID_LOCK_ID_COMP_LEVEL,
 };
 
 struct dlb_cmd_response {
@@ -160,6 +169,132 @@ enum dlb_user_interface_commands {
 	NUM_DLB_CMD,
 };
 
+/*********************************/
+/* 'domain' device file commands */
+/*********************************/
+
+/*
+ * DLB_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: Queue ID.
+ *
+ * Input parameters:
+ * @num_atomic_inflights: This specifies the amount of temporary atomic QE
+ *	storage for this queue. If zero, the queue will not support atomic
+ *	scheduling.
+ * @num_sequence_numbers: This specifies the number of sequence numbers used
+ *	by this queue. If zero, the queue will not support ordered scheduling.
+ *	If non-zero, the queue will not support unordered scheduling.
+ * @num_qid_inflights: The maximum number of QEs that can be inflight
+ *	(scheduled to a CQ but not completed) at any time. If
+ *	num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ *	to num_sequence_numbers.
+ * @lock_id_comp_level: Lock ID compression level. Specifies the number of
+ *	unique lock IDs the queue should compress down to. Valid compression
+ *	levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ *	0, the queue won't compress its lock IDs.
+ * @depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ * @padding0: Reserved for future use.
+ */
+struct dlb_create_ldb_queue_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 num_sequence_numbers;
+	__u32 num_qid_inflights;
+	__u32 num_atomic_inflights;
+	__u32 lock_id_comp_level;
+	__u32 depth_threshold;
+	__u32 padding0;
+};
+
+/*
+ * DLB_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: Queue ID.
+ *
+ * Input parameters:
+ * @port_id: Port ID. If the corresponding directed port is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the queue is being created before the port.
+ * @depth_threshold: DLB sets two bits in the received QE to indicate the
+ *	depth of the queue relative to the threshold before scheduling the
+ *	QE to a CQ:
+ *	- 2’b11: depth > threshold
+ *	- 2’b10: threshold >= depth > 0.75 * threshold
+ *	- 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ *	- 2’b00: depth <= 0.5 * threshold
+ */
+struct dlb_create_dir_queue_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__s32 port_id;
+	__u32 depth_threshold;
+};
+
+/*
+ * DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: queue depth.
+ *
+ * Input parameters:
+ * @queue_id: The load-balanced queue ID.
+ * @padding0: Reserved for future use.
+ */
+struct dlb_get_ldb_queue_depth_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+/*
+ * DLB_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: queue depth.
+ *
+ * Input parameters:
+ * @queue_id: The directed queue ID.
+ * @padding0: Reserved for future use.
+ */
+struct dlb_get_dir_queue_depth_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 queue_id;
+	__u32 padding0;
+};
+
+enum dlb_domain_user_interface_commands {
+	DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,
+	DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,
+	DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,
+	DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
+
+	/* NUM_DLB_DOMAIN_CMD must be last */
+	NUM_DLB_DOMAIN_CMD,
+};
+
 /********************/
 /* dlb ioctl codes */
 /********************/
@@ -178,5 +313,21 @@ enum dlb_user_interface_commands {
 		_IOR(DLB_IOC_MAGIC,				\
 		     DLB_CMD_GET_NUM_RESOURCES,			\
 		     struct dlb_get_num_resources_args)
+#define DLB_IOC_CREATE_LDB_QUEUE				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,		\
+		      struct dlb_create_ldb_queue_args)
+#define DLB_IOC_CREATE_DIR_QUEUE				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,		\
+		      struct dlb_create_dir_queue_args)
+#define DLB_IOC_GET_LDB_QUEUE_DEPTH				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,	\
+		      struct dlb_get_ldb_queue_depth_args)
+#define DLB_IOC_GET_DIR_QUEUE_DEPTH				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,	\
+		      struct dlb_get_dir_queue_depth_args)
 
 #endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 10/20] dlb: add register operations for queue management
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (8 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 09/20] dlb: add queue create, reset, get-depth ioctls Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 11/20] dlb: add ioctl to configure ports and query poll mode Mike Ximing Chen
                   ` (10 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the low-level code for configuring a new queue and querying its depth.
When configuring a queue, program the device based on the user-supplied
queue configuration ioctl arguments.

Add low-level code for resetting (draining) a non-empty queue during
scheduling domain reset. Draining a queue is an iterative process of
checking if the queue is empty, and if not then selecting a linked 'victim'
port and dequeueing the queue's events through this port. A port can only
receive a small number of events at a time, usually much fewer than the
queue depth, so draining a queue typically takes multiple iterations. This
process is finite since software cannot enqueue new events to the DLB's
(finite) on-device storage.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_hw_types.h |  46 +++
 drivers/misc/dlb/dlb_resource.c | 610 +++++++++++++++++++++++++++++++-
 2 files changed, 654 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index d382c414e2b0..c7827defa66a 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -50,6 +50,29 @@
 
 #define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
 
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB_DRV_LDB_PP_BASE   0x2300000
+#define DLB_DRV_LDB_PP_STRIDE 0x1000
+#define DLB_DRV_LDB_PP_BOUND  (DLB_DRV_LDB_PP_BASE + \
+				DLB_DRV_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS)
+#define DLB_DRV_DIR_PP_BASE   0x2200000
+#define DLB_DRV_DIR_PP_STRIDE 0x1000
+#define DLB_DRV_DIR_PP_BOUND  (DLB_DRV_DIR_PP_BASE + \
+				DLB_DRV_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS)
+#define DLB_LDB_PP_BASE       0x2100000
+#define DLB_LDB_PP_STRIDE     0x1000
+#define DLB_LDB_PP_BOUND      (DLB_LDB_PP_BASE + \
+				DLB_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS)
+#define DLB_LDB_PP_OFFS(id)   (DLB_LDB_PP_BASE + (id) * DLB_PP_SIZE)
+#define DLB_DIR_PP_BASE       0x2000000
+#define DLB_DIR_PP_STRIDE     0x1000
+#define DLB_DIR_PP_BOUND      (DLB_DIR_PP_BASE + \
+				DLB_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS)
+#define DLB_DIR_PP_OFFS(id)   (DLB_DIR_PP_BASE + (id) * DLB_PP_SIZE)
+
 struct dlb_resource_id {
 	u32 phys_id;
 	u32 virt_id;
@@ -68,6 +91,29 @@ static inline u32 dlb_freelist_count(struct dlb_freelist *list)
 	return list->bound - list->base - list->offset;
 }
 
+struct dlb_hcw {
+	u64 data;
+	/* Word 3 */
+	u16 opaque;
+	u8 qid;
+	u8 sched_type:2;
+	u8 priority:3;
+	u8 msg_type:3;
+	/* Word 4 */
+	u16 lock_id;
+	u8 ts_flag:1;
+	u8 rsvd1:2;
+	u8 no_dec:1;
+	u8 cmp_id:4;
+	u8 cq_token:1;
+	u8 qe_comp:1;
+	u8 qe_frag:1;
+	u8 qe_valid:1;
+	u8 int_arm:1;
+	u8 error:1;
+	u8 rsvd:2;
+};
+
 struct dlb_ldb_queue {
 	struct list_head domain_list;
 	struct list_head func_list;
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index b36f14a661fa..3c4f4c4af2ac 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -1,12 +1,24 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
 
+#include <linux/log2.h>
 #include "dlb_bitmap.h"
 #include "dlb_hw_types.h"
 #include "dlb_main.h"
 #include "dlb_regs.h"
 #include "dlb_resource.h"
 
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb_flush_csr(struct dlb_hw *hw)
+{
+	DLB_CSR_RD(hw, SYS_TOTAL_VAS);
+}
+
 static void dlb_init_fn_rsrc_lists(struct dlb_function_resources *rsrc)
 {
 	int i;
@@ -844,6 +856,148 @@ dlb_verify_create_dir_queue_args(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void dlb_configure_ldb_queue(struct dlb_hw *hw,
+				    struct dlb_hw_domain *domain,
+				    struct dlb_ldb_queue *queue,
+				    struct dlb_create_ldb_queue_args *args,
+				    bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_sn_group *sn_group;
+	unsigned int offs;
+	u32 reg = 0;
+	u32 alimit;
+	u32 level;
+
+	/* QID write permissions are turned on when the domain is started */
+	offs = domain->id.phys_id * DLB_MAX_NUM_LDB_QUEUES + queue->id.phys_id;
+
+	DLB_CSR_WR(hw, SYS_LDB_VASQID_V(offs), reg);
+
+	/*
+	 * Unordered QIDs get 4K inflights, ordered get as many as the number
+	 * of sequence numbers.
+	 */
+	BITS_SET(reg, args->num_qid_inflights, LSP_QID_LDB_INFL_LIM_LIMIT);
+	DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), reg);
+
+	alimit = queue->aqed_limit;
+
+	if (alimit > DLB_MAX_NUM_AQED_ENTRIES)
+		alimit = DLB_MAX_NUM_AQED_ENTRIES;
+
+	reg = 0;
+	BITS_SET(reg, alimit, LSP_QID_AQED_ACTIVE_LIM_LIMIT);
+	DLB_CSR_WR(hw, LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id), reg);
+
+	level = args->lock_id_comp_level;
+	if (level >= 64 && level <= 4096)
+		BITS_SET(reg, ilog2(level) - 5, AQED_QID_HID_WIDTH_COMPRESS_CODE);
+	else
+		reg = 0;
+
+	DLB_CSR_WR(hw, AQED_QID_HID_WIDTH(queue->id.phys_id), reg);
+
+	reg = 0;
+	/* Don't timestamp QEs that pass through this queue */
+	DLB_CSR_WR(hw, SYS_LDB_QID_ITS(queue->id.phys_id), reg);
+
+	BITS_SET(reg, args->depth_threshold, LSP_QID_ATM_DEPTH_THRSH_THRESH);
+	DLB_CSR_WR(hw, LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, args->depth_threshold, LSP_QID_NALDB_DEPTH_THRSH_THRESH);
+	DLB_CSR_WR(hw, LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id), reg);
+
+	/*
+	 * This register limits the number of inflight flows a queue can have
+	 * at one time.  It has an upper bound of 2048, but can be
+	 * over-subscribed. 512 is chosen so that a single queue doesn't use
+	 * the entire atomic storage, but can use a substantial portion if
+	 * needed.
+	 */
+	reg = 0;
+	BITS_SET(reg, 512, AQED_QID_FID_LIM_QID_FID_LIMIT);
+	DLB_CSR_WR(hw, AQED_QID_FID_LIM(queue->id.phys_id), reg);
+
+	/* Configure SNs */
+	reg = 0;
+	sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+	BITS_SET(reg, sn_group->mode, CHP_ORD_QID_SN_MAP_MODE);
+	BITS_SET(reg, queue->sn_slot, CHP_ORD_QID_SN_MAP_SLOT);
+	BITS_SET(reg, sn_group->id, CHP_ORD_QID_SN_MAP_GRP);
+
+	DLB_CSR_WR(hw, CHP_ORD_QID_SN_MAP(queue->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, (u32)(args->num_sequence_numbers != 0),
+		 SYS_LDB_QID_CFG_V_SN_CFG_V);
+	BITS_SET(reg, (u32)(args->num_atomic_inflights != 0),
+		 SYS_LDB_QID_CFG_V_FID_CFG_V);
+
+	DLB_CSR_WR(hw, SYS_LDB_QID_CFG_V(queue->id.phys_id), reg);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+		reg = 0;
+		reg |= SYS_VF_LDB_VQID_V_VQID_V;
+		DLB_CSR_WR(hw, SYS_VF_LDB_VQID_V(offs), reg);
+
+		reg = 0;
+		BITS_SET(reg, queue->id.phys_id, SYS_VF_LDB_VQID2QID_QID);
+		DLB_CSR_WR(hw, SYS_VF_LDB_VQID2QID(offs), reg);
+
+		reg = 0;
+		BITS_SET(reg, queue->id.virt_id, SYS_LDB_QID2VQID_VQID);
+		DLB_CSR_WR(hw, SYS_LDB_QID2VQID(queue->id.phys_id), reg);
+	}
+
+	reg = 0;
+	reg |= SYS_LDB_QID_V_QID_V;
+	DLB_CSR_WR(hw, SYS_LDB_QID_V(queue->id.phys_id), reg);
+}
+
+static void dlb_configure_dir_queue(struct dlb_hw *hw,
+				    struct dlb_hw_domain *domain,
+				    struct dlb_dir_pq_pair *queue,
+				    struct dlb_create_dir_queue_args *args,
+				    bool vdev_req, unsigned int vdev_id)
+{
+	unsigned int offs;
+	u32 reg = 0;
+
+	/* QID write permissions are turned on when the domain is started */
+	offs = domain->id.phys_id * DLB_MAX_NUM_DIR_QUEUES +
+		queue->id.phys_id;
+
+	DLB_CSR_WR(hw, SYS_DIR_VASQID_V(offs), reg);
+
+	/* Don't timestamp QEs that pass through this queue */
+	DLB_CSR_WR(hw, SYS_DIR_QID_ITS(queue->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, args->depth_threshold, LSP_QID_DIR_DEPTH_THRSH_THRESH);
+	DLB_CSR_WR(hw, LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id), reg);
+
+	if (vdev_req) {
+		offs = vdev_id * DLB_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+		reg = 0;
+		reg |= SYS_VF_DIR_VQID_V_VQID_V;
+		DLB_CSR_WR(hw, SYS_VF_DIR_VQID_V(offs), reg);
+
+		reg = 0;
+		BITS_SET(reg, queue->id.phys_id, SYS_VF_DIR_VQID2QID_QID);
+		DLB_CSR_WR(hw, SYS_VF_DIR_VQID2QID(offs), reg);
+	}
+
+	reg = 0;
+	reg |= SYS_DIR_QID_V_QID_V;
+	DLB_CSR_WR(hw, SYS_DIR_QID_V(queue->id.phys_id), reg);
+
+	queue->queue_configured = true;
+}
+
 static void dlb_configure_domain_credits(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
 {
@@ -971,6 +1125,56 @@ dlb_ldb_queue_attach_resources(struct dlb_hw *hw,
 	return 0;
 }
 
+static void dlb_ldb_port_cq_enable(struct dlb_hw *hw,
+				   struct dlb_ldb_port *port)
+{
+	u32 reg = 0;
+
+	/*
+	 * Don't re-enable the port if a removal is pending. The caller should
+	 * mark this port as enabled (if it isn't already), and when the
+	 * removal completes the port will be enabled.
+	 */
+	if (port->num_pending_removals)
+		return;
+
+	DLB_CSR_WR(hw, LSP_CQ_LDB_DSBL(port->id.phys_id), reg);
+
+	dlb_flush_csr(hw);
+}
+
+static void dlb_ldb_port_cq_disable(struct dlb_hw *hw,
+				    struct dlb_ldb_port *port)
+{
+	u32 reg = 0;
+
+	reg |= LSP_CQ_LDB_DSBL_DISABLED;
+	DLB_CSR_WR(hw, LSP_CQ_LDB_DSBL(port->id.phys_id), reg);
+
+	dlb_flush_csr(hw);
+}
+
+static void dlb_dir_port_cq_enable(struct dlb_hw *hw,
+				   struct dlb_dir_pq_pair *port)
+{
+	u32 reg = 0;
+
+	DLB_CSR_WR(hw, LSP_CQ_DIR_DSBL(port->id.phys_id), reg);
+
+	dlb_flush_csr(hw);
+}
+
+static void dlb_dir_port_cq_disable(struct dlb_hw *hw,
+				    struct dlb_dir_pq_pair *port)
+{
+	u32 reg = 0;
+
+	reg |= LSP_CQ_DIR_DSBL_DISABLED;
+	DLB_CSR_WR(hw, LSP_CQ_DIR_DSBL(port->id.phys_id), reg);
+
+	dlb_flush_csr(hw);
+}
+
 static void
 dlb_log_create_sched_domain_args(struct dlb_hw *hw,
 				 struct dlb_create_sched_domain_args *args,
@@ -1147,6 +1351,8 @@ int dlb_hw_create_ldb_queue(struct dlb_hw *hw, u32 domain_id,
 		return ret;
 	}
 
+	dlb_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
 	queue->num_mappings = 0;
 
 	queue->configured = true;
@@ -1223,6 +1429,8 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	dlb_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
 	/*
 	 * Configuration succeeded, so move the resource from the 'avail' to
 	 * the 'used' list (if it's not already there).
@@ -1240,6 +1448,92 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static u32 dlb_ldb_cq_inflight_count(struct dlb_hw *hw,
+				     struct dlb_ldb_port *port)
+{
+	u32 cnt;
+
+	cnt = DLB_CSR_RD(hw, LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+	return BITS_GET(cnt, LSP_CQ_LDB_INFL_CNT_COUNT);
+}
+
+static u32 dlb_ldb_cq_token_count(struct dlb_hw *hw, struct dlb_ldb_port *port)
+{
+	u32 cnt;
+
+	cnt = DLB_CSR_RD(hw, LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return BITS_GET(cnt, LSP_CQ_LDB_TKN_CNT_TOKEN_COUNT) - port->init_tkn_cnt;
+}
+
+static void __iomem *dlb_producer_port_addr(struct dlb_hw *hw, u8 port_id,
+					    bool is_ldb)
+{
+	struct dlb *dlb = container_of(hw, struct dlb, hw);
+	uintptr_t address = (uintptr_t)dlb->hw.func_kva;
+	unsigned long size;
+
+	if (is_ldb) {
+		size = DLB_LDB_PP_STRIDE;
+		address += DLB_DRV_LDB_PP_BASE + size * port_id;
+	} else {
+		size = DLB_DIR_PP_STRIDE;
+		address += DLB_DRV_DIR_PP_BASE + size * port_id;
+	}
+
+	return (void __iomem *)address;
+}
+
+static void dlb_drain_ldb_cq(struct dlb_hw *hw, struct dlb_ldb_port *port)
+{
+	u32 infl_cnt, tkn_cnt;
+	unsigned int i;
+
+	infl_cnt = dlb_ldb_cq_inflight_count(hw, port);
+	tkn_cnt = dlb_ldb_cq_token_count(hw, port);
+
+	if (infl_cnt || tkn_cnt) {
+		struct dlb_hcw hcw_mem[8], *hcw;
+		void __iomem *pp_addr;
+
+		pp_addr = dlb_producer_port_addr(hw, port->id.phys_id, true);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a completion and token return and
+		 * the other HCWs as NOOPS
+		 */
+
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->qe_comp = (infl_cnt > 0);
+		hcw->cq_token = (tkn_cnt > 0);
+		hcw->lock_id = tkn_cnt - 1;
+
+		/* Return tokens in the first HCW */
+		iosubmit_cmds512(pp_addr, hcw, 1);
+
+		hcw->cq_token = 0;
+
+		/* Issue remaining completions (if any) */
+		for (i = 1; i < infl_cnt; i++)
+			iosubmit_cmds512(pp_addr, hcw, 1);
+
+		/*
+		 * To ensure outstanding HCWs reach the device before subsequent device
+		 * accesses, fence them.
+		 */
+		mb();
+	}
+}
+
 static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 					   struct dlb_hw_domain *domain)
 {
@@ -1385,6 +1679,21 @@ static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 	return 0;
 }
 
+static u32 dlb_dir_queue_depth(struct dlb_hw *hw, struct dlb_dir_pq_pair *queue)
+{
+	u32 cnt;
+
+	cnt = DLB_CSR_RD(hw, LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+	return BITS_GET(cnt, LSP_QID_DIR_ENQUEUE_CNT_COUNT);
+}
+
+static bool dlb_dir_queue_is_empty(struct dlb_hw *hw,
+				   struct dlb_dir_pq_pair *queue)
+{
+	return dlb_dir_queue_depth(hw, queue) == 0;
+}
+
 static void dlb_log_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
 					u32 queue_id, bool vdev_req,
 					unsigned int vf_id)
@@ -1446,7 +1755,7 @@ int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
 		return -EINVAL;
 	}
 
-	resp->id = 0;
+	resp->id = dlb_dir_queue_depth(hw, queue);
 
 	return 0;
 }
@@ -1525,7 +1834,7 @@ int dlb_hw_get_ldb_queue_depth(struct dlb_hw *hw, u32 domain_id,
 		return -EINVAL;
 	}
 
-	resp->id = 0;
+	resp->id = dlb_ldb_queue_depth(hw, queue);
 
 	return 0;
 }
@@ -1894,6 +2203,21 @@ static void dlb_domain_reset_dir_queue_registers(struct dlb_hw *hw,
 	}
 }
 
+static u32 dlb_dir_cq_token_count(struct dlb_hw *hw,
+				  struct dlb_dir_pq_pair *port)
+{
+	u32 cnt;
+
+	cnt = DLB_CSR_RD(hw, LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+	/*
+	 * Account for the initial token count, which is used in order to
+	 * provide a CQ with depth less than 8.
+	 */
+
+	return BITS_GET(cnt, LSP_CQ_DIR_TKN_CNT_COUNT) - port->init_tkn_cnt;
+}
+
 static int dlb_domain_verify_reset_success(struct dlb_hw *hw,
 					   struct dlb_hw_domain *domain)
 {
@@ -1935,6 +2259,270 @@ static void dlb_domain_reset_registers(struct dlb_hw *hw,
 		   CHP_CFG_DIR_VAS_CRD_RST);
 }
 
+static void dlb_domain_drain_ldb_cqs(struct dlb_hw *hw,
+				     struct dlb_hw_domain *domain,
+				     bool toggle_port)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			if (toggle_port)
+				dlb_ldb_port_cq_disable(hw, port);
+
+			dlb_drain_ldb_cq(hw, port);
+
+			if (toggle_port)
+				dlb_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static bool dlb_domain_mapped_queues_empty(struct dlb_hw *hw,
+					   struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_queue *queue;
+
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		if (queue->num_mappings == 0)
+			continue;
+
+		if (!dlb_ldb_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb_domain_drain_mapped_queues(struct dlb_hw *hw,
+					  struct dlb_hw_domain *domain)
+{
+	int i;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	if (domain->num_pending_removals > 0) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: failed to unmap domain queues\n",
+			   __func__);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < DLB_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		dlb_domain_drain_ldb_cqs(hw, domain, true);
+
+		if (dlb_domain_mapped_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: failed to empty queues\n",
+			   __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	dlb_domain_drain_ldb_cqs(hw, domain, true);
+
+	return 0;
+}
+
+static int dlb_drain_dir_cq(struct dlb_hw *hw, struct dlb_dir_pq_pair *port)
+{
+	unsigned int port_id = port->id.phys_id;
+	u32 cnt;
+
+	/* Return any outstanding tokens */
+	cnt = dlb_dir_cq_token_count(hw, port);
+
+	if (cnt != 0) {
+		struct dlb_hcw hcw_mem[8], *hcw;
+		void __iomem *pp_addr;
+
+		pp_addr = dlb_producer_port_addr(hw, port_id, false);
+
+		/* Point hcw to a 64B-aligned location */
+		hcw = (struct dlb_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+		/*
+		 * Program the first HCW for a batch token return and
+		 * the rest as NOOPS
+		 */
+		memset(hcw, 0, 4 * sizeof(*hcw));
+		hcw->cq_token = 1;
+		hcw->lock_id = cnt - 1;
+
+		iosubmit_cmds512(pp_addr, hcw, 1);
+
+		/*
+		 * To ensure outstanding HCWs reach the device before subsequent device
+		 * accesses, fence them.
+		 */
+		mb();
+	}
+
+	return 0;
+}
+
+static int dlb_domain_drain_dir_cqs(struct dlb_hw *hw,
+				    struct dlb_hw_domain *domain,
+				    bool toggle_port)
+{
+	struct dlb_dir_pq_pair *port;
+	int ret;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+		/*
+		 * Can't drain a port if it's not configured, and there's
+		 * nothing to drain if its queue is unconfigured.
+		 */
+		if (!port->port_configured || !port->queue_configured)
+			continue;
+
+		if (toggle_port)
+			dlb_dir_port_cq_disable(hw, port);
+
+		ret = dlb_drain_dir_cq(hw, port);
+		if (ret)
+			return ret;
+
+		if (toggle_port)
+			dlb_dir_port_cq_enable(hw, port);
+	}
+
+	return 0;
+}
+
+static bool dlb_domain_dir_queues_empty(struct dlb_hw *hw,
+					struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *queue;
+
+	list_for_each_entry(queue, &domain->used_dir_pq_pairs, domain_list) {
+		if (!dlb_dir_queue_is_empty(hw, queue))
+			return false;
+	}
+
+	return true;
+}
+
+static int dlb_domain_drain_dir_queues(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain)
+{
+	int i, ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	for (i = 0; i < DLB_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+		ret = dlb_domain_drain_dir_cqs(hw, domain, true);
+		if (ret)
+			return ret;
+
+		if (dlb_domain_dir_queues_empty(hw, domain))
+			break;
+	}
+
+	if (i == DLB_MAX_QID_EMPTY_CHECK_LOOPS) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: failed to empty queues\n",
+			   __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * Drain the CQs one more time. For the queues to go empty, they would
+	 * have scheduled one or more QEs.
+	 */
+	ret = dlb_domain_drain_dir_cqs(hw, domain, true);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void
+dlb_domain_disable_ldb_queue_write_perms(struct dlb_hw *hw,
+					 struct dlb_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB_MAX_NUM_LDB_QUEUES;
+	struct dlb_ldb_queue *queue;
+
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		int idx = domain_offset + queue->id.phys_id;
+
+		DLB_CSR_WR(hw, SYS_LDB_VASQID_V(idx), 0);
+	}
+}
+
+static void
+dlb_domain_disable_dir_queue_write_perms(struct dlb_hw *hw,
+					 struct dlb_hw_domain *domain)
+{
+	int domain_offset = domain->id.phys_id * DLB_MAX_NUM_DIR_PORTS;
+	struct dlb_dir_pq_pair *queue;
+
+	list_for_each_entry(queue, &domain->used_dir_pq_pairs, domain_list) {
+		int idx = domain_offset + queue->id.phys_id;
+
+		DLB_CSR_WR(hw, SYS_DIR_VASQID_V(idx), 0);
+	}
+}
+
+static void dlb_domain_disable_dir_cqs(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *port;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+		port->enabled = false;
+
+		dlb_dir_port_cq_disable(hw, port);
+	}
+}
+
+static void dlb_domain_disable_ldb_cqs(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			port->enabled = false;
+
+			dlb_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb_domain_enable_ldb_cqs(struct dlb_hw *hw,
+				      struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			port->enabled = true;
+
+			dlb_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
 static void dlb_log_reset_domain(struct dlb_hw *hw, u32 domain_id,
 				 bool vdev_req, unsigned int vdev_id)
 {
@@ -1987,6 +2575,24 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	 * cause any traffic sent to it to be dropped. Well-behaved software
 	 * should not be sending QEs at this point.
 	 */
+	dlb_domain_disable_dir_queue_write_perms(hw, domain);
+
+	dlb_domain_disable_ldb_queue_write_perms(hw, domain);
+
+	/* Re-enable the CQs in order to drain the mapped queues. */
+	dlb_domain_enable_ldb_cqs(hw, domain);
+
+	ret = dlb_domain_drain_mapped_queues(hw, domain);
+	if (ret)
+		return ret;
+
+	/* Done draining LDB QEs, so disable the CQs. */
+	dlb_domain_disable_ldb_cqs(hw, domain);
+
+	dlb_domain_drain_dir_queues(hw, domain);
+
+	/* Done draining DIR QEs, so disable the CQs. */
+	dlb_domain_disable_dir_cqs(hw, domain);
 
 	ret = dlb_domain_verify_reset_success(hw, domain);
 	if (ret)
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 11/20] dlb: add ioctl to configure ports and query poll mode
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (9 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 10/20] dlb: add register operations for queue management Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 12/20] dlb: add register operations for port management Mike Ximing Chen
                   ` (9 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add high-level code for port configuration and poll-mode query ioctl
commands, argument verification, and placeholder functions for the
low-level register accesses. A subsequent commit will add the low-level
logic.

The port is a core's interface to the DLB, and it consists of an MMIO page
(the "producer port" (PP)) through which the core enqueues queue entries
and an in-memory queue (the "consumer queue" (CQ)) to which the device
schedules QEs. A subsequent commit will add the mmap interface for an
application to directly access the PP and CQ regions.

The driver allocates DMA memory for each port's CQ, and frees this memory
during domain reset or driver removal. Domain reset will also drains each
port's CQ and disables them from further scheduling.

The device supports two formats ("standard" and "sparse") for CQ entries,
dubbed the "poll mode". This (device-wide) mode is selected by the driver;
to determine the mode at run time, the driver provides an ioctl for
user-space software to query which mode the driver has configured. In this
way, the policy of which mode to use is decoupled from user-space software.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_ioctl.c    | 137 +++++++++++
 drivers/misc/dlb/dlb_main.c     |  51 +++++
 drivers/misc/dlb/dlb_main.h     |  24 ++
 drivers/misc/dlb/dlb_pf_ops.c   |  47 ++++
 drivers/misc/dlb/dlb_resource.c | 387 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |  16 ++
 include/uapi/linux/dlb.h        | 105 +++++++++
 7 files changed, 767 insertions(+)

diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 0fc20b32f0cf..84bf833631bd 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -52,6 +52,115 @@ DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
 
+/*
+ * Port creation ioctls don't use the callback template macro.
+ */
+static int dlb_domain_ioctl_create_ldb_port(struct dlb *dlb,
+					    struct dlb_domain *domain,
+					    unsigned long user_arg)
+{
+	struct dlb_create_ldb_port_args __user *uarg;
+	struct dlb_cmd_response response = {0};
+	struct dlb_create_ldb_port_args arg;
+	dma_addr_t cq_dma_base = 0;
+	void *cq_base;
+	int ret;
+
+	uarg = (void __user *)user_arg;
+	if (copy_from_user(&arg, uarg, sizeof(arg)))
+		return -EFAULT;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	cq_base = dma_alloc_coherent(&dlb->pdev->dev, DLB_CQ_SIZE, &cq_dma_base,
+				     GFP_KERNEL);
+	if (!cq_base) {
+		response.status = DLB_ST_NO_MEMORY;
+		ret = -ENOMEM;
+		goto unlock;
+	}
+
+	ret = dlb->ops->create_ldb_port(&dlb->hw, domain->id, &arg,
+					(uintptr_t)cq_dma_base, &response);
+	if (ret)
+		goto unlock;
+
+	/* Fill out the per-port data structure */
+	dlb->ldb_port[response.id].id = response.id;
+	dlb->ldb_port[response.id].is_ldb = true;
+	dlb->ldb_port[response.id].domain = domain;
+	dlb->ldb_port[response.id].cq_base = cq_base;
+	dlb->ldb_port[response.id].cq_dma_base = cq_dma_base;
+	dlb->ldb_port[response.id].valid = true;
+
+unlock:
+	if (ret && cq_dma_base)
+		dma_free_coherent(&dlb->pdev->dev, DLB_CQ_SIZE, cq_base,
+				  cq_dma_base);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+		return -EFAULT;
+
+	return ret;
+}
+
+static int dlb_domain_ioctl_create_dir_port(struct dlb *dlb,
+					    struct dlb_domain *domain,
+					    unsigned long user_arg)
+{
+	struct dlb_create_dir_port_args __user *uarg;
+	struct dlb_cmd_response response = {0};
+	struct dlb_create_dir_port_args arg;
+	dma_addr_t cq_dma_base = 0;
+	void *cq_base;
+	int ret;
+
+	uarg = (void __user *)user_arg;
+	if (copy_from_user(&arg, uarg, sizeof(arg)))
+		return -EFAULT;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	cq_base = dma_alloc_coherent(&dlb->pdev->dev, DLB_CQ_SIZE, &cq_dma_base,
+				     GFP_KERNEL);
+	if (!cq_base) {
+		response.status = DLB_ST_NO_MEMORY;
+		ret = -ENOMEM;
+		goto unlock;
+	}
+
+	ret = dlb->ops->create_dir_port(&dlb->hw, domain->id, &arg,
+					(uintptr_t)cq_dma_base, &response);
+	if (ret)
+		goto unlock;
+
+	/* Fill out the per-port data structure */
+	dlb->dir_port[response.id].id = response.id;
+	dlb->dir_port[response.id].is_ldb = false;
+	dlb->dir_port[response.id].domain = domain;
+	dlb->dir_port[response.id].cq_base = cq_base;
+	dlb->dir_port[response.id].cq_dma_base = cq_dma_base;
+	dlb->dir_port[response.id].valid = true;
+
+unlock:
+	if (ret && cq_dma_base)
+		dma_free_coherent(&dlb->pdev->dev, DLB_CQ_SIZE, cq_base,
+				  cq_dma_base);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+		return -EFAULT;
+
+	return ret;
+}
+
 long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 {
 	struct dlb_domain *dom = f->private_data;
@@ -66,6 +175,10 @@ long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 		return dlb_domain_ioctl_get_ldb_queue_depth(dlb, dom, arg);
 	case DLB_IOC_GET_DIR_QUEUE_DEPTH:
 		return dlb_domain_ioctl_get_dir_queue_depth(dlb, dom, arg);
+	case DLB_IOC_CREATE_LDB_PORT:
+		return dlb_domain_ioctl_create_ldb_port(dlb, dom, arg);
+	case DLB_IOC_CREATE_DIR_PORT:
+		return dlb_domain_ioctl_create_dir_port(dlb, dom, arg);
 	default:
 		return -ENOTTY;
 	}
@@ -189,6 +302,28 @@ static int dlb_ioctl_get_num_resources(struct dlb *dlb, unsigned long user_arg)
 	return ret;
 }
 
+static int dlb_ioctl_query_cq_poll_mode(struct dlb *dlb, unsigned long user_arg)
+{
+	struct dlb_query_cq_poll_mode_args __user *uarg;
+	struct dlb_cmd_response response = {0};
+	int ret;
+
+	uarg = (void __user *)user_arg;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	ret = dlb->ops->query_cq_poll_mode(dlb, &response);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	BUILD_BUG_ON(offsetof(typeof(*uarg), response) != 0);
+
+	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+		return -EFAULT;
+
+	return ret;
+}
+
 long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 {
 	struct dlb *dlb = f->private_data;
@@ -200,6 +335,8 @@ long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 		return dlb_ioctl_create_sched_domain(dlb, arg);
 	case DLB_IOC_GET_NUM_RESOURCES:
 		return dlb_ioctl_get_num_resources(dlb, arg);
+	case DLB_IOC_QUERY_CQ_POLL_MODE:
+		return dlb_ioctl_query_cq_poll_mode(dlb, arg);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 64915824ca03..e4c19714f1c4 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -130,12 +130,56 @@ int dlb_init_domain(struct dlb *dlb, u32 domain_id)
 	return 0;
 }
 
+static void dlb_release_port_memory(struct dlb *dlb,
+				    struct dlb_port *port,
+				    bool check_domain,
+				    u32 domain_id)
+{
+	if (port->valid &&
+	    (!check_domain || port->domain->id == domain_id))
+		dma_free_coherent(&dlb->pdev->dev,
+				  DLB_CQ_SIZE,
+				  port->cq_base,
+				  port->cq_dma_base);
+
+	port->valid = false;
+}
+
+static void dlb_release_domain_memory(struct dlb *dlb,
+				      bool check_domain,
+				      u32 domain_id)
+{
+	struct dlb_port *port;
+	int i;
+
+	for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+		port = &dlb->ldb_port[i];
+
+		dlb_release_port_memory(dlb, port, check_domain, domain_id);
+	}
+
+	for (i = 0; i < DLB_MAX_NUM_DIR_PORTS; i++) {
+		port = &dlb->dir_port[i];
+
+		dlb_release_port_memory(dlb, port, check_domain, domain_id);
+	}
+}
+
+static void dlb_release_device_memory(struct dlb *dlb)
+{
+	dlb_release_domain_memory(dlb, false, 0);
+}
+
 static int __dlb_free_domain(struct dlb_domain *domain)
 {
 	struct dlb *dlb = domain->dlb;
 	int ret;
 
 	ret = dlb->ops->reset_domain(&dlb->hw, domain->id);
+
+	/* Unpin and free all memory pages associated with the domain */
+	dlb_release_domain_memory(dlb, true, domain->id);
+
 	if (ret) {
 		dlb->domain_reset_failed = true;
 		dev_err(dlb->dev,
@@ -276,6 +320,8 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 	if (ret)
 		goto init_driver_state_fail;
 
+	dlb->ops->init_hardware(dlb);
+
 	/*
 	 * Undo the 'get' operation by the PCI layer during probe and
 	 * (if PF) immediately suspend the device. Since the device is only
@@ -312,6 +358,8 @@ static void dlb_remove(struct pci_dev *pdev)
 
 	dlb_resource_free(&dlb->hw);
 
+	dlb_release_device_memory(dlb);
+
 	device_destroy(dlb_class, dlb->dev_number);
 
 	pci_disable_pcie_error_reporting(pdev);
@@ -325,6 +373,9 @@ static void dlb_remove(struct pci_dev *pdev)
 static void dlb_reset_hardware_state(struct dlb *dlb)
 {
 	dlb_reset_device(dlb->pdev);
+
+	/* Reinitialize any other hardware state */
+	dlb->ops->init_hardware(dlb);
 }
 
 static int dlb_runtime_suspend(struct device *dev)
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 227630adf8ac..08dead13fb11 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -50,6 +50,16 @@ struct dlb_device_ops {
 				u32 domain_id,
 				struct dlb_create_dir_queue_args *args,
 				struct dlb_cmd_response *resp);
+	int (*create_ldb_port)(struct dlb_hw *hw,
+			       u32 domain_id,
+			       struct dlb_create_ldb_port_args *args,
+			       uintptr_t cq_dma_base,
+			       struct dlb_cmd_response *resp);
+	int (*create_dir_port)(struct dlb_hw *hw,
+			       u32 domain_id,
+			       struct dlb_create_dir_port_args *args,
+			       uintptr_t cq_dma_base,
+			       struct dlb_cmd_response *resp);
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
 	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
@@ -61,11 +71,23 @@ struct dlb_device_ops {
 				   u32 domain_id,
 				   struct dlb_get_dir_queue_depth_args *args,
 				   struct dlb_cmd_response *resp);
+	void (*init_hardware)(struct dlb *dlb);
+	int (*query_cq_poll_mode)(struct dlb *dlb,
+				  struct dlb_cmd_response *user_resp);
 };
 
 extern struct dlb_device_ops dlb_pf_ops;
 extern const struct file_operations dlb_domain_fops;
 
+struct dlb_port {
+	void *cq_base;
+	dma_addr_t cq_dma_base;
+	struct dlb_domain *domain;
+	int id;
+	u8 is_ldb;
+	u8 valid;
+};
+
 struct dlb_domain {
 	struct dlb *dlb;
 	struct kref refcnt;
@@ -79,6 +101,8 @@ struct dlb {
 	struct device *dev;
 	struct dlb_domain *sched_domains[DLB_MAX_NUM_DOMAINS];
 	struct file *f;
+	struct dlb_port ldb_port[DLB_MAX_NUM_LDB_PORTS];
+	struct dlb_port dir_port[DLB_MAX_NUM_DIR_PORTS];
 	/*
 	 * The resource mutex serializes access to driver data structures and
 	 * hardware registers.
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 32991c5f3366..c2ce03114f8b 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -103,6 +103,18 @@ static int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
 	return 0;
 }
 
+static bool dlb_sparse_cq_enabled = true;
+
+static void
+dlb_pf_init_hardware(struct dlb *dlb)
+{
+	if (dlb_sparse_cq_enabled) {
+		dlb_hw_enable_sparse_ldb_cq_mode(&dlb->hw);
+
+		dlb_hw_enable_sparse_dir_cq_mode(&dlb->hw);
+	}
+}
+
 /*****************************/
 /****** IOCTL callbacks ******/
 /*****************************/
@@ -130,6 +142,24 @@ dlb_pf_create_dir_queue(struct dlb_hw *hw, u32 id,
 	return dlb_hw_create_dir_queue(hw, id, args, resp, false, 0);
 }
 
+static int
+dlb_pf_create_ldb_port(struct dlb_hw *hw, u32 id,
+		       struct dlb_create_ldb_port_args *args,
+		       uintptr_t cq_dma_base, struct dlb_cmd_response *resp)
+{
+	return dlb_hw_create_ldb_port(hw, id, args, cq_dma_base,
+				       resp, false, 0);
+}
+
+static int
+dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
+		       struct dlb_create_dir_port_args *args,
+		       uintptr_t cq_dma_base, struct dlb_cmd_response *resp)
+{
+	return dlb_hw_create_dir_port(hw, id, args, cq_dma_base,
+				       resp, false, 0);
+}
+
 static int dlb_pf_get_num_resources(struct dlb_hw *hw,
 				    struct dlb_get_num_resources_args *args)
 {
@@ -158,6 +188,19 @@ dlb_pf_get_dir_queue_depth(struct dlb_hw *hw, u32 id,
 	return dlb_hw_get_dir_queue_depth(hw, id, args, resp, false, 0);
 }
 
+static int
+dlb_pf_query_cq_poll_mode(struct dlb *dlb, struct dlb_cmd_response *user_resp)
+{
+	user_resp->status = 0;
+
+	if (dlb_sparse_cq_enabled)
+		user_resp->id = DLB_CQ_POLL_MODE_SPARSE;
+	else
+		user_resp->id = DLB_CQ_POLL_MODE_STD;
+
+	return 0;
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -171,8 +214,12 @@ struct dlb_device_ops dlb_pf_ops = {
 	.create_sched_domain = dlb_pf_create_sched_domain,
 	.create_ldb_queue = dlb_pf_create_ldb_queue,
 	.create_dir_queue = dlb_pf_create_dir_queue,
+	.create_ldb_port = dlb_pf_create_ldb_port,
+	.create_dir_port = dlb_pf_create_dir_port,
 	.get_num_resources = dlb_pf_get_num_resources,
 	.reset_domain = dlb_pf_reset_domain,
 	.get_ldb_queue_depth = dlb_pf_get_ldb_queue_depth,
 	.get_dir_queue_depth = dlb_pf_get_dir_queue_depth,
+	.init_hardware = dlb_pf_init_hardware,
+	.query_cq_poll_mode = dlb_pf_query_cq_poll_mode,
 };
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 3c4f4c4af2ac..ac6c5889c435 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -998,6 +998,178 @@ static void dlb_configure_dir_queue(struct dlb_hw *hw,
 	queue->queue_configured = true;
 }
 
+static bool
+dlb_cq_depth_is_valid(u32 depth)
+{
+	u32 n = ilog2(depth);
+
+	/* Valid values for depth are
+	 * 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, and 1024.
+	 */
+	if (depth > 1024 || ((1U << n) != depth))
+		return false;
+
+	return true;
+}
+
+static int
+dlb_verify_create_ldb_port_args(struct dlb_hw *hw, u32 domain_id,
+				uintptr_t cq_dma_base,
+				struct dlb_create_ldb_port_args *args,
+				struct dlb_cmd_response *resp,
+				bool vdev_req, unsigned int vdev_id,
+				struct dlb_hw_domain **out_domain,
+				struct dlb_ldb_port **out_port, int *out_cos_id)
+{
+	struct dlb_ldb_port *port = NULL;
+	struct dlb_hw_domain *domain;
+	int i, id;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->cos_id >= DLB_NUM_COS_DOMAINS) {
+		resp->status = DLB_ST_INVALID_COS_ID;
+		return -EINVAL;
+	}
+
+	if (args->cos_strict) {
+		id = args->cos_id;
+		port = list_first_entry_or_null(&domain->avail_ldb_ports[id],
+						typeof(*port), domain_list);
+	} else {
+		for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+			id = (args->cos_id + i) % DLB_NUM_COS_DOMAINS;
+
+			port = list_first_entry_or_null(&domain->avail_ldb_ports[id],
+							typeof(*port), domain_list);
+			if (port)
+				break;
+		}
+	}
+
+	if (!port) {
+		resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	/* DLB requires 64B alignment */
+	if (!IS_ALIGNED(cq_dma_base, 64)) {
+		resp->status = DLB_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (!dlb_cq_depth_is_valid(args->cq_depth)) {
+		resp->status = DLB_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	/* The history list size must be >= 1 */
+	if (!args->cq_history_list_size) {
+		resp->status = DLB_ST_INVALID_HIST_LIST_DEPTH;
+		return -EINVAL;
+	}
+
+	if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+		resp->status = DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+	*out_port = port;
+	*out_cos_id = id;
+
+	return 0;
+}
+
+static int
+dlb_verify_create_dir_port_args(struct dlb_hw *hw, u32 domain_id,
+				uintptr_t cq_dma_base,
+				struct dlb_create_dir_port_args *args,
+				struct dlb_cmd_response *resp,
+				bool vdev_req, unsigned int vdev_id,
+				struct dlb_hw_domain **out_domain,
+				struct dlb_dir_pq_pair **out_port)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_dir_pq_pair *pq;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	if (args->queue_id != -1) {
+		/*
+		 * If the user claims the queue is already configured, validate
+		 * the queue ID, its domain, and whether the queue is
+		 * configured.
+		 */
+		pq = dlb_get_domain_used_dir_pq(args->queue_id,
+						vdev_req,
+						domain);
+
+		if (!pq || pq->domain_id.phys_id != domain->id.phys_id ||
+		    !pq->queue_configured) {
+			resp->status = DLB_ST_INVALID_DIR_QUEUE_ID;
+			return -EINVAL;
+		}
+	} else {
+		/*
+		 * If the port's queue is not configured, validate that a free
+		 * port-queue pair is available.
+		 */
+		pq = list_first_entry_or_null(&domain->avail_dir_pq_pairs,
+					      typeof(*pq), domain_list);
+		if (!pq) {
+			resp->status = DLB_ST_DIR_PORTS_UNAVAILABLE;
+			return -EINVAL;
+		}
+	}
+
+	/* DLB requires 64B alignment */
+	if (!IS_ALIGNED(cq_dma_base, 64)) {
+		resp->status = DLB_ST_INVALID_CQ_VIRT_ADDR;
+		return -EINVAL;
+	}
+
+	if (!dlb_cq_depth_is_valid(args->cq_depth)) {
+		resp->status = DLB_ST_INVALID_CQ_DEPTH;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+	*out_port = pq;
+
+	return 0;
+}
+
 static void dlb_configure_domain_credits(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
 {
@@ -1448,6 +1620,177 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void
+dlb_log_create_ldb_port_args(struct dlb_hw *hw, u32 domain_id,
+			     uintptr_t cq_dma_base,
+			     struct dlb_create_ldb_port_args *args,
+			     bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB create load-balanced port arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		   domain_id);
+	DLB_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		   args->cq_depth);
+	DLB_HW_DBG(hw, "\tCQ hist list size:         %d\n",
+		   args->cq_history_list_size);
+	DLB_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		   cq_dma_base);
+	DLB_HW_DBG(hw, "\tCoS ID:                    %u\n", args->cos_id);
+	DLB_HW_DBG(hw, "\tStrict CoS allocation:     %u\n",
+		   args->cos_strict);
+}
+
+/**
+ * dlb_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_ldb_port(struct dlb_hw *hw, u32 domain_id,
+			   struct dlb_create_ldb_port_args *args,
+			   uintptr_t cq_dma_base,
+			   struct dlb_cmd_response *resp,
+			   bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_port *port;
+	int ret, cos_id;
+
+	dlb_log_create_ldb_port_args(hw, domain_id, cq_dma_base,
+				     args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_create_ldb_port_args(hw, domain_id, cq_dma_base, args,
+					      resp, vdev_req, vdev_id, &domain,
+					      &port, &cos_id);
+	if (ret)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list.
+	 */
+	list_del(&port->domain_list);
+
+	list_add(&port->domain_list, &domain->used_ldb_ports[cos_id]);
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
+static void
+dlb_log_create_dir_port_args(struct dlb_hw *hw,
+			     u32 domain_id, uintptr_t cq_dma_base,
+			     struct dlb_create_dir_port_args *args,
+			     bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB create directed port arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID:                 %d\n",
+		   domain_id);
+	DLB_HW_DBG(hw, "\tCQ depth:                  %d\n",
+		   args->cq_depth);
+	DLB_HW_DBG(hw, "\tCQ base address:           0x%lx\n",
+		   cq_dma_base);
+}
+
+/**
+ * dlb_hw_create_dir_port() - create a directed port
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ *	    pointer address is not properly aligned, the domain is not
+ *	    configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
+			   struct dlb_create_dir_port_args *args,
+			   uintptr_t cq_dma_base,
+			   struct dlb_cmd_response *resp,
+			   bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_dir_pq_pair *port;
+	struct dlb_hw_domain *domain;
+	int ret;
+
+	dlb_log_create_dir_port_args(hw, domain_id, cq_dma_base, args,
+				     vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_create_dir_port_args(hw, domain_id, cq_dma_base,
+					      args, resp, vdev_req, vdev_id,
+					      &domain, &port);
+	if (ret)
+		return ret;
+
+	/*
+	 * Configuration succeeded, so move the resource from the 'avail' to
+	 * the 'used' list (if it's not already there).
+	 */
+	if (args->queue_id == -1) {
+		list_del(&port->domain_list);
+
+		list_add(&port->domain_list, &domain->used_dir_pq_pairs);
+	}
+
+	resp->status = 0;
+	resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+	return 0;
+}
+
 static u32 dlb_ldb_cq_inflight_count(struct dlb_hw *hw,
 				     struct dlb_ldb_port *port)
 {
@@ -2579,6 +2922,15 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 
 	dlb_domain_disable_ldb_queue_write_perms(hw, domain);
 
+	/*
+	 * Disable the LDB CQs and drain them in order to complete the map and
+	 * unmap procedures, which require zero CQ inflights and zero QID
+	 * inflights respectively.
+	 */
+	dlb_domain_disable_ldb_cqs(hw, domain);
+
+	dlb_domain_drain_ldb_cqs(hw, domain, false);
+
 	/* Re-enable the CQs in order to drain the mapped queues. */
 	dlb_domain_enable_ldb_cqs(hw, domain);
 
@@ -2684,3 +3036,38 @@ void dlb_clr_pmcsr_disable(struct dlb_hw *hw)
 
 	DLB_CSR_WR(hw, CM_CFG_PM_PMCSR_DISABLE, pmcsr_dis);
 }
+
+/**
+ * dlb_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ *      ports.
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb_hw_enable_sparse_ldb_cq_mode(struct dlb_hw *hw)
+{
+	u32 ctrl;
+
+	ctrl = DLB_CSR_RD(hw, CHP_CFG_CHP_CSR_CTRL);
+
+	ctrl |= CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_LDB_CQ_MODE;
+
+	DLB_CSR_WR(hw, CHP_CFG_CHP_CSR_CTRL, ctrl);
+}
+
+/**
+ * dlb_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb_hw_enable_sparse_dir_cq_mode(struct dlb_hw *hw)
+{
+	u32 ctrl;
+
+	ctrl = DLB_CSR_RD(hw, CHP_CFG_CHP_CSR_CTRL);
+
+	ctrl |= CHP_CFG_CHP_CSR_CTRL_CFG_64BYTES_QE_DIR_CQ_MODE;
+
+	DLB_CSR_WR(hw, CHP_CFG_CHP_CSR_CTRL, ctrl);
+}
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 50e674e46dbb..bbe25a417cd4 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -29,6 +29,18 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
 			    struct dlb_cmd_response *resp,
 			    bool vdev_req, unsigned int vdev_id);
 
+int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
+			   struct dlb_create_dir_port_args *args,
+			   uintptr_t cq_dma_base,
+			   struct dlb_cmd_response *resp,
+			   bool vdev_req, unsigned int vdev_id);
+
+int dlb_hw_create_ldb_port(struct dlb_hw *hw, u32 domain_id,
+			   struct dlb_create_ldb_port_args *args,
+			   uintptr_t cq_dma_base,
+			   struct dlb_cmd_response *resp,
+			   bool vdev_req, unsigned int vdev_id);
+
 int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id);
 
@@ -48,4 +60,8 @@ int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
 			       struct dlb_cmd_response *resp,
 			       bool vdev_req, unsigned int vdev_id);
 
+void dlb_hw_enable_sparse_ldb_cq_mode(struct dlb_hw *hw);
+
+void dlb_hw_enable_sparse_dir_cq_mode(struct dlb_hw *hw);
+
 #endif /* __DLB_RESOURCE_H */
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 0c09a9bbc9d3..9578d8f1c03b 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -28,6 +28,12 @@ enum dlb_error {
 	DLB_ST_DIR_QUEUES_UNAVAILABLE,
 	DLB_ST_INVALID_PORT_ID,
 	DLB_ST_INVALID_LOCK_ID_COMP_LEVEL,
+	DLB_ST_NO_MEMORY,
+	DLB_ST_INVALID_COS_ID,
+	DLB_ST_INVALID_CQ_VIRT_ADDR,
+	DLB_ST_INVALID_CQ_DEPTH,
+	DLB_ST_INVALID_HIST_LIST_DEPTH,
+	DLB_ST_INVALID_DIR_QUEUE_ID,
 };
 
 struct dlb_cmd_response {
@@ -160,10 +166,32 @@ struct dlb_get_num_resources_args {
 	__u32 num_dir_credits;
 };
 
+enum dlb_cq_poll_modes {
+	DLB_CQ_POLL_MODE_STD,
+	DLB_CQ_POLL_MODE_SPARSE,
+
+	/* NUM_DLB_CQ_POLL_MODE must be last */
+	NUM_DLB_CQ_POLL_MODE,
+};
+
+/*
+ * DLB_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: CQ poll mode (see enum dlb_cq_poll_modes).
+ */
+struct dlb_query_cq_poll_mode_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+};
+
 enum dlb_user_interface_commands {
 	DLB_CMD_GET_DEVICE_VERSION,
 	DLB_CMD_CREATE_SCHED_DOMAIN,
 	DLB_CMD_GET_NUM_RESOURCES,
+	DLB_CMD_QUERY_CQ_POLL_MODE,
 
 	/* NUM_DLB_CMD must be last */
 	NUM_DLB_CMD,
@@ -285,16 +313,81 @@ struct dlb_get_dir_queue_depth_args {
 	__u32 padding0;
 };
 
+/*
+ * DLB_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: port ID.
+ *
+ * Input parameters:
+ * @cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * @cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * @num_hist_list_entries: Number of history list entries. This must be
+ *	greater than or equal cq_depth.
+ * @cos_id: class-of-service to allocate this port from. Must be between 0 and
+ *	3, inclusive.
+ * @cos_strict: If set, return an error if there are no available ports in the
+ *	requested class-of-service. Else, allocate the port from a different
+ *	class-of-service if the requested class has no available ports.
+ */
+
+struct dlb_create_ldb_port_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__u16 cq_history_list_size;
+	__u8 cos_id;
+	__u8 cos_strict;
+};
+
+/*
+ * DLB_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: Port ID.
+ *
+ * Input parameters:
+ * @cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ *	1024, inclusive.
+ * @cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ *	the CQ interrupt won't fire until there are N or more outstanding CQ
+ *	tokens.
+ * @qid: Queue ID. If the corresponding directed queue is already created,
+ *	specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ *	that the port is being created before the queue.
+ */
+struct dlb_create_dir_port_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u16 cq_depth;
+	__u16 cq_depth_threshold;
+	__s32 queue_id;
+};
+
 enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,
 	DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,
 	DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,
 	DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
+	DLB_DOMAIN_CMD_CREATE_LDB_PORT,
+	DLB_DOMAIN_CMD_CREATE_DIR_PORT,
 
 	/* NUM_DLB_DOMAIN_CMD must be last */
 	NUM_DLB_DOMAIN_CMD,
 };
 
+#define DLB_CQ_SIZE 65536
+
 /********************/
 /* dlb ioctl codes */
 /********************/
@@ -313,6 +406,10 @@ enum dlb_domain_user_interface_commands {
 		_IOR(DLB_IOC_MAGIC,				\
 		     DLB_CMD_GET_NUM_RESOURCES,			\
 		     struct dlb_get_num_resources_args)
+#define DLB_IOC_QUERY_CQ_POLL_MODE				\
+		_IOR(DLB_IOC_MAGIC,				\
+		     DLB_CMD_QUERY_CQ_POLL_MODE,		\
+		     struct dlb_query_cq_poll_mode_args)
 #define DLB_IOC_CREATE_LDB_QUEUE				\
 		_IOWR(DLB_IOC_MAGIC,				\
 		      DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,		\
@@ -329,5 +426,13 @@ enum dlb_domain_user_interface_commands {
 		_IOWR(DLB_IOC_MAGIC,				\
 		      DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,	\
 		      struct dlb_get_dir_queue_depth_args)
+#define DLB_IOC_CREATE_LDB_PORT					\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_CREATE_LDB_PORT,		\
+		      struct dlb_create_ldb_port_args)
+#define DLB_IOC_CREATE_DIR_PORT					\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_CREATE_DIR_PORT,		\
+		      struct dlb_create_dir_port_args)
 
 #endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 12/20] dlb: add register operations for port management
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (10 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 11/20] dlb: add ioctl to configure ports and query poll mode Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 13/20] dlb: add port mmap support Mike Ximing Chen
                   ` (8 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the low-level code for configuring a new port, programming the
device-wide poll mode setting, and resetting a port.

The low-level port configuration functions program the device based on the
user-supplied ioctl arguments. These arguments are first verified, e.g.
to ensure that the port's CQ base address is properly cache-line aligned.

During domain reset, each port is drained until its inflight count and
owed-token count reaches 0, reflecting an empty CQ. Once the ports are
drained, the domain reset operation disables them from being candidates
for future scheduling decisions -- until they are re-assigned to a new
scheduling domain in the future and re-enabled.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_resource.c | 448 +++++++++++++++++++++++++++++++-
 1 file changed, 443 insertions(+), 5 deletions(-)

diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index ac6c5889c435..822c1f4f7849 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -890,7 +890,7 @@ static void dlb_configure_ldb_queue(struct dlb_hw *hw,
 	DLB_CSR_WR(hw, LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id), reg);
 
 	level = args->lock_id_comp_level;
-	if (level >= 64 && level <= 4096)
+	if (level >= 64 && level <= 4096 && is_power_of_2(level))
 		BITS_SET(reg, ilog2(level) - 5, AQED_QID_HID_WIDTH_COMPRESS_CODE);
 	else
 		reg = 0;
@@ -1001,12 +1001,10 @@ static void dlb_configure_dir_queue(struct dlb_hw *hw,
 static bool
 dlb_cq_depth_is_valid(u32 depth)
 {
-	u32 n = ilog2(depth);
-
 	/* Valid values for depth are
 	 * 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, and 1024.
 	 */
-	if (depth > 1024 || ((1U << n) != depth))
+	if (!is_power_of_2(depth) || depth > 1024)
 		return false;
 
 	return true;
@@ -1347,6 +1345,320 @@ static void dlb_dir_port_cq_disable(struct dlb_hw *hw,
 	dlb_flush_csr(hw);
 }
 
+static void dlb_ldb_port_configure_pp(struct dlb_hw *hw,
+				      struct dlb_hw_domain *domain,
+				      struct dlb_ldb_port *port)
+{
+	u32 reg = 0;
+
+	BITS_SET(reg, domain->id.phys_id, SYS_LDB_PP2VAS_VAS);
+	DLB_CSR_WR(hw, SYS_LDB_PP2VAS(port->id.phys_id), reg);
+
+	reg = 0;
+	reg |= SYS_LDB_PP_V_PP_V;
+	DLB_CSR_WR(hw, SYS_LDB_PP_V(port->id.phys_id), reg);
+}
+
+static int dlb_ldb_port_configure_cq(struct dlb_hw *hw,
+				     struct dlb_hw_domain *domain,
+				     struct dlb_ldb_port *port,
+				     uintptr_t cq_dma_base,
+				     struct dlb_create_ldb_port_args *args,
+				     bool vdev_req, unsigned int vdev_id)
+{
+	u32 hl_base = 0;
+	u32 reg = 0;
+	u32 ds = 0;
+	u32 n;
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	BITS_SET(reg, cq_dma_base >> 6, SYS_LDB_CQ_ADDR_L_ADDR_L);
+	DLB_CSR_WR(hw, SYS_LDB_CQ_ADDR_L(port->id.phys_id), reg);
+
+	reg = cq_dma_base >> 32;
+	DLB_CSR_WR(hw, SYS_LDB_CQ_ADDR_U(port->id.phys_id), reg);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	reg = 0;
+	BITS_SET(reg, vdev_id, SYS_LDB_CQ2VF_PF_RO_VF);
+	BITS_SET(reg, (u32)(!vdev_req), SYS_LDB_CQ2VF_PF_RO_IS_PF);
+	reg |= SYS_LDB_CQ2VF_PF_RO_RO;
+
+	DLB_CSR_WR(hw, SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), reg);
+
+	if (!dlb_cq_depth_is_valid(args->cq_depth)) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: invalid CQ depth\n",
+			   __func__, __LINE__);
+		return -EINVAL;
+	}
+
+	if (args->cq_depth <= 8) {
+		ds = 1;
+	} else {
+		n = ilog2(args->cq_depth);
+		ds = n - 2;
+	}
+
+	reg = 0;
+	BITS_SET(reg, ds, CHP_LDB_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT);
+	DLB_CSR_WR(hw, CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id), reg);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		reg = 0;
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		BITS_SET(reg, port->init_tkn_cnt, LSP_CQ_LDB_TKN_CNT_TOKEN_COUNT);
+		DLB_CSR_WR(hw, LSP_CQ_LDB_TKN_CNT(port->id.phys_id), reg);
+	} else {
+		DLB_CSR_WR(hw,
+			   LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+			   LSP_CQ_LDB_TKN_CNT_RST);
+	}
+
+	reg = 0;
+	BITS_SET(reg, ds, LSP_CQ_LDB_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT);
+	DLB_CSR_WR(hw, LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id), reg);
+
+	/* Reset the CQ write pointer */
+	DLB_CSR_WR(hw,
+		   CHP_LDB_CQ_WPTR(port->id.phys_id),
+		   CHP_LDB_CQ_WPTR_RST);
+
+	reg = 0;
+	BITS_SET(reg, port->hist_list_entry_limit - 1, CHP_HIST_LIST_LIM_LIMIT);
+	DLB_CSR_WR(hw, CHP_HIST_LIST_LIM(port->id.phys_id), reg);
+
+	BITS_SET(hl_base, port->hist_list_entry_base, CHP_HIST_LIST_BASE_BASE);
+	DLB_CSR_WR(hw, CHP_HIST_LIST_BASE(port->id.phys_id), hl_base);
+
+	/*
+	 * The inflight limit sets a cap on the number of QEs for which this CQ
+	 * can owe completions at one time.
+	 */
+	reg = 0;
+	BITS_SET(reg, args->cq_history_list_size, LSP_CQ_LDB_INFL_LIM_LIMIT);
+	DLB_CSR_WR(hw, LSP_CQ_LDB_INFL_LIM(port->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, BITS_GET(hl_base, CHP_HIST_LIST_BASE_BASE),
+		 CHP_HIST_LIST_PUSH_PTR_PUSH_PTR);
+	DLB_CSR_WR(hw, CHP_HIST_LIST_PUSH_PTR(port->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, BITS_GET(hl_base, CHP_HIST_LIST_BASE_BASE),
+		 CHP_HIST_LIST_POP_PTR_POP_PTR);
+	DLB_CSR_WR(hw, CHP_HIST_LIST_POP_PTR(port->id.phys_id), reg);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+
+	reg = 0;
+	DLB_CSR_WR(hw, SYS_LDB_CQ_AT(port->id.phys_id), reg);
+	DLB_CSR_WR(hw, SYS_LDB_CQ_PASID(port->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, domain->id.phys_id, CHP_LDB_CQ2VAS_CQ2VAS);
+	DLB_CSR_WR(hw, CHP_LDB_CQ2VAS(port->id.phys_id), reg);
+
+	/* Disable the port's QID mappings */
+	reg = 0;
+	DLB_CSR_WR(hw, LSP_CQ2PRIOV(port->id.phys_id), reg);
+
+	return 0;
+}
+
+static int dlb_configure_ldb_port(struct dlb_hw *hw, struct dlb_hw_domain *domain,
+				  struct dlb_ldb_port *port,
+				  uintptr_t cq_dma_base,
+				  struct dlb_create_ldb_port_args *args,
+				  bool vdev_req, unsigned int vdev_id)
+{
+	int ret, i;
+
+	port->hist_list_entry_base = domain->hist_list_entry_base +
+				     domain->hist_list_entry_offset;
+	port->hist_list_entry_limit = port->hist_list_entry_base +
+				      args->cq_history_list_size;
+
+	domain->hist_list_entry_offset += args->cq_history_list_size;
+	domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+	ret = dlb_ldb_port_configure_cq(hw,
+					domain,
+					port,
+					cq_dma_base,
+					args,
+					vdev_req,
+					vdev_id);
+	if (ret)
+		return ret;
+
+	dlb_ldb_port_configure_pp(hw, domain, port);
+
+	dlb_ldb_port_cq_enable(hw, port);
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+		port->qid_map[i].state = DLB_QUEUE_UNMAPPED;
+	port->num_mappings = 0;
+
+	port->enabled = true;
+
+	port->configured = true;
+
+	return 0;
+}
+
+static void dlb_dir_port_configure_pp(struct dlb_hw *hw,
+				      struct dlb_hw_domain *domain,
+				      struct dlb_dir_pq_pair *port)
+{
+	u32 reg = 0;
+
+	BITS_SET(reg, domain->id.phys_id, SYS_DIR_PP2VAS_VAS);
+	DLB_CSR_WR(hw, SYS_DIR_PP2VAS(port->id.phys_id), reg);
+
+	reg = 0;
+	reg |= SYS_DIR_PP_V_PP_V;
+	DLB_CSR_WR(hw, SYS_DIR_PP_V(port->id.phys_id), reg);
+}
+
+static int dlb_dir_port_configure_cq(struct dlb_hw *hw,
+				     struct dlb_hw_domain *domain,
+				     struct dlb_dir_pq_pair *port,
+				     uintptr_t cq_dma_base,
+				     struct dlb_create_dir_port_args *args,
+				     bool vdev_req, unsigned int vdev_id)
+{
+	u32 reg = 0;
+	u32 ds = 0;
+	u32 n;
+
+	/* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+	BITS_SET(reg, cq_dma_base >> 6, SYS_DIR_CQ_ADDR_L_ADDR_L);
+	DLB_CSR_WR(hw, SYS_DIR_CQ_ADDR_L(port->id.phys_id), reg);
+
+	reg = cq_dma_base >> 32;
+	DLB_CSR_WR(hw, SYS_DIR_CQ_ADDR_U(port->id.phys_id), reg);
+
+	/*
+	 * 'ro' == relaxed ordering. This setting allows DLB to write
+	 * cache lines out-of-order (but QEs within a cache line are always
+	 * updated in-order).
+	 */
+	reg = 0;
+	BITS_SET(reg, vdev_id, SYS_DIR_CQ2VF_PF_RO_VF);
+	BITS_SET(reg, (u32)(!vdev_req), SYS_DIR_CQ2VF_PF_RO_IS_PF);
+	reg |= SYS_DIR_CQ2VF_PF_RO_RO;
+
+	DLB_CSR_WR(hw, SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), reg);
+
+	if (!dlb_cq_depth_is_valid(args->cq_depth)) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: invalid CQ depth\n",
+			   __func__, __LINE__);
+		return -EINVAL;
+	}
+
+	if (args->cq_depth <= 8) {
+		ds = 1;
+	} else {
+		n = ilog2(args->cq_depth);
+		ds = n - 2;
+	}
+
+	reg = 0;
+	BITS_SET(reg, ds, CHP_DIR_CQ_TKN_DEPTH_SEL_TOKEN_DEPTH_SELECT);
+	DLB_CSR_WR(hw, CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id), reg);
+
+	/*
+	 * To support CQs with depth less than 8, program the token count
+	 * register with a non-zero initial value. Operations such as domain
+	 * reset must take this initial value into account when quiescing the
+	 * CQ.
+	 */
+	port->init_tkn_cnt = 0;
+
+	if (args->cq_depth < 8) {
+		reg = 0;
+		port->init_tkn_cnt = 8 - args->cq_depth;
+
+		BITS_SET(reg, port->init_tkn_cnt, LSP_CQ_DIR_TKN_CNT_COUNT);
+		DLB_CSR_WR(hw, LSP_CQ_DIR_TKN_CNT(port->id.phys_id), reg);
+	} else {
+		DLB_CSR_WR(hw,
+			   LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+			   LSP_CQ_DIR_TKN_CNT_RST);
+	}
+
+	reg = 0;
+	BITS_SET(reg, ds, LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_TOKEN_DEPTH_SELECT);
+	DLB_CSR_WR(hw, LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id), reg);
+
+	/* Reset the CQ write pointer */
+	DLB_CSR_WR(hw,
+		   CHP_DIR_CQ_WPTR(port->id.phys_id),
+		   CHP_DIR_CQ_WPTR_RST);
+
+	/* Virtualize the PPID */
+	reg = 0;
+	DLB_CSR_WR(hw, SYS_DIR_CQ_FMT(port->id.phys_id), reg);
+
+	/*
+	 * Address translation (AT) settings: 0: untranslated, 2: translated
+	 * (see ATS spec regarding Address Type field for more details)
+	 */
+	reg = 0;
+	DLB_CSR_WR(hw, SYS_DIR_CQ_AT(port->id.phys_id), reg);
+
+	DLB_CSR_WR(hw, SYS_DIR_CQ_PASID(port->id.phys_id), reg);
+
+	reg = 0;
+	BITS_SET(reg, domain->id.phys_id, CHP_DIR_CQ2VAS_CQ2VAS);
+	DLB_CSR_WR(hw, CHP_DIR_CQ2VAS(port->id.phys_id), reg);
+
+	return 0;
+}
+
+static int dlb_configure_dir_port(struct dlb_hw *hw, struct dlb_hw_domain *domain,
+				  struct dlb_dir_pq_pair *port,
+				  uintptr_t cq_dma_base,
+				  struct dlb_create_dir_port_args *args,
+				  bool vdev_req, unsigned int vdev_id)
+{
+	int ret;
+
+	ret = dlb_dir_port_configure_cq(hw, domain, port, cq_dma_base,
+					args, vdev_req, vdev_id);
+
+	if (ret)
+		return ret;
+
+	dlb_dir_port_configure_pp(hw, domain, port);
+
+	dlb_dir_port_cq_enable(hw, port);
+
+	port->enabled = true;
+
+	port->port_configured = true;
+
+	return 0;
+}
+
 static void
 dlb_log_create_sched_domain_args(struct dlb_hw *hw,
 				 struct dlb_create_sched_domain_args *args,
@@ -1693,6 +2005,11 @@ int dlb_hw_create_ldb_port(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	ret = dlb_configure_ldb_port(hw, domain, port, cq_dma_base,
+				     args, vdev_req, vdev_id);
+	if (ret)
+		return ret;
+
 	/*
 	 * Configuration succeeded, so move the resource from the 'avail' to
 	 * the 'used' list.
@@ -1775,6 +2092,11 @@ int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	ret = dlb_configure_dir_port(hw, domain, port, cq_dma_base,
+				     args, vdev_req, vdev_id);
+	if (ret)
+		return ret;
+
 	/*
 	 * Configuration succeeded, so move the resource from the 'avail' to
 	 * the 'used' list (if it's not already there).
@@ -1877,6 +2199,33 @@ static void dlb_drain_ldb_cq(struct dlb_hw *hw, struct dlb_ldb_port *port)
 	}
 }
 
+static int dlb_domain_wait_for_ldb_cqs_to_empty(struct dlb_hw *hw,
+						struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			int j;
+
+			for (j = 0; j < DLB_MAX_CQ_COMP_CHECK_LOOPS; j++) {
+				if (dlb_ldb_cq_inflight_count(hw, port) == 0)
+					break;
+			}
+
+			if (j == DLB_MAX_CQ_COMP_CHECK_LOOPS) {
+				DLB_HW_ERR(hw,
+					   "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+					   __func__, port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	return 0;
+}
+
 static int dlb_domain_reset_software_state(struct dlb_hw *hw,
 					   struct dlb_hw_domain *domain)
 {
@@ -2564,7 +2913,10 @@ static u32 dlb_dir_cq_token_count(struct dlb_hw *hw,
 static int dlb_domain_verify_reset_success(struct dlb_hw *hw,
 					   struct dlb_hw_domain *domain)
 {
+	struct dlb_dir_pq_pair *dir_port;
+	struct dlb_ldb_port *ldb_port;
 	struct dlb_ldb_queue *queue;
+	int i;
 
 	/*
 	 * Confirm that all the domain's queue's inflight counts and AQED
@@ -2579,6 +2931,35 @@ static int dlb_domain_verify_reset_success(struct dlb_hw *hw,
 		}
 	}
 
+	/* Confirm that all the domain's CQs inflight and token counts are 0. */
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(ldb_port, &domain->used_ldb_ports[i], domain_list) {
+			if (dlb_ldb_cq_inflight_count(hw, ldb_port) ||
+			    dlb_ldb_cq_token_count(hw, ldb_port)) {
+				DLB_HW_ERR(hw,
+					   "[%s()] Internal error: failed to empty ldb port %d\n",
+					   __func__, ldb_port->id.phys_id);
+				return -EFAULT;
+			}
+		}
+	}
+
+	list_for_each_entry(dir_port, &domain->used_dir_pq_pairs, domain_list) {
+		if (!dlb_dir_queue_is_empty(hw, dir_port)) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: failed to empty dir queue %d\n",
+				   __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+
+		if (dlb_dir_cq_token_count(hw, dir_port)) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: failed to empty dir port %d\n",
+				   __func__, dir_port->id.phys_id);
+			return -EFAULT;
+		}
+	}
+
 	return 0;
 }
 
@@ -2796,6 +3177,51 @@ static int dlb_domain_drain_dir_queues(struct dlb_hw *hw,
 	return 0;
 }
 
+static void
+dlb_domain_disable_dir_producer_ports(struct dlb_hw *hw,
+				      struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *port;
+	u32 pp_v = 0;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+		DLB_CSR_WR(hw, SYS_DIR_PP_V(port->id.phys_id), pp_v);
+	}
+}
+
+static void
+dlb_domain_disable_ldb_producer_ports(struct dlb_hw *hw,
+				      struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	u32 pp_v = 0;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			DLB_CSR_WR(hw,
+				   SYS_LDB_PP_V(port->id.phys_id),
+				   pp_v);
+		}
+	}
+}
+
+static void dlb_domain_disable_ldb_seq_checks(struct dlb_hw *hw,
+					      struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	u32 chk_en = 0;
+	int i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			DLB_CSR_WR(hw,
+				   CHP_SN_CHK_ENBL(port->id.phys_id),
+				   chk_en);
+		}
+	}
+}
+
 static void
 dlb_domain_disable_ldb_queue_write_perms(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
@@ -2922,6 +3348,9 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 
 	dlb_domain_disable_ldb_queue_write_perms(hw, domain);
 
+	/* Turn off completion tracking on all the domain's PPs. */
+	dlb_domain_disable_ldb_seq_checks(hw, domain);
+
 	/*
 	 * Disable the LDB CQs and drain them in order to complete the map and
 	 * unmap procedures, which require zero CQ inflights and zero QID
@@ -2931,6 +3360,10 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 
 	dlb_domain_drain_ldb_cqs(hw, domain, false);
 
+	ret = dlb_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+	if (ret)
+		return ret;
+
 	/* Re-enable the CQs in order to drain the mapped queues. */
 	dlb_domain_enable_ldb_cqs(hw, domain);
 
@@ -2946,6 +3379,11 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	/* Done draining DIR QEs, so disable the CQs. */
 	dlb_domain_disable_dir_cqs(hw, domain);
 
+	/* Disable PPs */
+	dlb_domain_disable_dir_producer_ports(hw, domain);
+
+	dlb_domain_disable_ldb_producer_ports(hw, domain);
+
 	ret = dlb_domain_verify_reset_success(hw, domain);
 	if (ret)
 		return ret;
@@ -3039,7 +3477,7 @@ void dlb_clr_pmcsr_disable(struct dlb_hw *hw)
 
 /**
  * dlb_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
- *      ports.
+ *	ports.
  * @hw: dlb_hw handle for a particular device.
  *
  * This function must be called prior to configuring scheduling domains.
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 13/20] dlb: add port mmap support
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (11 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 12/20] dlb: add register operations for port management Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 14/20] dlb: add start domain ioctl Mike Ximing Chen
                   ` (7 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Once a port is created, the application can mmap the corresponding DMA
memory and MMIO into user-space. This allows user-space applications to
do (performance-sensitive) enqueue and dequeue independent of the kernel
driver.

The mmap callback is only available through special port files: a producer
port (PP) file and a consumer queue (CQ) file. User-space gets an fd for
these files by calling a new ioctl, DLB_DOMAIN_CMD_GET_{LDB,
DIR}_PORT_{PP, CQ}_FD, and passing in a port ID. If the ioctl succeeds, the
returned fd can be used to mmap that port's PP/CQ.

Device reset requires first unmapping all user-space mappings, to prevent
applications from interfering with the reset operation. To this end, the
driver uses a single inode -- allocated when the first PP/CQ file is
created, and freed when the last such file is closed -- and attaches all
port files to this common inode, as done elsewhere in Linux (e.g. cxl,
dax).

Allocating this inode requires creating a pseudo-filesystem. The driver
initializes this FS when the inode is allocated, and frees the FS after the
inode is freed.

The driver doesn't use anon_inode_getfd() for these port mmap files because
the anon inode layer uses a single inode that is shared with other kernel
components -- calling unmap_mapping_range() on that shared inode would
likely break the kernel.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/Makefile       |   1 +
 drivers/misc/dlb/dlb_file.c     | 149 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_hw_types.h |   4 +-
 drivers/misc/dlb/dlb_ioctl.c    | 143 ++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_main.c     | 118 +++++++++++++++++++++++++
 drivers/misc/dlb/dlb_main.h     |  24 +++++
 drivers/misc/dlb/dlb_pf_ops.c   |  18 ++++
 drivers/misc/dlb/dlb_resource.c | 131 ++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |   6 ++
 include/uapi/linux/dlb.h        |  59 +++++++++++++
 10 files changed, 651 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/dlb/dlb_file.c

diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index aaafb3086d8d..66676222ca07 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -8,3 +8,4 @@ obj-$(CONFIG_INTEL_DLB) := dlb.o
 
 dlb-objs := dlb_main.o
 dlb-objs += dlb_pf_ops.o dlb_resource.o dlb_ioctl.o
+dlb-objs += dlb_file.o
diff --git a/drivers/misc/dlb/dlb_file.c b/drivers/misc/dlb/dlb_file.c
new file mode 100644
index 000000000000..310b86735353
--- /dev/null
+++ b/drivers/misc/dlb/dlb_file.c
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+#include <linux/module.h>
+#include <linux/mount.h>
+#include <linux/pseudo_fs.h>
+
+#include "dlb_main.h"
+
+/*
+ * dlb tracks its memory mappings so it can revoke them when an FLR is
+ * requested and user-space cannot be allowed to access the device. To achieve
+ * that, the driver creates a single inode through which all driver-created
+ * files can share a struct address_space, and unmaps the inode's address space
+ * during the reset preparation phase. Since the anon inode layer shares its
+ * inode with multiple kernel components, we cannot use that here.
+ *
+ * Doing so requires a custom pseudo-filesystem to allocate the inode. The FS
+ * and the inode are allocated on demand when a file is created, and both are
+ * freed when the last such file is closed.
+ *
+ * This is inspired by other drivers (cxl, dax, mem) and the anon inode layer.
+ */
+static int dlb_fs_cnt;
+static struct vfsmount *dlb_vfs_mount;
+
+#define DLBFS_MAGIC 0x444C4232 /* ASCII for DLB */
+static int dlb_init_fs_context(struct fs_context *fc)
+{
+	return init_pseudo(fc, DLBFS_MAGIC) ? 0 : -ENOMEM;
+}
+
+static struct file_system_type dlb_fs_type = {
+	.name	 = "dlb",
+	.owner	 = THIS_MODULE,
+	.init_fs_context = dlb_init_fs_context,
+	.kill_sb = kill_anon_super,
+};
+
+/* Allocate an anonymous inode. Must hold the resource mutex while calling. */
+static struct inode *dlb_alloc_inode(struct dlb *dlb)
+{
+	struct inode *inode;
+	int ret;
+
+	/* Increment the pseudo-FS's refcnt and (if not already) mount it. */
+	ret = simple_pin_fs(&dlb_fs_type, &dlb_vfs_mount, &dlb_fs_cnt);
+	if (ret < 0) {
+		dev_err(dlb->dev,
+			"[%s()] Cannot mount pseudo filesystem: %d\n",
+			__func__, ret);
+		return ERR_PTR(ret);
+	}
+
+	dlb->inode_cnt++;
+
+	if (dlb->inode_cnt > 1) {
+		/*
+		 * Return the previously allocated inode. In this case, there
+		 * is guaranteed >= 1 reference and so ihold() is safe to call.
+		 */
+		ihold(dlb->inode);
+		return dlb->inode;
+	}
+
+	inode = alloc_anon_inode(dlb_vfs_mount->mnt_sb);
+	if (IS_ERR(inode)) {
+		dev_err(dlb->dev,
+			"[%s()] Cannot allocate inode: %ld\n",
+			__func__, PTR_ERR(inode));
+		dlb->inode_cnt = 0;
+		simple_release_fs(&dlb_vfs_mount, &dlb_fs_cnt);
+	}
+
+	dlb->inode = inode;
+
+	return inode;
+}
+
+/*
+ * Decrement the inode reference count and release the FS. Intended for
+ * unwinding dlb_alloc_inode(). Must hold the resource mutex while calling.
+ */
+static void dlb_free_inode(struct inode *inode)
+{
+	iput(inode);
+	simple_release_fs(&dlb_vfs_mount, &dlb_fs_cnt);
+}
+
+/*
+ * Release the FS. Intended for use in a file_operations release callback,
+ * which decrements the inode reference count separately. Must hold the
+ * resource mutex while calling.
+ */
+void dlb_release_fs(struct dlb *dlb)
+{
+	mutex_lock(&dlb_driver_mutex);
+
+	simple_release_fs(&dlb_vfs_mount, &dlb_fs_cnt);
+
+	dlb->inode_cnt--;
+
+	/* When the fs refcnt reaches zero, the inode has been freed */
+	if (dlb->inode_cnt == 0)
+		dlb->inode = NULL;
+
+	mutex_unlock(&dlb_driver_mutex);
+}
+
+/*
+ * Allocate a file with the requested flags, file operations, and name that
+ * uses the device's shared inode. Must hold the resource mutex while calling.
+ *
+ * Caller must separately allocate an fd and install the file in that fd.
+ */
+struct file *dlb_getfile(struct dlb *dlb,
+			 int flags,
+			 const struct file_operations *fops,
+			 const char *name)
+{
+	struct inode *inode;
+	struct file *f;
+
+	if (!try_module_get(THIS_MODULE))
+		return ERR_PTR(-ENOENT);
+
+	mutex_lock(&dlb_driver_mutex);
+
+	inode = dlb_alloc_inode(dlb);
+	if (IS_ERR(inode)) {
+		mutex_unlock(&dlb_driver_mutex);
+		module_put(THIS_MODULE);
+		return ERR_CAST(inode);
+	}
+
+	f = alloc_file_pseudo(inode, dlb_vfs_mount, name, flags, fops);
+	if (IS_ERR(f)) {
+		dlb_free_inode(inode);
+		mutex_unlock(&dlb_driver_mutex);
+		module_put(THIS_MODULE);
+		return f;
+	}
+
+	mutex_unlock(&dlb_driver_mutex);
+
+	return f;
+}
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index c7827defa66a..b892a9dd172c 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -66,12 +66,12 @@
 #define DLB_LDB_PP_STRIDE     0x1000
 #define DLB_LDB_PP_BOUND      (DLB_LDB_PP_BASE + \
 				DLB_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS)
-#define DLB_LDB_PP_OFFS(id)   (DLB_LDB_PP_BASE + (id) * DLB_PP_SIZE)
+#define DLB_LDB_PP_OFFSET(id) (DLB_LDB_PP_BASE + (id) * DLB_PP_SIZE)
 #define DLB_DIR_PP_BASE       0x2000000
 #define DLB_DIR_PP_STRIDE     0x1000
 #define DLB_DIR_PP_BOUND      (DLB_DIR_PP_BASE + \
 				DLB_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS)
-#define DLB_DIR_PP_OFFS(id)   (DLB_DIR_PP_BASE + (id) * DLB_PP_SIZE)
+#define DLB_DIR_PP_OFFSET(id) (DLB_DIR_PP_BASE + (id) * DLB_PP_SIZE)
 
 struct dlb_resource_id {
 	u32 phys_id;
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 84bf833631bd..6a311b969643 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -161,6 +161,141 @@ static int dlb_domain_ioctl_create_dir_port(struct dlb *dlb,
 	return ret;
 }
 
+static int dlb_create_port_fd(struct dlb *dlb, const char *prefix, u32 id,
+			      const struct file_operations *fops,
+			      int *fd, struct file **f)
+{
+	char *name;
+	int ret;
+
+	ret = get_unused_fd_flags(O_RDWR);
+	if (ret < 0)
+		return ret;
+
+	*fd = ret;
+
+	name = kasprintf(GFP_KERNEL, "%s:%d", prefix, id);
+	if (!name) {
+		put_unused_fd(*fd);
+		return -ENOMEM;
+	}
+
+	*f = dlb_getfile(dlb, O_RDWR | O_CLOEXEC, fops, name);
+
+	kfree(name);
+
+	if (IS_ERR(*f)) {
+		put_unused_fd(*fd);
+		return PTR_ERR(*f);
+	}
+
+	return 0;
+}
+
+static int dlb_domain_get_port_fd(struct dlb *dlb, struct dlb_domain *domain,
+				  unsigned long user_arg, const char *name,
+				  const struct file_operations *fops,
+				  bool is_ldb)
+{
+	struct dlb_get_port_fd_args __user *uarg;
+	struct dlb_cmd_response response = {0};
+	struct dlb_get_port_fd_args arg;
+	struct dlb_port *port;
+	struct file *file;
+	int ret, fd;
+
+	uarg = (void __user *)user_arg;
+	if (copy_from_user(&arg, uarg, sizeof(arg)))
+		return -EFAULT;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	if ((is_ldb &&
+	     dlb->ops->ldb_port_owned_by_domain(&dlb->hw, domain->id,
+						arg.port_id) != 1)) {
+		response.status = DLB_ST_INVALID_PORT_ID;
+		ret = -EINVAL;
+		goto end;
+	}
+
+	if (!is_ldb &&
+	    dlb->ops->dir_port_owned_by_domain(&dlb->hw, domain->id,
+					       arg.port_id) != 1) {
+		response.status = DLB_ST_INVALID_PORT_ID;
+		ret = -EINVAL;
+		goto end;
+	}
+
+	port = (is_ldb) ? &dlb->ldb_port[arg.port_id] :
+			  &dlb->dir_port[arg.port_id];
+
+	if (!port->valid) {
+		response.status = DLB_ST_INVALID_PORT_ID;
+		ret = -EINVAL;
+		goto end;
+	}
+
+	ret = dlb_create_port_fd(dlb, name, arg.port_id, fops, &fd, &file);
+	if (ret < 0)
+		goto end;
+
+	file->private_data = port;
+
+	response.id = fd;
+
+end:
+	BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+	if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+		ret = -EFAULT;
+
+	/*
+	 * Save fd_install() until after the last point of failure. The domain
+	 * refcnt is decremented in the close callback.
+	 */
+	if (ret == 0) {
+		kref_get(&domain->refcnt);
+
+		fd_install(fd, file);
+	}
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	return ret;
+}
+
+static int dlb_domain_ioctl_get_ldb_port_pp_fd(struct dlb *dlb,
+					       struct dlb_domain *domain,
+					       unsigned long user_arg)
+{
+	return dlb_domain_get_port_fd(dlb, domain, user_arg,
+				      "dlb_ldb_pp:", &dlb_pp_fops, true);
+}
+
+static int dlb_domain_ioctl_get_ldb_port_cq_fd(struct dlb *dlb,
+					       struct dlb_domain *domain,
+					       unsigned long user_arg)
+{
+	return dlb_domain_get_port_fd(dlb, domain, user_arg,
+				      "dlb_ldb_cq:", &dlb_cq_fops, true);
+}
+
+static int dlb_domain_ioctl_get_dir_port_pp_fd(struct dlb *dlb,
+					       struct dlb_domain *domain,
+					       unsigned long user_arg)
+{
+	return dlb_domain_get_port_fd(dlb, domain, user_arg,
+				      "dlb_dir_pp:", &dlb_pp_fops, false);
+}
+
+static int dlb_domain_ioctl_get_dir_port_cq_fd(struct dlb *dlb,
+					       struct dlb_domain *domain,
+					       unsigned long user_arg)
+{
+	return dlb_domain_get_port_fd(dlb, domain, user_arg,
+				      "dlb_dir_cq:", &dlb_cq_fops, false);
+}
+
 long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 {
 	struct dlb_domain *dom = f->private_data;
@@ -179,6 +314,14 @@ long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 		return dlb_domain_ioctl_create_ldb_port(dlb, dom, arg);
 	case DLB_IOC_CREATE_DIR_PORT:
 		return dlb_domain_ioctl_create_dir_port(dlb, dom, arg);
+	case DLB_IOC_GET_LDB_PORT_PP_FD:
+		return dlb_domain_ioctl_get_ldb_port_pp_fd(dlb, dom, arg);
+	case DLB_IOC_GET_LDB_PORT_CQ_FD:
+		return dlb_domain_ioctl_get_ldb_port_cq_fd(dlb, dom, arg);
+	case DLB_IOC_GET_DIR_PORT_PP_FD:
+		return dlb_domain_ioctl_get_dir_port_pp_fd(dlb, dom, arg);
+	case DLB_IOC_GET_DIR_PORT_CQ_FD:
+		return dlb_domain_ioctl_get_dir_port_cq_fd(dlb, dom, arg);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index e4c19714f1c4..69ab9b532ed4 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -17,6 +17,9 @@
 MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
 
+/* The driver mutex protects data structures that used by multiple devices. */
+DEFINE_MUTEX(dlb_driver_mutex);
+
 static struct class *dlb_class;
 static struct cdev dlb_cdev;
 static dev_t dlb_devt;
@@ -233,6 +236,121 @@ const struct file_operations dlb_domain_fops = {
 	.compat_ioctl = compat_ptr_ioctl,
 };
 
+static unsigned long dlb_get_pp_addr(struct dlb *dlb, struct dlb_port *port)
+{
+	unsigned long pgoff = dlb->hw.func_phys_addr;
+
+	if (port->is_ldb)
+		pgoff += DLB_LDB_PP_OFFSET(port->id);
+	else
+		pgoff += DLB_DIR_PP_OFFSET(port->id);
+
+	return pgoff;
+}
+
+static int dlb_pp_mmap(struct file *f, struct vm_area_struct *vma)
+{
+	struct dlb_port *port = f->private_data;
+	struct dlb_domain *domain = port->domain;
+	struct dlb *dlb = domain->dlb;
+	unsigned long pgoff;
+	pgprot_t pgprot;
+	int ret;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	if ((vma->vm_end - vma->vm_start) != DLB_PP_SIZE) {
+		ret = -EINVAL;
+		goto end;
+	}
+
+	pgprot = pgprot_noncached(vma->vm_page_prot);
+
+	pgoff = dlb_get_pp_addr(dlb, port);
+	ret = io_remap_pfn_range(vma,
+				 vma->vm_start,
+				 pgoff >> PAGE_SHIFT,
+				 vma->vm_end - vma->vm_start,
+				 pgprot);
+
+end:
+	mutex_unlock(&dlb->resource_mutex);
+
+	return ret;
+}
+
+static int dlb_cq_mmap(struct file *f, struct vm_area_struct *vma)
+{
+	struct dlb_port *port = f->private_data;
+	struct dlb_domain *domain = port->domain;
+	struct dlb *dlb = domain->dlb;
+	struct page *page;
+	int ret;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	if ((vma->vm_end - vma->vm_start) != DLB_CQ_SIZE) {
+		ret = -EINVAL;
+		goto end;
+	}
+
+	page = virt_to_page(port->cq_base);
+
+	ret = remap_pfn_range(vma,
+			      vma->vm_start,
+			      page_to_pfn(page),
+			      vma->vm_end - vma->vm_start,
+			      vma->vm_page_prot);
+end:
+	mutex_unlock(&dlb->resource_mutex);
+
+	return ret;
+}
+
+static void dlb_port_unmap(struct dlb *dlb, struct dlb_port *port)
+{
+	if (!port->cq_base) {
+		unmap_mapping_range(dlb->inode->i_mapping,
+				    (unsigned long)port->cq_base,
+				    DLB_CQ_SIZE, 1);
+	} else {
+		unmap_mapping_range(dlb->inode->i_mapping,
+				    dlb_get_pp_addr(dlb, port),
+				    DLB_PP_SIZE, 1);
+	}
+}
+
+static int dlb_port_close(struct inode *i, struct file *f)
+{
+	struct dlb_port *port = f->private_data;
+	struct dlb_domain *domain = port->domain;
+	struct dlb *dlb = domain->dlb;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	kref_put(&domain->refcnt, dlb_free_domain);
+
+	dlb_port_unmap(dlb, port);
+	/* Decrement the refcnt of the pseudo-FS used to allocate the inode */
+	dlb_release_fs(dlb);
+
+	mutex_unlock(&dlb->resource_mutex);
+
+	return 0;
+}
+
+const struct file_operations dlb_pp_fops = {
+	.owner   = THIS_MODULE,
+	.release = dlb_port_close,
+	.mmap    = dlb_pp_mmap,
+};
+
+const struct file_operations dlb_cq_fops = {
+	.owner   = THIS_MODULE,
+	.release = dlb_port_close,
+	.mmap    = dlb_cq_mmap,
+};
+
 /**********************************/
 /****** PCI driver callbacks ******/
 /**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 08dead13fb11..477974e1a178 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -11,6 +11,7 @@
 #include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/types.h>
+#include <linux/file.h>
 
 #include <uapi/linux/dlb.h>
 
@@ -27,6 +28,8 @@
 #define DLB_NUM_FUNCS_PER_DEVICE (1 + DLB_MAX_NUM_VDEVS)
 #define DLB_MAX_NUM_DEVICES	 (DLB_MAX_NUM_PFS * DLB_NUM_FUNCS_PER_DEVICE)
 
+extern struct mutex dlb_driver_mutex;
+
 enum dlb_device_type {
 	DLB_PF,
 };
@@ -63,6 +66,12 @@ struct dlb_device_ops {
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
 	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
+	int (*ldb_port_owned_by_domain)(struct dlb_hw *hw,
+					u32 domain_id,
+					u32 port_id);
+	int (*dir_port_owned_by_domain)(struct dlb_hw *hw,
+					u32 domain_id,
+					u32 port_id);
 	int (*get_ldb_queue_depth)(struct dlb_hw *hw,
 				   u32 domain_id,
 				   struct dlb_get_ldb_queue_depth_args *args,
@@ -78,6 +87,8 @@ struct dlb_device_ops {
 
 extern struct dlb_device_ops dlb_pf_ops;
 extern const struct file_operations dlb_domain_fops;
+extern const struct file_operations dlb_pp_fops;
+extern const struct file_operations dlb_cq_fops;
 
 struct dlb_port {
 	void *cq_base;
@@ -103,6 +114,11 @@ struct dlb {
 	struct file *f;
 	struct dlb_port ldb_port[DLB_MAX_NUM_LDB_PORTS];
 	struct dlb_port dir_port[DLB_MAX_NUM_DIR_PORTS];
+	/*
+	 * Anonymous inode used to share an address_space for all domain
+	 * device file mappings.
+	 */
+	struct inode *inode;
 	/*
 	 * The resource mutex serializes access to driver data structures and
 	 * hardware registers.
@@ -110,6 +126,7 @@ struct dlb {
 	struct mutex resource_mutex;
 	enum dlb_device_type type;
 	int id;
+	u32 inode_cnt;
 	dev_t dev_number;
 	u8 domain_reset_failed;
 };
@@ -118,6 +135,13 @@ struct dlb {
 long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
 long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
 
+/* Prototypes for dlb_file.c */
+void dlb_release_fs(struct dlb *dlb);
+struct file *dlb_getfile(struct dlb *dlb,
+			 int flags,
+			 const struct file_operations *fops,
+			 const char *name);
+
 int dlb_init_domain(struct dlb *dlb, u32 domain_id);
 void dlb_free_domain(struct kref *kref);
 
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index c2ce03114f8b..02a188aa5a60 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -201,6 +201,22 @@ dlb_pf_query_cq_poll_mode(struct dlb *dlb, struct dlb_cmd_response *user_resp)
 	return 0;
 }
 
+/**************************************/
+/****** Resource query callbacks ******/
+/**************************************/
+
+static int
+dlb_pf_ldb_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id)
+{
+	return dlb_ldb_port_owned_by_domain(hw, domain_id, port_id, false, 0);
+}
+
+static int
+dlb_pf_dir_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id)
+{
+	return dlb_dir_port_owned_by_domain(hw, domain_id, port_id, false, 0);
+}
+
 /********************************/
 /****** DLB PF Device Ops ******/
 /********************************/
@@ -218,6 +234,8 @@ struct dlb_device_ops dlb_pf_ops = {
 	.create_dir_port = dlb_pf_create_dir_port,
 	.get_num_resources = dlb_pf_get_num_resources,
 	.reset_domain = dlb_pf_reset_domain,
+	.ldb_port_owned_by_domain = dlb_pf_ldb_port_owned_by_domain,
+	.dir_port_owned_by_domain = dlb_pf_dir_port_owned_by_domain,
 	.get_ldb_queue_depth = dlb_pf_get_ldb_queue_depth,
 	.get_dir_queue_depth = dlb_pf_get_dir_queue_depth,
 	.init_hardware = dlb_pf_init_hardware,
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 822c1f4f7849..2659190527a7 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -236,6 +236,32 @@ static struct dlb_hw_domain *dlb_get_domain_from_id(struct dlb_hw *hw, u32 id,
 	return NULL;
 }
 
+static struct dlb_ldb_port *
+dlb_get_domain_ldb_port(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	if (id >= DLB_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+		}
+
+		list_for_each_entry(port, &domain->avail_ldb_ports[i], domain_list) {
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+		}
+	}
+
+	return NULL;
+}
+
 static struct dlb_dir_pq_pair *
 dlb_get_domain_used_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 {
@@ -253,6 +279,29 @@ dlb_get_domain_used_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 	return NULL;
 }
 
+static struct dlb_dir_pq_pair *
+dlb_get_domain_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
+{
+	struct dlb_dir_pq_pair *port;
+
+	if (id >= DLB_MAX_NUM_DIR_PORTS)
+		return NULL;
+
+	list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+	}
+
+	list_for_each_entry(port, &domain->avail_dir_pq_pairs, domain_list) {
+		if ((!vdev_req && port->id.phys_id == id) ||
+		    (vdev_req && port->id.virt_id == id))
+			return port;
+	}
+
+	return NULL;
+}
+
 static struct dlb_ldb_queue *
 dlb_get_domain_ldb_queue(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 {
@@ -3394,6 +3443,88 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	return dlb_domain_reset_software_state(hw, domain);
 }
 
+/**
+ * dlb_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb_ldb_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_port *port;
+
+	if (vdev_req && vdev_id >= DLB_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain || !domain->configured)
+		return -EINVAL;
+
+	port = dlb_get_domain_ldb_port(port_id, vdev_req, domain);
+
+	if (!port)
+		return -EINVAL;
+
+	return port->domain_id.phys_id == domain->id.phys_id;
+}
+
+/**
+ * dlb_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: port ID.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb_dir_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_dir_pq_pair *port;
+	struct dlb_hw_domain *domain;
+
+	if (vdev_req && vdev_id >= DLB_MAX_NUM_VDEVS)
+		return -EINVAL;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain || !domain->configured)
+		return -EINVAL;
+
+	port = dlb_get_domain_dir_pq(port_id, vdev_req, domain);
+
+	if (!port)
+		return -EINVAL;
+
+	return port->domain_id.phys_id == domain->id.phys_id;
+}
+
 /**
  * dlb_hw_get_num_resources() - query the PCI function's available resources
  * @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index bbe25a417cd4..8a3c37b6ab92 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -44,6 +44,12 @@ int dlb_hw_create_ldb_port(struct dlb_hw *hw, u32 domain_id,
 int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id);
 
+int dlb_ldb_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id,
+				 bool vdev_req, unsigned int vdev_id);
+
+int dlb_dir_port_owned_by_domain(struct dlb_hw *hw, u32 domain_id, u32 port_id,
+				 bool vdev_req, unsigned int vdev_id);
+
 int dlb_hw_get_num_resources(struct dlb_hw *hw,
 			     struct dlb_get_num_resources_args *arg,
 			     bool vdev_req, unsigned int vdev_id);
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 9578d8f1c03b..6b7eceecae8a 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -374,6 +374,40 @@ struct dlb_create_dir_port_args {
 	__s32 queue_id;
 };
 
+/*
+ * DLB_CMD_GET_LDB_PORT_PP_FD: Get file descriptor to mmap a load-balanced
+ *	port's producer port (PP).
+ * DLB_CMD_GET_LDB_PORT_CQ_FD: Get file descriptor to mmap a load-balanced
+ *	port's consumer queue (CQ).
+ *
+ *	The load-balanced port must have been previously created with the ioctl
+ *	DLB_CMD_CREATE_LDB_PORT. The fd is used to mmap the PP/CQ region.
+ *
+ * DLB_CMD_GET_DIR_PORT_PP_FD: Get file descriptor to mmap a directed port's
+ *	producer port (PP).
+ * DLB_CMD_GET_DIR_PORT_CQ_FD: Get file descriptor to mmap a directed port's
+ *	consumer queue (CQ).
+ *
+ *	The directed port must have been previously created with the ioctl
+ *	DLB_CMD_CREATE_DIR_PORT. The fd is used to mmap PP/CQ region.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: fd.
+ *
+ * Input parameters:
+ * @port_id: port ID.
+ * @padding0: Reserved for future use.
+ */
+struct dlb_get_port_fd_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
 enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,
 	DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -381,12 +415,21 @@ enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
 	DLB_DOMAIN_CMD_CREATE_LDB_PORT,
 	DLB_DOMAIN_CMD_CREATE_DIR_PORT,
+	DLB_DOMAIN_CMD_GET_LDB_PORT_PP_FD,
+	DLB_DOMAIN_CMD_GET_LDB_PORT_CQ_FD,
+	DLB_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
+	DLB_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
 
 	/* NUM_DLB_DOMAIN_CMD must be last */
 	NUM_DLB_DOMAIN_CMD,
 };
 
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
 #define DLB_CQ_SIZE 65536
+#define DLB_PP_SIZE 4096
 
 /********************/
 /* dlb ioctl codes */
@@ -434,5 +477,21 @@ enum dlb_domain_user_interface_commands {
 		_IOWR(DLB_IOC_MAGIC,				\
 		      DLB_DOMAIN_CMD_CREATE_DIR_PORT,		\
 		      struct dlb_create_dir_port_args)
+#define DLB_IOC_GET_LDB_PORT_PP_FD				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_LDB_PORT_PP_FD,	\
+		      struct dlb_get_port_fd_args)
+#define DLB_IOC_GET_LDB_PORT_CQ_FD				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_LDB_PORT_CQ_FD,	\
+		      struct dlb_get_port_fd_args)
+#define DLB_IOC_GET_DIR_PORT_PP_FD				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_DIR_PORT_PP_FD,	\
+		      struct dlb_get_port_fd_args)
+#define DLB_IOC_GET_DIR_PORT_CQ_FD				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,	\
+		      struct dlb_get_port_fd_args)
 
 #endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 14/20] dlb: add start domain ioctl
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (12 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 13/20] dlb: add port mmap support Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-03-09  9:29   ` Greg KH
  2021-02-10 17:54 ` [PATCH v10 15/20] dlb: add queue map, unmap, and pending unmap operations Mike Ximing Chen
                   ` (6 subsequent siblings)
  20 siblings, 1 reply; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add ioctl to start a domain. Once a scheduling domain and its resources
have been configured, this ioctl is called to allow the domain's ports to
begin enqueueing to the device. Once started, the domain's resources cannot
be configured again until after the domain is reset.

This ioctl instructs the DLB device to start load-balancing operations.
It corresponds to rte_event_dev_start() function in DPDK' eventdev library.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_ioctl.c    |   3 +
 drivers/misc/dlb/dlb_main.h     |   4 ++
 drivers/misc/dlb/dlb_pf_ops.c   |   9 +++
 drivers/misc/dlb/dlb_resource.c | 116 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |   4 ++
 include/uapi/linux/dlb.h        |  22 ++++++
 6 files changed, 158 insertions(+)

diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 6a311b969643..9b05344f03c8 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -51,6 +51,7 @@ DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
 
 /*
  * Port creation ioctls don't use the callback template macro.
@@ -322,6 +323,8 @@ long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 		return dlb_domain_ioctl_get_dir_port_pp_fd(dlb, dom, arg);
 	case DLB_IOC_GET_DIR_PORT_CQ_FD:
 		return dlb_domain_ioctl_get_dir_port_cq_fd(dlb, dom, arg);
+	case DLB_IOC_START_DOMAIN:
+		return dlb_domain_ioctl_start_domain(dlb, dom, arg);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 477974e1a178..2f3096a45b1e 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -63,6 +63,10 @@ struct dlb_device_ops {
 			       struct dlb_create_dir_port_args *args,
 			       uintptr_t cq_dma_base,
 			       struct dlb_cmd_response *resp);
+	int (*start_domain)(struct dlb_hw *hw,
+			    u32 domain_id,
+			    struct dlb_start_domain_args *args,
+			    struct dlb_cmd_response *resp);
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
 	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 02a188aa5a60..ce9d29b94a55 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -160,6 +160,14 @@ dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
 				       resp, false, 0);
 }
 
+static int
+dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
+		    struct dlb_start_domain_args *args,
+		    struct dlb_cmd_response *resp)
+{
+	return dlb_hw_start_domain(hw, id, args, resp, false, 0);
+}
+
 static int dlb_pf_get_num_resources(struct dlb_hw *hw,
 				    struct dlb_get_num_resources_args *args)
 {
@@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
 	.create_dir_queue = dlb_pf_create_dir_queue,
 	.create_ldb_port = dlb_pf_create_ldb_port,
 	.create_dir_port = dlb_pf_create_dir_port,
+	.start_domain = dlb_pf_start_domain,
 	.get_num_resources = dlb_pf_get_num_resources,
 	.reset_domain = dlb_pf_reset_domain,
 	.ldb_port_owned_by_domain = dlb_pf_ldb_port_owned_by_domain,
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 2659190527a7..ee95c93aac7b 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -1217,6 +1217,35 @@ dlb_verify_create_dir_port_args(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static int dlb_verify_start_domain_args(struct dlb_hw *hw, u32 domain_id,
+					struct dlb_cmd_response *resp,
+					bool vdev_req, unsigned int vdev_id,
+					struct dlb_hw_domain **out_domain)
+{
+	struct dlb_hw_domain *domain;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	if (domain->started) {
+		resp->status = DLB_ST_DOMAIN_STARTED;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+
+	return 0;
+}
+
 static void dlb_configure_domain_credits(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
 {
@@ -2712,6 +2741,93 @@ static void dlb_domain_reset_ldb_port_registers(struct dlb_hw *hw,
 	}
 }
 
+static void dlb_log_start_domain(struct dlb_hw *hw, u32 domain_id,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB start domain arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb_hw_start_domain() - start a scheduling domain
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @unused: unused.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function starts a scheduling domain, which allows applications to send
+ * traffic through it. Once a domain is started, its resources can no longer be
+ * configured (besides QID remapping and port enable/disable).
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error.
+ *
+ * Errors:
+ * EINVAL - the domain is not configured, or the domain is already started.
+ */
+int
+dlb_hw_start_domain(struct dlb_hw *hw, u32 domain_id, void *unused,
+		    struct dlb_cmd_response *resp, bool vdev_req,
+		    unsigned int vdev_id)
+{
+	struct dlb_dir_pq_pair *dir_queue;
+	struct dlb_ldb_queue *ldb_queue;
+	struct dlb_hw_domain *domain;
+	int ret;
+
+	dlb_log_start_domain(hw, domain_id, vdev_req, vdev_id);
+
+	ret = dlb_verify_start_domain_args(hw, domain_id, resp,
+					   vdev_req, vdev_id, &domain);
+	if (ret)
+		return ret;
+
+	/*
+	 * Enable load-balanced and directed queue write permissions for the
+	 * queues this domain owns. Without this, the DLB will drop all
+	 * incoming traffic to those queues.
+	 */
+	list_for_each_entry(ldb_queue, &domain->used_ldb_queues, domain_list) {
+		u32 vasqid_v = 0;
+		unsigned int offs;
+
+		vasqid_v |= SYS_LDB_VASQID_V_VASQID_V;
+
+		offs = domain->id.phys_id * DLB_MAX_NUM_LDB_QUEUES +
+			ldb_queue->id.phys_id;
+
+		DLB_CSR_WR(hw, SYS_LDB_VASQID_V(offs), vasqid_v);
+	}
+
+	list_for_each_entry(dir_queue, &domain->used_dir_pq_pairs, domain_list) {
+		u32 vasqid_v = 0;
+		unsigned int offs;
+
+		vasqid_v |= SYS_DIR_VASQID_V_VASQID_V;
+
+		offs = domain->id.phys_id * DLB_MAX_NUM_DIR_PORTS +
+			dir_queue->id.phys_id;
+
+		DLB_CSR_WR(hw, SYS_DIR_VASQID_V(offs), vasqid_v);
+	}
+
+	dlb_flush_csr(hw);
+
+	domain->started = true;
+
+	resp->status = 0;
+
+	return 0;
+}
+
 static void
 __dlb_domain_reset_dir_port_registers(struct dlb_hw *hw,
 				      struct dlb_dir_pq_pair *port)
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 8a3c37b6ab92..f4852d744dca 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -41,6 +41,10 @@ int dlb_hw_create_ldb_port(struct dlb_hw *hw, u32 domain_id,
 			   struct dlb_cmd_response *resp,
 			   bool vdev_req, unsigned int vdev_id);
 
+int dlb_hw_start_domain(struct dlb_hw *hw, u32 domain_id, void *unused,
+			struct dlb_cmd_response *resp,
+			bool vdev_req, unsigned int vdev_id);
+
 int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id);
 
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 6b7eceecae8a..84a877ad6824 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -408,6 +408,23 @@ struct dlb_get_port_fd_args {
 	__u32 padding0;
 };
 
+/*
+ * DLB_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This
+ *	must be called before passing QEs into the device, and no configuration
+ *	ioctls can be issued once the domain has started. Sending QEs into the
+ *	device before calling this ioctl will result in undefined behavior.
+ * Input parameters:
+ * - (None)
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb_start_domain_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+};
+
 enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,
 	DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -419,6 +436,7 @@ enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_GET_LDB_PORT_CQ_FD,
 	DLB_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
 	DLB_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
+	DLB_DOMAIN_CMD_START_DOMAIN,
 
 	/* NUM_DLB_DOMAIN_CMD must be last */
 	NUM_DLB_DOMAIN_CMD,
@@ -493,5 +511,9 @@ enum dlb_domain_user_interface_commands {
 		_IOWR(DLB_IOC_MAGIC,				\
 		      DLB_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,	\
 		      struct dlb_get_port_fd_args)
+#define DLB_IOC_START_DOMAIN					\
+		_IOR(DLB_IOC_MAGIC,				\
+		     DLB_DOMAIN_CMD_START_DOMAIN,		\
+		     struct dlb_start_domain_args)
 
 #endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 15/20] dlb: add queue map, unmap, and pending unmap operations
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (13 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 14/20] dlb: add start domain ioctl Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 16/20] dlb: add port map/unmap state machine Mike Ximing Chen
                   ` (5 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the high-level code for queue map, unmap, and pending unmap query ioctl
commands and argument verification -- with stubs for the low-level register
accesses and the queue map/unmap state machine, to be filled in a later
commit.

The queue map/unmap in this commit refers to link/unlink between DLB's
load-balanced queues (internal) and consumer ports.See Documentation/
misc-devices/dlb.rst for details.

Load-balanced queues can be "mapped" to any number of load-balanced ports.
Once mapped, the port becomes a candidate to which the device can schedule
queue entries from the queue. If a port is unmapped from a queue, it is no
longer a candidate for scheduling from that queue.

The pending unmaps function queries how many unmap operations are
in-progress for a given port. These operations are asynchronous, so
multiple may be in-flight at any given time.

These operations support rte_event_port_link(), rte_event_port_unlink()
and rte_event_port_unlinks_in_progress() functions of DPDK's eventdev
library.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_ioctl.c    |   9 +
 drivers/misc/dlb/dlb_main.h     |  12 ++
 drivers/misc/dlb/dlb_pf_ops.c   |  25 +++
 drivers/misc/dlb/dlb_resource.c | 358 ++++++++++++++++++++++++++++++++
 drivers/misc/dlb/dlb_resource.h |  15 ++
 include/uapi/linux/dlb.h        |  83 ++++++++
 6 files changed, 502 insertions(+)

diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 9b05344f03c8..3ce6d3ef3706 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -52,6 +52,9 @@ DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
 DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(map_qid)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(unmap_qid)
+DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(pending_port_unmaps)
 
 /*
  * Port creation ioctls don't use the callback template macro.
@@ -325,6 +328,12 @@ long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
 		return dlb_domain_ioctl_get_dir_port_cq_fd(dlb, dom, arg);
 	case DLB_IOC_START_DOMAIN:
 		return dlb_domain_ioctl_start_domain(dlb, dom, arg);
+	case DLB_IOC_MAP_QID:
+		return dlb_domain_ioctl_map_qid(dlb, dom, arg);
+	case DLB_IOC_UNMAP_QID:
+		return dlb_domain_ioctl_unmap_qid(dlb, dom, arg);
+	case DLB_IOC_PENDING_PORT_UNMAPS:
+		return dlb_domain_ioctl_pending_port_unmaps(dlb, dom, arg);
 	default:
 		return -ENOTTY;
 	}
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 2f3096a45b1e..5942fbf22cbb 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -67,6 +67,18 @@ struct dlb_device_ops {
 			    u32 domain_id,
 			    struct dlb_start_domain_args *args,
 			    struct dlb_cmd_response *resp);
+	int (*map_qid)(struct dlb_hw *hw,
+		       u32 domain_id,
+		       struct dlb_map_qid_args *args,
+		       struct dlb_cmd_response *resp);
+	int (*unmap_qid)(struct dlb_hw *hw,
+			 u32 domain_id,
+			 struct dlb_unmap_qid_args *args,
+			 struct dlb_cmd_response *resp);
+	int (*pending_port_unmaps)(struct dlb_hw *hw,
+				   u32 domain_id,
+				   struct dlb_pending_port_unmaps_args *args,
+				   struct dlb_cmd_response *resp);
 	int (*get_num_resources)(struct dlb_hw *hw,
 				 struct dlb_get_num_resources_args *args);
 	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index ce9d29b94a55..0c32b30955cf 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -168,6 +168,28 @@ dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
 	return dlb_hw_start_domain(hw, id, args, resp, false, 0);
 }
 
+static int
+dlb_pf_map_qid(struct dlb_hw *hw, u32 id, struct dlb_map_qid_args *args,
+	       struct dlb_cmd_response *resp)
+{
+	return dlb_hw_map_qid(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb_pf_unmap_qid(struct dlb_hw *hw, u32 id, struct dlb_unmap_qid_args *args,
+		 struct dlb_cmd_response *resp)
+{
+	return dlb_hw_unmap_qid(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb_pf_pending_port_unmaps(struct dlb_hw *hw, u32 id,
+			   struct dlb_pending_port_unmaps_args *args,
+			   struct dlb_cmd_response *resp)
+{
+	return dlb_hw_pending_port_unmaps(hw, id, args, resp, false, 0);
+}
+
 static int dlb_pf_get_num_resources(struct dlb_hw *hw,
 				    struct dlb_get_num_resources_args *args)
 {
@@ -241,6 +263,9 @@ struct dlb_device_ops dlb_pf_ops = {
 	.create_ldb_port = dlb_pf_create_ldb_port,
 	.create_dir_port = dlb_pf_create_dir_port,
 	.start_domain = dlb_pf_start_domain,
+	.map_qid = dlb_pf_map_qid,
+	.unmap_qid = dlb_pf_unmap_qid,
+	.pending_port_unmaps = dlb_pf_pending_port_unmaps,
 	.get_num_resources = dlb_pf_get_num_resources,
 	.reset_domain = dlb_pf_reset_domain,
 	.ldb_port_owned_by_domain = dlb_pf_ldb_port_owned_by_domain,
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index ee95c93aac7b..f39853fc664f 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -236,6 +236,32 @@ static struct dlb_hw_domain *dlb_get_domain_from_id(struct dlb_hw *hw, u32 id,
 	return NULL;
 }
 
+static struct dlb_ldb_port *
+dlb_get_domain_used_ldb_port(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	if (id >= DLB_MAX_NUM_LDB_PORTS)
+		return NULL;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+		}
+
+		list_for_each_entry(port, &domain->avail_ldb_ports[i], domain_list) {
+			if ((!vdev_req && port->id.phys_id == id) ||
+			    (vdev_req && port->id.virt_id == id))
+				return port;
+		}
+	}
+
+	return NULL;
+}
+
 static struct dlb_ldb_port *
 dlb_get_domain_ldb_port(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 {
@@ -1246,6 +1272,124 @@ static int dlb_verify_start_domain_args(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static int dlb_verify_map_qid_args(struct dlb_hw *hw, u32 domain_id,
+				   struct dlb_map_qid_args *args,
+				   struct dlb_cmd_response *resp,
+				   bool vdev_req, unsigned int vdev_id,
+				   struct dlb_hw_domain **out_domain,
+				   struct dlb_ldb_port **out_port,
+				   struct dlb_ldb_queue **out_queue)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	struct dlb_ldb_port *port;
+	int id;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (!port || !port->configured) {
+		resp->status = DLB_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (args->priority >= DLB_QID_PRIORITIES) {
+		resp->status = DLB_ST_INVALID_PRIORITY;
+		return -EINVAL;
+	}
+
+	queue = dlb_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (!queue || !queue->configured) {
+		resp->status = DLB_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (queue->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+	*out_queue = queue;
+	*out_port = port;
+
+	return 0;
+}
+
+static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
+				     struct dlb_unmap_qid_args *args,
+				     struct dlb_cmd_response *resp,
+				     bool vdev_req, unsigned int vdev_id,
+				     struct dlb_hw_domain **out_domain,
+				     struct dlb_ldb_port **out_port,
+				     struct dlb_ldb_queue **out_queue)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	struct dlb_ldb_port *port;
+	int id;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	if (!domain->configured) {
+		resp->status = DLB_ST_DOMAIN_NOT_CONFIGURED;
+		return -EINVAL;
+	}
+
+	id = args->port_id;
+
+	port = dlb_get_domain_used_ldb_port(id, vdev_req, domain);
+
+	if (!port || !port->configured) {
+		resp->status = DLB_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	if (port->domain_id.phys_id != domain->id.phys_id) {
+		resp->status = DLB_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	queue = dlb_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+	if (!queue || !queue->configured) {
+		DLB_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+			   __func__, args->qid);
+		resp->status = DLB_ST_INVALID_QID;
+		return -EINVAL;
+	}
+
+	*out_domain = domain;
+	*out_port = port;
+	*out_queue = queue;
+
+	return 0;
+}
+
 static void dlb_configure_domain_credits(struct dlb_hw *hw,
 					 struct dlb_hw_domain *domain)
 {
@@ -2191,6 +2335,163 @@ int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void dlb_log_map_qid(struct dlb_hw *hw, u32 domain_id,
+			    struct dlb_map_qid_args *args,
+			    bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB map QID arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n",
+		   domain_id);
+	DLB_HW_DBG(hw, "\tPort ID:   %d\n",
+		   args->port_id);
+	DLB_HW_DBG(hw, "\tQueue ID:  %d\n",
+		   args->qid);
+	DLB_HW_DBG(hw, "\tPriority:  %d\n",
+		   args->priority);
+}
+
+/**
+ * dlb_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
+		   struct dlb_map_qid_args *args,
+		   struct dlb_cmd_response *resp,
+		   bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	struct dlb_ldb_port *port;
+	int ret;
+
+	dlb_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_map_qid_args(hw, domain_id, args, resp, vdev_req,
+				      vdev_id, &domain, &port, &queue);
+	if (ret)
+		return ret;
+
+	resp->status = 0;
+
+	return 0;
+}
+
+static void dlb_log_unmap_qid(struct dlb_hw *hw, u32 domain_id,
+			      struct dlb_unmap_qid_args *args,
+			      bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB unmap QID arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tDomain ID: %d\n",
+		   domain_id);
+	DLB_HW_DBG(hw, "\tPort ID:   %d\n",
+		   args->port_id);
+	DLB_HW_DBG(hw, "\tQueue ID:  %d\n",
+		   args->qid);
+	if (args->qid < DLB_MAX_NUM_LDB_QUEUES)
+		DLB_HW_DBG(hw, "\tQueue's num mappings:  %d\n",
+			   hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+/**
+ * dlb_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ *	    the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
+		     struct dlb_unmap_qid_args *args,
+		     struct dlb_cmd_response *resp,
+		     bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+	struct dlb_ldb_port *port;
+	int ret;
+
+	dlb_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+	/*
+	 * Verify that hardware resources are available before attempting to
+	 * satisfy the request. This simplifies the error unwinding code.
+	 */
+	ret = dlb_verify_unmap_qid_args(hw, domain_id, args, resp, vdev_req,
+					vdev_id, &domain, &port, &queue);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
 static u32 dlb_ldb_cq_inflight_count(struct dlb_hw *hw,
 				     struct dlb_ldb_port *port)
 {
@@ -2530,6 +2831,63 @@ int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void
+dlb_log_pending_port_unmaps_args(struct dlb_hw *hw,
+				 struct dlb_pending_port_unmaps_args *args,
+				 bool vdev_req, unsigned int vdev_id)
+{
+	DLB_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+	if (vdev_req)
+		DLB_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+	DLB_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+/**
+ * dlb_hw_pending_port_unmaps() - returns the number of unmap operations in
+ *	progress.
+ * @hw: dlb_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb_hw_pending_port_unmaps(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_pending_port_unmaps_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id)
+{
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_port *port;
+
+	dlb_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+	if (!domain) {
+		resp->status = DLB_ST_INVALID_DOMAIN_ID;
+		return -EINVAL;
+	}
+
+	port = dlb_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+	if (!port || !port->configured) {
+		resp->status = DLB_ST_INVALID_PORT_ID;
+		return -EINVAL;
+	}
+
+	resp->id = port->num_pending_removals;
+
+	return 0;
+}
+
 static u32 dlb_ldb_queue_depth(struct dlb_hw *hw, struct dlb_ldb_queue *queue)
 {
 	u32 aqed, ldb, atm;
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index f4852d744dca..e3de9eb94d5d 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -45,6 +45,16 @@ int dlb_hw_start_domain(struct dlb_hw *hw, u32 domain_id, void *unused,
 			struct dlb_cmd_response *resp,
 			bool vdev_req, unsigned int vdev_id);
 
+int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
+		   struct dlb_map_qid_args *args,
+		   struct dlb_cmd_response *resp,
+		   bool vdev_req, unsigned int vdev_id);
+
+int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
+		     struct dlb_unmap_qid_args *args,
+		     struct dlb_cmd_response *resp,
+		     bool vdev_req, unsigned int vdev_id);
+
 int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 		     unsigned int vdev_id);
 
@@ -70,6 +80,11 @@ int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
 			       struct dlb_cmd_response *resp,
 			       bool vdev_req, unsigned int vdev_id);
 
+int dlb_hw_pending_port_unmaps(struct dlb_hw *hw, u32 domain_id,
+			       struct dlb_pending_port_unmaps_args *args,
+			       struct dlb_cmd_response *resp,
+			       bool vdev_req, unsigned int vdev_id);
+
 void dlb_hw_enable_sparse_ldb_cq_mode(struct dlb_hw *hw);
 
 void dlb_hw_enable_sparse_dir_cq_mode(struct dlb_hw *hw);
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index 84a877ad6824..37eccb6ec230 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -34,6 +34,8 @@ enum dlb_error {
 	DLB_ST_INVALID_CQ_DEPTH,
 	DLB_ST_INVALID_HIST_LIST_DEPTH,
 	DLB_ST_INVALID_DIR_QUEUE_ID,
+	DLB_ST_INVALID_PRIORITY,
+	DLB_ST_NO_QID_SLOTS_AVAILABLE,
 };
 
 struct dlb_cmd_response {
@@ -425,6 +427,72 @@ struct dlb_start_domain_args {
 	struct dlb_cmd_response response;
 };
 
+/*
+ * DLB_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ *
+ * Input parameters:
+ * @port_id: Load-balanced port ID.
+ * @qid: Load-balanced queue ID.
+ * @priority: Queue->port service priority.
+ * @padding0: Reserved for future use.
+ */
+struct dlb_map_qid_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+	__u32 priority;
+	__u32 padding0;
+};
+
+/*
+ * DLB_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ *	port.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ *
+ * Input parameters:
+ * @port_id: Load-balanced port ID.
+ * @qid: Load-balanced queue ID.
+ */
+struct dlb_unmap_qid_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 qid;
+};
+
+/*
+ * DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ *	progress for a load-balanced port.
+ *
+ *	Note: This is a snapshot; the number of unmap operations in progress
+ *	is subject to change at any time.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ *	ioctl request arg is invalid, the driver won't set status.
+ * @response.id: number of unmaps in progress.
+ *
+ * Input parameters:
+ * @port_id: Load-balanced port ID.
+ */
+struct dlb_pending_port_unmaps_args {
+	/* Output parameters */
+	struct dlb_cmd_response response;
+	/* Input parameters */
+	__u32 port_id;
+	__u32 padding0;
+};
+
 enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_CREATE_LDB_QUEUE,
 	DLB_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -437,6 +505,9 @@ enum dlb_domain_user_interface_commands {
 	DLB_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
 	DLB_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
 	DLB_DOMAIN_CMD_START_DOMAIN,
+	DLB_DOMAIN_CMD_MAP_QID,
+	DLB_DOMAIN_CMD_UNMAP_QID,
+	DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS,
 
 	/* NUM_DLB_DOMAIN_CMD must be last */
 	NUM_DLB_DOMAIN_CMD,
@@ -515,5 +586,17 @@ enum dlb_domain_user_interface_commands {
 		_IOR(DLB_IOC_MAGIC,				\
 		     DLB_DOMAIN_CMD_START_DOMAIN,		\
 		     struct dlb_start_domain_args)
+#define DLB_IOC_MAP_QID						\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_MAP_QID,			\
+		      struct dlb_map_qid_args)
+#define DLB_IOC_UNMAP_QID					\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_UNMAP_QID,			\
+		      struct dlb_unmap_qid_args)
+#define DLB_IOC_PENDING_PORT_UNMAPS				\
+		_IOWR(DLB_IOC_MAGIC,				\
+		      DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS,	\
+		      struct dlb_pending_port_unmaps_args)
 
 #endif /* __DLB_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 16/20] dlb: add port map/unmap state machine
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (14 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 15/20] dlb: add queue map, unmap, and pending unmap operations Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 17/20] dlb: add static queue map register operations Mike Ximing Chen
                   ` (4 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add support for the port map/unmap state machine. Each load-balanced port
has eight "slots", one for each queue it is linked to, and each slot can be
in one of five states:
1. Queue unmapped
2. Queue mapped
3. Queue unmap in progress
4. Queue map in progress
5. Queue unmap in progress, with a map pending when the unmap completes.

These states exist because the map and unmap operations can be asynchronous
(with respect to the ioctl command). If the domain is already started, the
map operation must (temporarily) disable the queue and wait for it to
quiesce. Similarly, the unmap operation must (temporarily) disable the port
and wait for it to quiesce. 'Quiesce' here means the user processes any
in-flight QEs.

The queue map/unmap in this commit refers to link/unlink between DLB's
load-balanced queues (internal) and consumer ports. See Documentation/
misc-devices/dlb.rst for details.

It's possible that the thread that requires the map/unmap is the same one
which is responsible for doing the processing that would quiesce the
queue/port, in which case the driver may have to complete the operation
asynchronously.

To support this asynchronous operation while also providing a reasonably
usable user-interface, the driver maintains two views of the queue map
(slot) state:
- The hardware view: the actual state in the device
- The user/software view: the state as though the operations were
  synchronous.

While a map/unmap operation is inflight, these two views are out-of-sync.
When the user requests a new map/unmap operation, the driver verifies the
request against the software view, so any errors are synchronous from the
user’s perspective, then adds the request to the queue of in-progress
operations. When possible -- for example if the user requests to map a
queue and then immediately requests to unmap it -- the driver will coalesce
or cancel outstanding operations.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_resource.c | 433 +++++++++++++++++++++++++++++++-
 1 file changed, 431 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index f39853fc664f..a830b547dadf 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -1335,6 +1335,225 @@ static int dlb_verify_map_qid_args(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static bool dlb_port_find_slot(struct dlb_ldb_port *port,
+			       enum dlb_qid_map_state state, int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb_port_find_slot_queue(struct dlb_ldb_port *port,
+				     enum dlb_qid_map_state state,
+				     struct dlb_ldb_queue *queue, int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		if (port->qid_map[i].state == state &&
+		    port->qid_map[i].qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool
+dlb_port_find_slot_with_pending_map_queue(struct dlb_ldb_port *port,
+					  struct dlb_ldb_queue *queue, int *slot)
+{
+	int i;
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb_ldb_port_qid_map *map = &port->qid_map[i];
+
+		if (map->state == DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+		    map->pending_qid == queue->id.phys_id)
+			break;
+	}
+
+	*slot = i;
+
+	return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static int dlb_port_slot_state_transition(struct dlb_hw *hw,
+					  struct dlb_ldb_port *port,
+					  struct dlb_ldb_queue *queue, int slot,
+					  enum dlb_qid_map_state new_state)
+{
+	enum dlb_qid_map_state curr_state = port->qid_map[slot].state;
+	struct dlb_hw_domain *domain;
+	int domain_id;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, false, 0);
+	if (!domain) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: unable to find domain %d\n",
+			   __func__, domain_id);
+		return -EINVAL;
+	}
+
+	switch (curr_state) {
+	case DLB_QUEUE_UNMAPPED:
+		switch (new_state) {
+		case DLB_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			break;
+		case DLB_QUEUE_MAP_IN_PROG:
+			queue->num_pending_additions++;
+			domain->num_pending_additions++;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB_QUEUE_MAPPED:
+		switch (new_state) {
+		case DLB_QUEUE_UNMAPPED:
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB_QUEUE_UNMAP_IN_PROG:
+			port->num_pending_removals++;
+			domain->num_pending_removals++;
+			break;
+		case DLB_QUEUE_MAPPED:
+			/* Priority change, nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB_QUEUE_MAP_IN_PROG:
+		switch (new_state) {
+		case DLB_QUEUE_UNMAPPED:
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		case DLB_QUEUE_MAPPED:
+			queue->num_mappings++;
+			port->num_mappings++;
+			queue->num_pending_additions--;
+			domain->num_pending_additions--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB_QUEUE_UNMAP_IN_PROG:
+		switch (new_state) {
+		case DLB_QUEUE_UNMAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			queue->num_mappings--;
+			port->num_mappings--;
+			break;
+		case DLB_QUEUE_MAPPED:
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		case DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+			/* Nothing to update */
+			break;
+		default:
+			goto error;
+		}
+		break;
+	case DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+		switch (new_state) {
+		case DLB_QUEUE_UNMAP_IN_PROG:
+			/* Nothing to update */
+			break;
+		case DLB_QUEUE_UNMAPPED:
+			/*
+			 * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+			 * becomes UNMAPPED before it transitions to
+			 * MAP_IN_PROG.
+			 */
+			queue->num_mappings--;
+			port->num_mappings--;
+			port->num_pending_removals--;
+			domain->num_pending_removals--;
+			break;
+		default:
+			goto error;
+		}
+		break;
+	default:
+		goto error;
+	}
+
+	port->qid_map[slot].state = new_state;
+
+	DLB_HW_DBG(hw,
+		   "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+		   __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return 0;
+
+error:
+	DLB_HW_ERR(hw,
+		   "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+		   __func__, queue->id.phys_id, port->id.phys_id,
+		    curr_state, new_state);
+	return -EFAULT;
+}
+
+static int dlb_verify_map_qid_slot_available(struct dlb_ldb_port *port,
+					     struct dlb_ldb_queue *queue,
+					     struct dlb_cmd_response *resp)
+{
+	enum dlb_qid_map_state state;
+	int i;
+
+	/* Unused slot available? */
+	if (port->num_mappings < DLB_MAX_NUM_QIDS_PER_LDB_CQ)
+		return 0;
+
+	/*
+	 * If the queue is already mapped (from the application's perspective),
+	 * this is simply a priority update.
+	 */
+	state = DLB_QUEUE_MAPPED;
+	if (dlb_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	state = DLB_QUEUE_MAP_IN_PROG;
+	if (dlb_port_find_slot_queue(port, state, queue, &i))
+		return 0;
+
+	if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i))
+		return 0;
+
+	/*
+	 * If the slot contains an unmap in progress, it's considered
+	 * available.
+	 */
+	state = DLB_QUEUE_UNMAP_IN_PROG;
+	if (dlb_port_find_slot(port, state, &i))
+		return 0;
+
+	state = DLB_QUEUE_UNMAPPED;
+	if (dlb_port_find_slot(port, state, &i))
+		return 0;
+
+	resp->status = DLB_ST_NO_QID_SLOTS_AVAILABLE;
+	return -EINVAL;
+}
+
 static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
 				     struct dlb_unmap_qid_args *args,
 				     struct dlb_cmd_response *resp,
@@ -1343,9 +1562,11 @@ static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
 				     struct dlb_ldb_port **out_port,
 				     struct dlb_ldb_queue **out_queue)
 {
+	enum dlb_qid_map_state state;
 	struct dlb_hw_domain *domain;
 	struct dlb_ldb_queue *queue;
 	struct dlb_ldb_port *port;
+	int slot;
 	int id;
 
 	domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
@@ -1383,6 +1604,26 @@ static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
 		return -EINVAL;
 	}
 
+	/*
+	 * Verify that the port has the queue mapped. From the application's
+	 * perspective a queue is mapped if it is actually mapped, the map is
+	 * in progress, or the map is blocked pending an unmap.
+	 */
+	state = DLB_QUEUE_MAPPED;
+	if (dlb_port_find_slot_queue(port, state, queue, &slot))
+		goto done;
+
+	state = DLB_QUEUE_MAP_IN_PROG;
+	if (dlb_port_find_slot_queue(port, state, queue, &slot))
+		goto done;
+
+	if (dlb_port_find_slot_with_pending_map_queue(port, queue, &slot))
+		goto done;
+
+	resp->status = DLB_ST_INVALID_QID;
+	return -EINVAL;
+
+done:
 	*out_domain = domain;
 	*out_port = port;
 	*out_queue = queue;
@@ -1881,6 +2122,21 @@ static int dlb_configure_dir_port(struct dlb_hw *hw, struct dlb_hw_domain *domai
 	return 0;
 }
 
+static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
+					     struct dlb_ldb_port *port, int slot,
+					     struct dlb_map_qid_args *args)
+{
+	/* Placeholder */
+}
+
+static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
+				struct dlb_ldb_port *port,
+				struct dlb_ldb_queue *queue, u8 prio)
+{
+	/* Placeholder */
+	return 0;
+}
+
 static void
 dlb_log_create_sched_domain_args(struct dlb_hw *hw,
 				 struct dlb_create_sched_domain_args *args,
@@ -2400,8 +2656,10 @@ int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
 {
 	struct dlb_hw_domain *domain;
 	struct dlb_ldb_queue *queue;
+	enum dlb_qid_map_state st;
 	struct dlb_ldb_port *port;
-	int ret;
+	int ret, i;
+	u8 prio;
 
 	dlb_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
 
@@ -2414,6 +2672,124 @@ int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	prio = args->priority;
+
+	/*
+	 * If there are any outstanding detach operations for this port,
+	 * attempt to complete them. This may be necessary to free up a QID
+	 * slot for this requested mapping.
+	 */
+	ret = dlb_verify_map_qid_slot_available(port, queue, resp);
+	if (ret)
+		return ret;
+
+	/* Hardware requires disabling the CQ before mapping QIDs. */
+	if (port->enabled)
+		dlb_ldb_port_cq_disable(hw, port);
+
+	/*
+	 * If this is only a priority change, don't perform the full QID->CQ
+	 * mapping procedure
+	 */
+	st = DLB_QUEUE_MAPPED;
+	if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+		if (prio != port->qid_map[i].priority) {
+			dlb_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB_HW_DBG(hw, "DLB map: priority change\n");
+		}
+
+		st = DLB_QUEUE_MAPPED;
+		ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	st = DLB_QUEUE_UNMAP_IN_PROG;
+	if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+		if (prio != port->qid_map[i].priority) {
+			dlb_ldb_port_change_qid_priority(hw, port, i, args);
+			DLB_HW_DBG(hw, "DLB map: priority change\n");
+		}
+
+		st = DLB_QUEUE_MAPPED;
+		ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on an in-progress mapping, don't
+	 * perform the full QID->CQ mapping procedure.
+	 */
+	st = DLB_QUEUE_MAP_IN_PROG;
+	if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+		port->qid_map[i].priority = prio;
+
+		DLB_HW_DBG(hw, "DLB map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If this is a priority change on a pending mapping, update the
+	 * pending priority
+	 */
+	if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		port->qid_map[i].pending_priority = prio;
+
+		DLB_HW_DBG(hw, "DLB map: priority change only\n");
+
+		goto map_qid_done;
+	}
+
+	/*
+	 * If all the CQ's slots are in use, then there's an unmap in progress
+	 * (guaranteed by dlb_verify_map_qid_slot_available()), so add this
+	 * mapping to pending_map and return. When the removal is completed for
+	 * the slot's current occupant, this mapping will be performed.
+	 */
+	if (!dlb_port_find_slot(port, DLB_QUEUE_UNMAPPED, &i)) {
+		if (dlb_port_find_slot(port, DLB_QUEUE_UNMAP_IN_PROG, &i)) {
+			enum dlb_qid_map_state new_st;
+
+			port->qid_map[i].pending_qid = queue->id.phys_id;
+			port->qid_map[i].pending_priority = prio;
+
+			new_st = DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+			ret = dlb_port_slot_state_transition(hw, port, queue,
+							     i, new_st);
+			if (ret)
+				return ret;
+
+			DLB_HW_DBG(hw, "DLB map: map pending removal\n");
+
+			goto map_qid_done;
+		}
+	}
+
+	/*
+	 * If the domain has started, a special "dynamic" CQ->queue mapping
+	 * procedure is required in order to safely update the CQ<->QID tables.
+	 * The "static" procedure cannot be used when traffic is flowing,
+	 * because the CQ<->QID tables cannot be updated atomically and the
+	 * scheduler won't see the new mapping unless the queue's if_status
+	 * changes, which isn't guaranteed.
+	 */
+	ret = dlb_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+	/* If ret is less than zero, it's due to an internal error */
+	if (ret < 0)
+		return ret;
+
+map_qid_done:
+	if (port->enabled)
+		dlb_ldb_port_cq_enable(hw, port);
+
 	resp->status = 0;
 
 	return 0;
@@ -2475,8 +2851,9 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 {
 	struct dlb_hw_domain *domain;
 	struct dlb_ldb_queue *queue;
+	enum dlb_qid_map_state st;
 	struct dlb_ldb_port *port;
-	int ret;
+	int i, ret;
 
 	dlb_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
 
@@ -2489,6 +2866,58 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the queue hasn't been mapped yet, we need to update the slot's
+	 * state and re-enable the queue's inflights.
+	 */
+	st = DLB_QUEUE_MAP_IN_PROG;
+	if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+		st = DLB_QUEUE_UNMAPPED;
+		ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	/*
+	 * If the queue mapping is on hold pending an unmap, we simply need to
+	 * update the slot's state.
+	 */
+	if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+		st = DLB_QUEUE_UNMAP_IN_PROG;
+		ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+		if (ret)
+			return ret;
+
+		goto unmap_qid_done;
+	}
+
+	st = DLB_QUEUE_MAPPED;
+	if (!dlb_port_find_slot_queue(port, st, queue, &i)) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: no available CQ slots\n",
+			   __func__);
+		return -EFAULT;
+	}
+
+	/*
+	 * QID->CQ mapping removal is an asychronous procedure. It requires
+	 * stopping the DLB from scheduling this CQ, draining all inflights
+	 * from the CQ, then unmapping the queue from the CQ. This function
+	 * simply marks the port as needing the queue unmapped, and (if
+	 * necessary) starts the unmapping worker thread.
+	 */
+	dlb_ldb_port_cq_disable(hw, port);
+
+	st = DLB_QUEUE_UNMAP_IN_PROG;
+	ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+	if (ret)
+		return ret;
+
+unmap_qid_done:
+	resp->status = 0;
+
 	return 0;
 }
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 17/20] dlb: add static queue map register operations
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (15 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 16/20] dlb: add port map/unmap state machine Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 18/20] dlb: add dynamic " Mike Ximing Chen
                   ` (3 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the register accesses that implement the static queue map operation and
handle an unmap request when a queue map operation is in progress.

If a queue map operation is requested before the domain is started, it is a
synchronous procedure on "static"/unchanging hardware. (The "dynamic"
operation, when traffic is flowing in the device, will be added in a later
commit.)

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_resource.c | 144 +++++++++++++++++++++++++++++++-
 1 file changed, 141 insertions(+), 3 deletions(-)

diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index a830b547dadf..95ccb7eddb8b 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -2122,19 +2122,150 @@ static int dlb_configure_dir_port(struct dlb_hw *hw, struct dlb_hw_domain *domai
 	return 0;
 }
 
+static int dlb_ldb_port_map_qid_static(struct dlb_hw *hw, struct dlb_ldb_port *p,
+				       struct dlb_ldb_queue *q, u8 priority)
+{
+	enum dlb_qid_map_state state;
+	u32 lsp_qid2cq2;
+	u32 lsp_qid2cq;
+	u32 atm_qid2cq;
+	u32 cq2priov;
+	u32 cq2qid;
+	int i;
+
+	/* Look for a pending or already mapped slot, else an unused slot */
+	if (!dlb_port_find_slot_queue(p, DLB_QUEUE_MAP_IN_PROG, q, &i) &&
+	    !dlb_port_find_slot_queue(p, DLB_QUEUE_MAPPED, q, &i) &&
+	    !dlb_port_find_slot(p, DLB_QUEUE_UNMAPPED, &i)) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+			   __func__, __LINE__);
+		return -EFAULT;
+	}
+
+	/* Read-modify-write the priority and valid bit register */
+	cq2priov = DLB_CSR_RD(hw, LSP_CQ2PRIOV(p->id.phys_id));
+
+	cq2priov |= (1U << (i + LSP_CQ2PRIOV_V_LOC)) & LSP_CQ2PRIOV_V;
+	cq2priov |= ((priority & 0x7) << (i + LSP_CQ2PRIOV_PRIO_LOC) * 3)
+		    & LSP_CQ2PRIOV_PRIO;
+
+	DLB_CSR_WR(hw, LSP_CQ2PRIOV(p->id.phys_id), cq2priov);
+
+	/* Read-modify-write the QID map register */
+	if (i < 4)
+		cq2qid = DLB_CSR_RD(hw, LSP_CQ2QID0(p->id.phys_id));
+	else
+		cq2qid = DLB_CSR_RD(hw, LSP_CQ2QID1(p->id.phys_id));
+
+	if (i == 0 || i == 4)
+		BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P0);
+	if (i == 1 || i == 5)
+		BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P1);
+	if (i == 2 || i == 6)
+		BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P2);
+	if (i == 3 || i == 7)
+		BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P3);
+
+	if (i < 4)
+		DLB_CSR_WR(hw, LSP_CQ2QID0(p->id.phys_id), cq2qid);
+	else
+		DLB_CSR_WR(hw, LSP_CQ2QID1(p->id.phys_id), cq2qid);
+
+	atm_qid2cq = DLB_CSR_RD(hw,
+				ATM_QID2CQIDIX(q->id.phys_id,
+					       p->id.phys_id / 4));
+
+	lsp_qid2cq = DLB_CSR_RD(hw,
+				LSP_QID2CQIDIX(q->id.phys_id,
+					       p->id.phys_id / 4));
+
+	lsp_qid2cq2 = DLB_CSR_RD(hw,
+				 LSP_QID2CQIDIX2(q->id.phys_id,
+						 p->id.phys_id / 4));
+
+	switch (p->id.phys_id % 4) {
+	case 0:
+		atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P0_LOC));
+		lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P0_LOC));
+		lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P0_LOC));
+		break;
+
+	case 1:
+		atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P1_LOC));
+		lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P1_LOC));
+		lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P1_LOC));
+		break;
+
+	case 2:
+		atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P2_LOC));
+		lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P2_LOC));
+		lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P2_LOC));
+		break;
+
+	case 3:
+		atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P3_LOC));
+		lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P3_LOC));
+		lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P3_LOC));
+		break;
+	}
+
+	DLB_CSR_WR(hw,
+		   ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		   atm_qid2cq);
+
+	DLB_CSR_WR(hw,
+		   LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+		   lsp_qid2cq);
+
+	DLB_CSR_WR(hw,
+		   LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+		   lsp_qid2cq2);
+
+	dlb_flush_csr(hw);
+
+	p->qid_map[i].qid = q->id.phys_id;
+	p->qid_map[i].priority = priority;
+
+	state = DLB_QUEUE_MAPPED;
+
+	return dlb_port_slot_state_transition(hw, p, q, i, state);
+}
+
 static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
 					     struct dlb_ldb_port *port, int slot,
 					     struct dlb_map_qid_args *args)
 {
-	/* Placeholder */
+	u32 cq2priov;
+
+	/* Read-modify-write the priority and valid bit register */
+	cq2priov = DLB_CSR_RD(hw, LSP_CQ2PRIOV(port->id.phys_id));
+
+	cq2priov |= (1 << (slot + LSP_CQ2PRIOV_V_LOC)) & LSP_CQ2PRIOV_V;
+	cq2priov |= ((args->priority & 0x7) << slot * 3) & LSP_CQ2PRIOV_PRIO;
+
+	DLB_CSR_WR(hw, LSP_CQ2PRIOV(port->id.phys_id), cq2priov);
+
+	dlb_flush_csr(hw);
+
+	port->qid_map[slot].priority = args->priority;
+}
+
+static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
+					     struct dlb_ldb_queue *queue)
+{
+	u32 infl_lim = 0;
+
+	BITS_SET(infl_lim, queue->num_qid_inflights, LSP_QID_LDB_INFL_LIM_LIMIT);
+
+	DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), infl_lim);
 }
 
 static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
 				struct dlb_ldb_port *port,
 				struct dlb_ldb_queue *queue, u8 prio)
 {
-	/* Placeholder */
-	return 0;
+	return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
 }
 
 static void
@@ -2872,6 +3003,13 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 	 */
 	st = DLB_QUEUE_MAP_IN_PROG;
 	if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+		/*
+		 * Since the in-progress map was aborted, re-enable the QID's
+		 * inflights.
+		 */
+		if (queue->num_pending_additions == 0)
+			dlb_ldb_queue_set_inflight_limit(hw, queue);
+
 		st = DLB_QUEUE_UNMAPPED;
 		ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
 		if (ret)
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 18/20] dlb: add dynamic queue map register operations
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (16 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 17/20] dlb: add static queue map register operations Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 19/20] dlb: add queue unmap " Mike Ximing Chen
                   ` (2 subsequent siblings)
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Adds the "dynamic" map procedure and register operations. If a queue map
is requested after the domain is started, the driver must disable the
requested queue and wait for it to quiesce before mapping it to the
requested port.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_resource.c | 393 +++++++++++++++++++++++++++++++-
 1 file changed, 392 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 95ccb7eddb8b..93a3de642024 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -328,6 +328,37 @@ dlb_get_domain_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 	return NULL;
 }
 
+static struct dlb_ldb_queue *
+dlb_get_ldb_queue_from_id(struct dlb_hw *hw, u32 id, bool vdev_req,
+			  unsigned int vdev_id)
+{
+	struct dlb_function_resources *rsrcs;
+	struct dlb_hw_domain *domain;
+	struct dlb_ldb_queue *queue;
+
+	if (id >= DLB_MAX_NUM_LDB_QUEUES)
+		return NULL;
+
+	rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+	if (!vdev_req)
+		return &hw->rsrcs.ldb_queues[id];
+
+	list_for_each_entry(domain, &rsrcs->used_domains, func_list) {
+		list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+			if (queue->id.virt_id == id)
+				return queue;
+		}
+	}
+
+	list_for_each_entry(queue, &rsrcs->avail_ldb_queues, func_list) {
+		if (queue->id.virt_id == id)
+			return queue;
+	}
+
+	return NULL;
+}
+
 static struct dlb_ldb_queue *
 dlb_get_domain_ldb_queue(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
 {
@@ -2251,6 +2282,75 @@ static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
 	port->qid_map[slot].priority = args->priority;
 }
 
+static int dlb_ldb_port_set_has_work_bits(struct dlb_hw *hw,
+					  struct dlb_ldb_port *port,
+					  struct dlb_ldb_queue *queue, int slot)
+{
+	u32 ctrl = 0;
+	u32 active;
+	u32 enq;
+
+	/* Set the atomic scheduling haswork bit */
+	active = DLB_CSR_RD(hw, LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+	BITS_SET(ctrl, (u32)(BITS_GET(active, LSP_QID_AQED_ACTIVE_CNT_COUNT) > 0),
+		 LSP_LDB_SCHED_CTRL_RLIST_HASWORK_V);
+
+	/* Set the non-atomic scheduling haswork bit */
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	enq = DLB_CSR_RD(hw,
+			 LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+	memset(&ctrl, 0, sizeof(ctrl));
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+	BITS_SET(ctrl, (u32)(BITS_GET(enq, LSP_QID_LDB_ENQUEUE_CNT_COUNT) > 0),
+		 LSP_LDB_SCHED_CTRL_NALB_HASWORK_V);
+
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	dlb_flush_csr(hw);
+
+	return 0;
+}
+
+static void dlb_ldb_port_clear_queue_if_status(struct dlb_hw *hw,
+					       struct dlb_ldb_port *port,
+					       int slot)
+{
+	u32 ctrl = 0;
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V;
+
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	dlb_flush_csr(hw);
+}
+
+static void dlb_ldb_port_set_queue_if_status(struct dlb_hw *hw,
+					     struct dlb_ldb_port *port,
+					     int slot)
+{
+	u32 ctrl = 0;
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+	ctrl |= LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V;
+
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	dlb_flush_csr(hw);
+}
+
 static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
 					     struct dlb_ldb_queue *queue)
 {
@@ -2261,11 +2361,222 @@ static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
 	DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), infl_lim);
 }
 
+static void dlb_ldb_queue_clear_inflight_limit(struct dlb_hw *hw,
+					       struct dlb_ldb_queue *queue)
+{
+	DLB_CSR_WR(hw,
+		   LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+		   LSP_QID_LDB_INFL_LIM_RST);
+}
+
+/*
+ * dlb_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb_ldb_queue_disable_mapped_cqs(struct dlb_hw *hw,
+					     struct dlb_hw_domain *domain,
+					     struct dlb_ldb_queue *queue)
+{
+	struct dlb_ldb_port *port;
+	int slot, i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			enum dlb_qid_map_state state = DLB_QUEUE_MAPPED;
+
+			if (!dlb_port_find_slot_queue(port, state,
+						      queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb_ldb_port_cq_disable(hw, port);
+		}
+	}
+}
+
+static void dlb_ldb_queue_enable_mapped_cqs(struct dlb_hw *hw,
+					    struct dlb_hw_domain *domain,
+					    struct dlb_ldb_queue *queue)
+{
+	struct dlb_ldb_port *port;
+	int slot, i;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			enum dlb_qid_map_state state = DLB_QUEUE_MAPPED;
+
+			if (!dlb_port_find_slot_queue(port, state,
+						      queue, &slot))
+				continue;
+
+			if (port->enabled)
+				dlb_ldb_port_cq_enable(hw, port);
+		}
+	}
+}
+
+static int dlb_ldb_port_finish_map_qid_dynamic(struct dlb_hw *hw,
+					       struct dlb_hw_domain *domain,
+					       struct dlb_ldb_port *port,
+					       struct dlb_ldb_queue *queue)
+{
+	enum dlb_qid_map_state state;
+	int slot, ret, i;
+	u32 infl_cnt;
+	u8 prio;
+
+	infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: non-zero QID inflight count\n",
+			   __func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * Static map the port and set its corresponding has_work bits.
+	 */
+	state = DLB_QUEUE_MAP_IN_PROG;
+	if (!dlb_port_find_slot_queue(port, state, queue, &slot))
+		return -EINVAL;
+
+	prio = port->qid_map[slot].priority;
+
+	/*
+	 * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+	 * the port's qid_map state.
+	 */
+	ret = dlb_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (ret)
+		return ret;
+
+	ret = dlb_ldb_port_set_has_work_bits(hw, port, queue, slot);
+	if (ret)
+		return ret;
+
+	/*
+	 * Ensure IF_status(cq,qid) is 0 before enabling the port to
+	 * prevent spurious schedules to cause the queue's inflight
+	 * count to increase.
+	 */
+	dlb_ldb_port_clear_queue_if_status(hw, port, slot);
+
+	/* Reset the queue's inflight status */
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+			state = DLB_QUEUE_MAPPED;
+			if (!dlb_port_find_slot_queue(port, state,
+						      queue, &slot))
+				continue;
+
+			dlb_ldb_port_set_queue_if_status(hw, port, slot);
+		}
+	}
+
+	dlb_ldb_queue_set_inflight_limit(hw, queue);
+
+	/* Re-enable CQs mapped to this queue */
+	dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+	/* If this queue has other mappings pending, clear its inflight limit */
+	if (queue->num_pending_additions > 0)
+		dlb_ldb_queue_clear_inflight_limit(hw, queue);
+
+	return 0;
+}
+
+/**
+ * dlb_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb_ldb_port_map_qid_dynamic(struct dlb_hw *hw,
+					struct dlb_ldb_port *port,
+					struct dlb_ldb_queue *queue,
+					u8 priority)
+{
+	enum dlb_qid_map_state state;
+	struct dlb_hw_domain *domain;
+	int domain_id, slot, ret;
+	u32 infl_cnt;
+
+	domain_id = port->domain_id.phys_id;
+
+	domain = dlb_get_domain_from_id(hw, domain_id, false, 0);
+	if (!domain) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: unable to find domain %d\n",
+			   __func__, port->domain_id.phys_id);
+		return -EINVAL;
+	}
+
+	/*
+	 * Set the QID inflight limit to 0 to prevent further scheduling of the
+	 * queue.
+	 */
+	DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+	if (!dlb_port_find_slot(port, DLB_QUEUE_UNMAPPED, &slot)) {
+		DLB_HW_ERR(hw,
+			   "Internal error: No available unmapped slots\n");
+		return -EFAULT;
+	}
+
+	port->qid_map[slot].qid = queue->id.phys_id;
+	port->qid_map[slot].priority = priority;
+
+	state = DLB_QUEUE_MAP_IN_PROG;
+	ret = dlb_port_slot_state_transition(hw, port, queue, slot, state);
+	if (ret)
+		return ret;
+
+	infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT))
+		return 1;
+
+	/*
+	 * Disable the affected CQ, and the CQs already mapped to the QID,
+	 * before reading the QID's inflight count a second time. There is an
+	 * unlikely race in which the QID may schedule one more QE after we
+	 * read an inflight count of 0, and disabling the CQs guarantees that
+	 * the race will not occur after a re-read of the inflight count
+	 * register.
+	 */
+	if (port->enabled)
+		dlb_ldb_port_cq_disable(hw, port);
+
+	dlb_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+	infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+	if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+		if (port->enabled)
+			dlb_ldb_port_cq_enable(hw, port);
+
+		dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+		return 1;
+	}
+
+	return dlb_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
 static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
 				struct dlb_ldb_port *port,
 				struct dlb_ldb_queue *queue, u8 prio)
 {
-	return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
+	if (domain->started)
+		return dlb_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+	else
+		return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
 }
 
 static void
@@ -2722,6 +3033,82 @@ int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void dlb_domain_finish_map_port(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain,
+				       struct dlb_ldb_port *port)
+{
+	int i;
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb_ldb_queue *queue;
+		u32 infl_cnt;
+		int qid;
+
+		if (port->qid_map[i].state != DLB_QUEUE_MAP_IN_PROG)
+			continue;
+
+		qid = port->qid_map[i].qid;
+
+		queue = dlb_get_ldb_queue_from_id(hw, qid, false, 0);
+
+		if (!queue) {
+			DLB_HW_ERR(hw,
+				   "[%s()] Internal error: unable to find queue %d\n",
+				   __func__, qid);
+			continue;
+		}
+
+		infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(qid));
+
+		if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT))
+			continue;
+
+		/*
+		 * Disable the affected CQ, and the CQs already mapped to the
+		 * QID, before reading the QID's inflight count a second time.
+		 * There is an unlikely race in which the QID may schedule one
+		 * more QE after we read an inflight count of 0, and disabling
+		 * the CQs guarantees that the race will not occur after a
+		 * re-read of the inflight count register.
+		 */
+		if (port->enabled)
+			dlb_ldb_port_cq_disable(hw, port);
+
+		dlb_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+		infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(qid));
+
+		if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+			if (port->enabled)
+				dlb_ldb_port_cq_enable(hw, port);
+
+			dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+			continue;
+		}
+
+		dlb_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+	}
+}
+
+static unsigned int
+dlb_domain_finish_map_qid_procedures(struct dlb_hw *hw,
+				     struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	if (!domain->configured || domain->num_pending_additions == 0)
+		return 0;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list)
+			dlb_domain_finish_map_port(hw, domain, port);
+	}
+
+	return domain->num_pending_additions;
+}
+
 static void dlb_log_map_qid(struct dlb_hw *hw, u32 domain_id,
 			    struct dlb_map_qid_args *args,
 			    bool vdev_req, unsigned int vdev_id)
@@ -4454,6 +4841,10 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	if (ret)
 		return ret;
 
+	ret = dlb_domain_finish_map_qid_procedures(hw, domain);
+	if (ret)
+		return ret;
+
 	/* Re-enable the CQs in order to drain the mapped queues. */
 	dlb_domain_enable_ldb_cqs(hw, domain);
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 19/20] dlb: add queue unmap register operations
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (17 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 18/20] dlb: add dynamic " Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-02-10 17:54 ` [PATCH v10 20/20] dlb: queue map/unmap workqueue Mike Ximing Chen
  2021-03-10  9:02 ` [PATCH v10 00/20] dlb: introduce DLB device driver Greg KH
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add the "dynamic" unmap procedure and associated register operations.
Unmapping a load-balanced queue from a port removes that port from the
queue's load-balancing candidates. If a queue unmap is requested after the
domain is started, the driver must disable the requested queue and wait for
it to quiesce before mapping it to the requested port.

Add the code to drain unmapped queues during domain reset. This consists of
mapping a port to the queue, then calling the function to drain a mapped
queue.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_resource.c | 289 ++++++++++++++++++++++++++++++++
 1 file changed, 289 insertions(+)

diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 93a3de642024..f4bd2049557a 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -2320,6 +2320,28 @@ static int dlb_ldb_port_set_has_work_bits(struct dlb_hw *hw,
 	return 0;
 }
 
+static void dlb_ldb_port_clear_has_work_bits(struct dlb_hw *hw,
+					     struct dlb_ldb_port *port, u8 slot)
+{
+	u32 ctrl = 0;
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_RLIST_HASWORK_V;
+
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	memset(&ctrl, 0, sizeof(ctrl));
+
+	BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+	BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+	ctrl |= LSP_LDB_SCHED_CTRL_NALB_HASWORK_V;
+
+	DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+	dlb_flush_csr(hw);
+}
+
 static void dlb_ldb_port_clear_queue_if_status(struct dlb_hw *hw,
 					       struct dlb_ldb_port *port,
 					       int slot)
@@ -2579,6 +2601,87 @@ static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
 		return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
 }
 
+static int dlb_ldb_port_unmap_qid(struct dlb_hw *hw, struct dlb_ldb_port *port,
+				  struct dlb_ldb_queue *queue)
+{
+	enum dlb_qid_map_state mapped, in_progress, pending_map, unmapped;
+	u32 lsp_qid2cq2;
+	u32 lsp_qid2cq;
+	u32 atm_qid2cq;
+	u32 cq2priov;
+	u32 queue_id;
+	u32 port_id;
+	int i;
+
+	/* Find the queue's slot */
+	mapped = DLB_QUEUE_MAPPED;
+	in_progress = DLB_QUEUE_UNMAP_IN_PROG;
+	pending_map = DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+	if (!dlb_port_find_slot_queue(port, mapped, queue, &i) &&
+	    !dlb_port_find_slot_queue(port, in_progress, queue, &i) &&
+	    !dlb_port_find_slot_queue(port, pending_map, queue, &i)) {
+		DLB_HW_ERR(hw,
+			   "[%s():%d] Internal error: QID %d isn't mapped\n",
+			   __func__, __LINE__, queue->id.phys_id);
+		return -EFAULT;
+	}
+
+	port_id = port->id.phys_id;
+	queue_id = queue->id.phys_id;
+
+	/* Read-modify-write the priority and valid bit register */
+	cq2priov = DLB_CSR_RD(hw, LSP_CQ2PRIOV(port_id));
+
+	cq2priov &= ~(1 << (i + LSP_CQ2PRIOV_V_LOC));
+
+	DLB_CSR_WR(hw, LSP_CQ2PRIOV(port_id), cq2priov);
+
+	atm_qid2cq = DLB_CSR_RD(hw, ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+	lsp_qid2cq = DLB_CSR_RD(hw, LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+	lsp_qid2cq2 = DLB_CSR_RD(hw, LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+	switch (port_id % 4) {
+	case 0:
+		atm_qid2cq &= ~(1 << (i + ATM_QID2CQIDIX_00_CQ_P0_LOC));
+		lsp_qid2cq &= ~(1 << (i + LSP_QID2CQIDIX_00_CQ_P0_LOC));
+		lsp_qid2cq2 &= ~(1 << (i + LSP_QID2CQIDIX2_00_CQ_P0_LOC));
+		break;
+
+	case 1:
+		atm_qid2cq &= ~(1 << (i + ATM_QID2CQIDIX_00_CQ_P1_LOC));
+		lsp_qid2cq &= ~(1 << (i + LSP_QID2CQIDIX_00_CQ_P1_LOC));
+		lsp_qid2cq2 &= ~(1 << (i + LSP_QID2CQIDIX2_00_CQ_P1_LOC));
+		break;
+
+	case 2:
+		atm_qid2cq &= ~(1 << (i + ATM_QID2CQIDIX_00_CQ_P2_LOC));
+		lsp_qid2cq &= ~(1 << (i + LSP_QID2CQIDIX_00_CQ_P2_LOC));
+		lsp_qid2cq2 &= ~(1 << (i + LSP_QID2CQIDIX2_00_CQ_P2_LOC));
+		break;
+
+	case 3:
+		atm_qid2cq &= ~(1 << (i + ATM_QID2CQIDIX_00_CQ_P3_LOC));
+		lsp_qid2cq &= ~(1 << (i + LSP_QID2CQIDIX_00_CQ_P3_LOC));
+		lsp_qid2cq2 &= ~(1 << (i + LSP_QID2CQIDIX2_00_CQ_P3_LOC));
+		break;
+	}
+
+	DLB_CSR_WR(hw, ATM_QID2CQIDIX(queue_id, port_id / 4), atm_qid2cq);
+
+	DLB_CSR_WR(hw, LSP_QID2CQIDIX(queue_id, port_id / 4), lsp_qid2cq);
+
+	DLB_CSR_WR(hw, LSP_QID2CQIDIX2(queue_id, port_id / 4), lsp_qid2cq2);
+
+	dlb_flush_csr(hw);
+
+	unmapped = DLB_QUEUE_UNMAPPED;
+
+	return dlb_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
 static void
 dlb_log_create_sched_domain_args(struct dlb_hw *hw,
 				 struct dlb_create_sched_domain_args *args,
@@ -3033,6 +3136,106 @@ int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
 	return 0;
 }
 
+static void
+dlb_domain_finish_unmap_port_slot(struct dlb_hw *hw,
+				  struct dlb_hw_domain *domain,
+				  struct dlb_ldb_port *port, int slot)
+{
+	enum dlb_qid_map_state state;
+	struct dlb_ldb_queue *queue;
+
+	queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+	state = port->qid_map[slot].state;
+
+	/* Update the QID2CQIDX and CQ2QID vectors */
+	dlb_ldb_port_unmap_qid(hw, port, queue);
+
+	/*
+	 * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+	 * the has_work bits
+	 */
+	dlb_ldb_port_clear_has_work_bits(hw, port, slot);
+
+	/* Reset the {CQ, slot} to its default state */
+	dlb_ldb_port_set_queue_if_status(hw, port, slot);
+
+	/* Re-enable the CQ if it wasn't manually disabled by the user */
+	if (port->enabled)
+		dlb_ldb_port_cq_enable(hw, port);
+
+	/*
+	 * If there is a mapping that is pending this slot's removal, perform
+	 * the mapping now.
+	 */
+	if (state == DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+		struct dlb_ldb_port_qid_map *map;
+		struct dlb_ldb_queue *map_queue;
+		u8 prio;
+
+		map = &port->qid_map[slot];
+
+		map->qid = map->pending_qid;
+		map->priority = map->pending_priority;
+
+		map_queue = &hw->rsrcs.ldb_queues[map->qid];
+		prio = map->priority;
+
+		dlb_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+	}
+}
+
+static bool dlb_domain_finish_unmap_port(struct dlb_hw *hw,
+					 struct dlb_hw_domain *domain,
+					 struct dlb_ldb_port *port)
+{
+	u32 infl_cnt;
+	int i;
+
+	if (port->num_pending_removals == 0)
+		return false;
+
+	/*
+	 * The unmap requires all the CQ's outstanding inflights to be
+	 * completed.
+	 */
+	infl_cnt = DLB_CSR_RD(hw, LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+	if (BITS_GET(infl_cnt, LSP_CQ_LDB_INFL_CNT_COUNT) > 0)
+		return false;
+
+	for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+		struct dlb_ldb_port_qid_map *map;
+
+		map = &port->qid_map[i];
+
+		if (map->state != DLB_QUEUE_UNMAP_IN_PROG &&
+		    map->state != DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+			continue;
+
+		dlb_domain_finish_unmap_port_slot(hw, domain, port, i);
+	}
+
+	return true;
+}
+
+static unsigned int
+dlb_domain_finish_unmap_qid_procedures(struct dlb_hw *hw,
+				       struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_port *port;
+	int i;
+
+	if (!domain->configured || domain->num_pending_removals == 0)
+		return 0;
+
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list)
+			dlb_domain_finish_unmap_port(hw, domain, port);
+	}
+
+	return domain->num_pending_removals;
+}
+
 static void dlb_domain_finish_map_port(struct dlb_hw *hw,
 				       struct dlb_hw_domain *domain,
 				       struct dlb_ldb_port *port)
@@ -3197,6 +3400,9 @@ int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
 	 * attempt to complete them. This may be necessary to free up a QID
 	 * slot for this requested mapping.
 	 */
+	if (port->num_pending_removals)
+		dlb_domain_finish_unmap_port(hw, domain, port);
+
 	ret = dlb_verify_map_qid_slot_available(port, queue, resp);
 	if (ret)
 		return ret;
@@ -3440,6 +3646,13 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 	if (ret)
 		return ret;
 
+	/*
+	 * Attempt to finish the unmapping now, in case the port has no
+	 * outstanding inflights. If that's not the case, this will fail and
+	 * the unmapping will be completed at a later time.
+	 */
+	dlb_domain_finish_unmap_port(hw, domain, port);
+
 unmap_qid_done:
 	resp->status = 0;
 
@@ -4539,6 +4752,74 @@ static int dlb_domain_drain_mapped_queues(struct dlb_hw *hw,
 	return 0;
 }
 
+static int dlb_domain_drain_unmapped_queue(struct dlb_hw *hw,
+					   struct dlb_hw_domain *domain,
+					   struct dlb_ldb_queue *queue)
+{
+	struct dlb_ldb_port *port = NULL;
+	int ret, i;
+
+	/* If a domain has LDB queues, it must have LDB ports */
+	for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+		port = list_first_entry_or_null(&domain->used_ldb_ports[i],
+						typeof(*port), domain_list);
+		if (port)
+			break;
+	}
+
+	if (!port) {
+		DLB_HW_ERR(hw,
+			   "[%s()] Internal error: No configured LDB ports\n",
+			   __func__);
+		return -EFAULT;
+	}
+
+	/* If necessary, free up a QID slot in this CQ */
+	if (port->num_mappings == DLB_MAX_NUM_QIDS_PER_LDB_CQ) {
+		struct dlb_ldb_queue *mapped_queue;
+
+		mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+		ret = dlb_ldb_port_unmap_qid(hw, port, mapped_queue);
+		if (ret)
+			return ret;
+	}
+
+	ret = dlb_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+	if (ret)
+		return ret;
+
+	return dlb_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb_domain_drain_unmapped_queues(struct dlb_hw *hw,
+					    struct dlb_hw_domain *domain)
+{
+	struct dlb_ldb_queue *queue;
+	int ret;
+
+	/* If the domain hasn't been started, there's no traffic to drain */
+	if (!domain->started)
+		return 0;
+
+	/*
+	 * Pre-condition: the unattached queue must not have any outstanding
+	 * completions. This is ensured by calling dlb_domain_drain_ldb_cqs()
+	 * prior to this in dlb_domain_drain_mapped_queues().
+	 */
+	list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+		if (queue->num_mappings != 0 ||
+		    dlb_ldb_queue_is_empty(hw, queue))
+			continue;
+
+		ret = dlb_domain_drain_unmapped_queue(hw, domain, queue);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 static int dlb_drain_dir_cq(struct dlb_hw *hw, struct dlb_dir_pq_pair *port)
 {
 	unsigned int port_id = port->id.phys_id;
@@ -4841,6 +5122,10 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	if (ret)
 		return ret;
 
+	ret = dlb_domain_finish_unmap_qid_procedures(hw, domain);
+	if (ret)
+		return ret;
+
 	ret = dlb_domain_finish_map_qid_procedures(hw, domain);
 	if (ret)
 		return ret;
@@ -4852,6 +5137,10 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
 	if (ret)
 		return ret;
 
+	ret = dlb_domain_drain_unmapped_queues(hw, domain);
+	if (ret)
+		return ret;
+
 	/* Done draining LDB QEs, so disable the CQs. */
 	dlb_domain_disable_ldb_cqs(hw, domain);
 
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* [PATCH v10 20/20] dlb: queue map/unmap workqueue
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (18 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 19/20] dlb: add queue unmap " Mike Ximing Chen
@ 2021-02-10 17:54 ` Mike Ximing Chen
  2021-03-10  9:02 ` [PATCH v10 00/20] dlb: introduce DLB device driver Greg KH
  20 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-02-10 17:54 UTC (permalink / raw)
  To: netdev
  Cc: davem, kuba, arnd, gregkh, dan.j.williams, pierre-louis.bossart,
	Gage Eads

Add workqueue support for asynchronous queue map/unmap operations.

When mapping or unmapped a queue, it's possible that the thread that
requires the map/unmap is the same one which is responsible for doing the
processing that would quiesce the queue/port, thus the driver may have to
complete the operation asynchronously.

To that end, the driver uses a workqueue that periodically checks whether
any outstanding operations can be completed. This workqueue function is
only scheduled when there is at least one outstanding map/unmap operation.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/misc/dlb/dlb_main.c     |   5 ++
 drivers/misc/dlb/dlb_main.h     |   2 +
 drivers/misc/dlb/dlb_resource.c | 110 +++++++++++++++++++++++++++++++-
 drivers/misc/dlb/dlb_resource.h |   2 +
 4 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 69ab9b532ed4..678be16d103d 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -440,6 +440,9 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
 
 	dlb->ops->init_hardware(dlb);
 
+	/* Initialize a workqueue for queue-port linking. */
+	dlb_init_work(dlb);
+
 	/*
 	 * Undo the 'get' operation by the PCI layer during probe and
 	 * (if PF) immediately suspend the device. Since the device is only
@@ -596,6 +599,8 @@ static int __init dlb_init_module(void)
 
 static void __exit dlb_exit_module(void)
 {
+	flush_scheduled_work();
+
 	pci_unregister_driver(&dlb_pci_driver);
 
 	cdev_del(&dlb_cdev);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 5942fbf22cbb..097b070b98fa 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -140,11 +140,13 @@ struct dlb {
 	 * hardware registers.
 	 */
 	struct mutex resource_mutex;
+	struct work_struct work;
 	enum dlb_device_type type;
 	int id;
 	u32 inode_cnt;
 	dev_t dev_number;
 	u8 domain_reset_failed;
+	u8 worker_launched;
 };
 
 /* Prototypes for dlb_ioctl.c */
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index f4bd2049557a..084ff6857f15 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -2509,6 +2509,61 @@ static int dlb_ldb_port_finish_map_qid_dynamic(struct dlb_hw *hw,
 	return 0;
 }
 
+static unsigned int dlb_finish_unmap_qid_procedures(struct dlb_hw *hw);
+static unsigned int dlb_finish_map_qid_procedures(struct dlb_hw *hw);
+
+/*
+ * The workqueue callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void dlb_complete_queue_map_unmap(struct work_struct *work)
+{
+	struct dlb *dlb = container_of(work, struct dlb, work);
+	int ret;
+
+	mutex_lock(&dlb->resource_mutex);
+
+	ret = dlb_finish_unmap_qid_procedures(&dlb->hw);
+	ret += dlb_finish_map_qid_procedures(&dlb->hw);
+
+	if (ret != 0)
+		/*
+		 * Relinquish the CPU so the application can process its CQs,
+		 * so this function doesn't deadlock.
+		 */
+		schedule_work(&dlb->work);
+	else
+		dlb->worker_launched = false;
+
+	mutex_unlock(&dlb->resource_mutex);
+}
+
+void dlb_init_work(struct dlb *dlb)
+{
+	INIT_WORK(&dlb->work, dlb_complete_queue_map_unmap);
+}
+
+/**
+ * dlb_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static void dlb_schedule_work(struct dlb_hw *hw)
+{
+	struct dlb *dlb = container_of(hw, struct dlb, hw);
+
+	/* Nothing to do if the worker is already running */
+	if (dlb->worker_launched)
+		return;
+
+	schedule_work(&dlb->work);
+
+	dlb->worker_launched = true;
+}
+
 /**
  * dlb_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
  * @hw: dlb_hw handle for a particular device.
@@ -2561,8 +2616,16 @@ static int dlb_ldb_port_map_qid_dynamic(struct dlb_hw *hw,
 
 	infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
 
-	if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT))
+	if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		dlb_schedule_work(hw);
+
 		return 1;
+	}
 
 	/*
 	 * Disable the affected CQ, and the CQs already mapped to the QID,
@@ -2585,6 +2648,13 @@ static int dlb_ldb_port_map_qid_dynamic(struct dlb_hw *hw,
 
 		dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
 
+		/*
+		 * The queue is owed completions so it's not safe to map it
+		 * yet. Schedule a kernel thread to complete the mapping later,
+		 * once software has completed all the queue's inflight events.
+		 */
+		dlb_schedule_work(hw);
+
 		return 1;
 	}
 
@@ -3236,6 +3306,20 @@ dlb_domain_finish_unmap_qid_procedures(struct dlb_hw *hw,
 	return domain->num_pending_removals;
 }
 
+static unsigned int dlb_finish_unmap_qid_procedures(struct dlb_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue unmap jobs for any domain that needs it */
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+		struct dlb_hw_domain *domain = &hw->domains[i];
+
+		num += dlb_domain_finish_unmap_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
 static void dlb_domain_finish_map_port(struct dlb_hw *hw,
 				       struct dlb_hw_domain *domain,
 				       struct dlb_ldb_port *port)
@@ -3312,6 +3396,20 @@ dlb_domain_finish_map_qid_procedures(struct dlb_hw *hw,
 	return domain->num_pending_additions;
 }
 
+static unsigned int dlb_finish_map_qid_procedures(struct dlb_hw *hw)
+{
+	int i, num = 0;
+
+	/* Finish queue map jobs for any domain that needs it */
+	for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+		struct dlb_hw_domain *domain = &hw->domains[i];
+
+		num += dlb_domain_finish_map_qid_procedures(hw, domain);
+	}
+
+	return num;
+}
+
 static void dlb_log_map_qid(struct dlb_hw *hw, u32 domain_id,
 			    struct dlb_map_qid_args *args,
 			    bool vdev_req, unsigned int vdev_id)
@@ -3577,6 +3675,7 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 	struct dlb_ldb_queue *queue;
 	enum dlb_qid_map_state st;
 	struct dlb_ldb_port *port;
+	bool unmap_complete;
 	int i, ret;
 
 	dlb_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
@@ -3651,7 +3750,14 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
 	 * outstanding inflights. If that's not the case, this will fail and
 	 * the unmapping will be completed at a later time.
 	 */
-	dlb_domain_finish_unmap_port(hw, domain, port);
+	unmap_complete = dlb_domain_finish_unmap_port(hw, domain, port);
+
+	/*
+	 * If the unmapping couldn't complete immediately, launch the worker
+	 * thread (if it isn't already launched) to finish it later.
+	 */
+	if (!unmap_complete)
+		dlb_schedule_work(hw);
 
 unmap_qid_done:
 	resp->status = 0;
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index e3de9eb94d5d..82f14388581b 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -89,4 +89,6 @@ void dlb_hw_enable_sparse_ldb_cq_mode(struct dlb_hw *hw);
 
 void dlb_hw_enable_sparse_dir_cq_mode(struct dlb_hw *hw);
 
+void dlb_init_work(struct dlb *dlb);
+
 #endif /* __DLB_RESOURCE_H */
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
  2021-02-10 17:54 ` [PATCH v10 01/20] dlb: add skeleton for DLB driver Mike Ximing Chen
@ 2021-02-18  7:34   ` Chen, Mike Ximing
  2021-02-18  7:52     ` gregkh
  0 siblings, 1 reply; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-02-18  7:34 UTC (permalink / raw)
  To: Chen, Mike Ximing, netdev, Linux Kernel Mailing List
  Cc: davem, kuba, arnd, gregkh, Williams, Dan J, pierre-louis.bossart



> -----Original Message-----
> From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> Sent: Wednesday, February 10, 2021 12:54 PM
> To: netdev@vger.kernel.org
> Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>; pierre-
> louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> devices/dlb.rst
> new file mode 100644
> index 000000000000..aa79be07ee49
> --- /dev/null
> +++ b/Documentation/misc-devices/dlb.rst
> @@ -0,0 +1,259 @@
> +.. SPDX-License-Identifier: GPL-2.0-only
> +
> +===========================================
> +Intel(R) Dynamic Load Balancer Overview
> +===========================================
> +
> +:Authors: Gage Eads and Mike Ximing Chen
> +
> +Contents
> +========
> +
> +- Introduction
> +- Scheduling
> +- Queue Entry
> +- Port
> +- Queue
> +- Credits
> +- Scheduling Domain
> +- Interrupts
> +- Power Management
> +- User Interface
> +- Reset
> +
> +Introduction
> +============
> +
> +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> +provides load-balanced, prioritized scheduling of core-to-core communication.
> +
> +Intel DLB is an accelerator for the event-driven programming model of
> +DPDK's Event Device Library[2]. The library is used in packet processing
> +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> +variety of packet distribution and synchronization schemes.
> +
> +Intel DLB device consists of queues and arbiters that connect producer
> +cores and consumer cores. The device implements load-balanced queueing
> features
> +including:
> +- Lock-free multi-producer/multi-consumer operation.
> +- Multiple priority levels for varying traffic types.
> +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> +- Simple unordered load-balanced distribution.
> +- Atomic lock free load balancing across multiple consumers.
> +- Queue element reordering feature allowing ordered load-balanced distribution.
> +

Hi Jakub/Dave,
This is a device driver for a HW core-to-core communication accelerator. It is submitted 
to "linux-kernel" for a module under device/misc. Greg suggested (see below) that we
also sent it to you for any potential feedback in case there is any interaction with
networking initiatives. The device is used to handle the load balancing among CPU cores
after the packets are received and forwarded to CPU. We don't think it interferes
with networking operations, but would appreciate very much your review/comment on this.
 
Thanks for your help.
Mike

> As this is a networking related thing, I would like you to get the
>proper reviews/acks from the networking maintainers before I can take
>this.
>
>Or, if they think it has nothing to do with networking, that's fine too,
>but please do not try to route around them.
>
>thanks,
>
>greg k-h


^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
  2021-02-18  7:34   ` Chen, Mike Ximing
@ 2021-02-18  7:52     ` gregkh
  2021-02-18 15:37       ` Chen, Mike Ximing
  2021-03-07 13:59       ` Chen, Mike Ximing
  0 siblings, 2 replies; 50+ messages in thread
From: gregkh @ 2021-02-18  7:52 UTC (permalink / raw)
  To: Chen, Mike Ximing
  Cc: netdev, Linux Kernel Mailing List, davem, kuba, arnd, Williams,
	Dan J, pierre-louis.bossart

On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> 
> 
> > -----Original Message-----
> > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > Sent: Wednesday, February 10, 2021 12:54 PM
> > To: netdev@vger.kernel.org
> > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>; pierre-
> > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > 
> > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > devices/dlb.rst
> > new file mode 100644
> > index 000000000000..aa79be07ee49
> > --- /dev/null
> > +++ b/Documentation/misc-devices/dlb.rst
> > @@ -0,0 +1,259 @@
> > +.. SPDX-License-Identifier: GPL-2.0-only
> > +
> > +===========================================
> > +Intel(R) Dynamic Load Balancer Overview
> > +===========================================
> > +
> > +:Authors: Gage Eads and Mike Ximing Chen
> > +
> > +Contents
> > +========
> > +
> > +- Introduction
> > +- Scheduling
> > +- Queue Entry
> > +- Port
> > +- Queue
> > +- Credits
> > +- Scheduling Domain
> > +- Interrupts
> > +- Power Management
> > +- User Interface
> > +- Reset
> > +
> > +Introduction
> > +============
> > +
> > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > +
> > +Intel DLB is an accelerator for the event-driven programming model of
> > +DPDK's Event Device Library[2]. The library is used in packet processing
> > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > +variety of packet distribution and synchronization schemes.
> > +
> > +Intel DLB device consists of queues and arbiters that connect producer
> > +cores and consumer cores. The device implements load-balanced queueing
> > features
> > +including:
> > +- Lock-free multi-producer/multi-consumer operation.
> > +- Multiple priority levels for varying traffic types.
> > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > +- Simple unordered load-balanced distribution.
> > +- Atomic lock free load balancing across multiple consumers.
> > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > +
> 
> Hi Jakub/Dave,
> This is a device driver for a HW core-to-core communication accelerator. It is submitted 
> to "linux-kernel" for a module under device/misc. Greg suggested (see below) that we
> also sent it to you for any potential feedback in case there is any interaction with
> networking initiatives. The device is used to handle the load balancing among CPU cores
> after the packets are received and forwarded to CPU. We don't think it interferes
> with networking operations, but would appreciate very much your review/comment on this.

It's the middle of the merge window, getting maintainers to review new
stuff until after 5.12-rc1 is out is going to be a very difficult thing
to do.

In the meantime, why don't you all help out and review submitted patches
to the mailing lists for the subsystems you all are trying to get this
patch into.  I know maintainers would appreciate the help, right?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
  2021-02-18  7:52     ` gregkh
@ 2021-02-18 15:37       ` Chen, Mike Ximing
  2021-03-07 13:59       ` Chen, Mike Ximing
  1 sibling, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-02-18 15:37 UTC (permalink / raw)
  To: gregkh
  Cc: netdev, Linux Kernel Mailing List, davem, kuba, arnd, Williams,
	Dan J, pierre-louis.bossart



> -----Original Message-----
> From: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
> Sent: Thursday, February 18, 2021 2:53 AM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> Williams, Dan J <dan.j.williams@intel.com>; pierre-louis.bossart@linux.intel.com
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Sent: Wednesday, February 10, 2021 12:54 PM
> > > To: netdev@vger.kernel.org
> > > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>;
> pierre-
> > > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > > devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > > +variety of packet distribution and synchronization schemes.
> > > +
> > > +Intel DLB device consists of queues and arbiters that connect producer
> > > +cores and consumer cores. The device implements load-balanced queueing
> > > features
> > > +including:
> > > +- Lock-free multi-producer/multi-consumer operation.
> > > +- Multiple priority levels for varying traffic types.
> > > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > > +- Simple unordered load-balanced distribution.
> > > +- Atomic lock free load balancing across multiple consumers.
> > > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > > +
> >
> > Hi Jakub/Dave,
> > This is a device driver for a HW core-to-core communication accelerator. It is
> submitted
> > to "linux-kernel" for a module under device/misc. Greg suggested (see below) that
> we
> > also sent it to you for any potential feedback in case there is any interaction with
> > networking initiatives. The device is used to handle the load balancing among CPU
> cores
> > after the packets are received and forwarded to CPU. We don't think it interferes
> > with networking operations, but would appreciate very much your
> review/comment on this.
> 
> It's the middle of the merge window, getting maintainers to review new
> stuff until after 5.12-rc1 is out is going to be a very difficult thing
> to do.
> 
> In the meantime, why don't you all help out and review submitted patches
> to the mailing lists for the subsystems you all are trying to get this
> patch into.  I know maintainers would appreciate the help, right?
> 
> thanks,
> 
> greg k-h

Sure. I am a little new to the community and process, but will try to help.
Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
  2021-02-18  7:52     ` gregkh
  2021-02-18 15:37       ` Chen, Mike Ximing
@ 2021-03-07 13:59       ` Chen, Mike Ximing
  1 sibling, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-07 13:59 UTC (permalink / raw)
  To: gregkh
  Cc: netdev, Linux Kernel Mailing List, davem, kuba, arnd, Williams,
	Dan J, pierre-louis.bossart, Brandeburg, Jesse



> -----Original Message-----
> From: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
> Sent: Thursday, February 18, 2021 2:53 AM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> Williams, Dan J <dan.j.williams@intel.com>; pierre-louis.bossart@linux.intel.com
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Sent: Wednesday, February 10, 2021 12:54 PM
> > > To: netdev@vger.kernel.org
> > > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>;
> pierre-
> > > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > > devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > > +variety of packet distribution and synchronization schemes.
> > > +
> > > +Intel DLB device consists of queues and arbiters that connect producer
> > > +cores and consumer cores. The device implements load-balanced queueing
> > > features
> > > +including:
> > > +- Lock-free multi-producer/multi-consumer operation.
> > > +- Multiple priority levels for varying traffic types.
> > > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > > +- Simple unordered load-balanced distribution.
> > > +- Atomic lock free load balancing across multiple consumers.
> > > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > > +
> >
> > Hi Jakub/Dave,
> > This is a device driver for a HW core-to-core communication accelerator. It is
> submitted
> > to "linux-kernel" for a module under device/misc. Greg suggested (see below) that
> we
> > also sent it to you for any potential feedback in case there is any interaction with
> > networking initiatives. The device is used to handle the load balancing among CPU
> cores
> > after the packets are received and forwarded to CPU. We don't think it interferes
> > with networking operations, but would appreciate very much your
> review/comment on this.
> 
> It's the middle of the merge window, getting maintainers to review new
> stuff until after 5.12-rc1 is out is going to be a very difficult thing
> to do.
> 

Hi Jakub/Dave,
Just wonder if you had a chance to take a look at our patch. With the close of 5.12
merge window, we would like to get the process moving again.

> In the meantime, why don't you all help out and review submitted patches
> to the mailing lists for the subsystems you all are trying to get this
> patch into.  I know maintainers would appreciate the help, right?
> 
> thanks,
> 
> greg k-h

Did a few reviews last weekend, and will continue to help.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 03/20] dlb: add resource and device initialization
  2021-02-10 17:54 ` [PATCH v10 03/20] dlb: add resource and device initialization Mike Ximing Chen
@ 2021-03-09  9:24   ` Greg KH
  2021-03-10  1:33     ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-09  9:24 UTC (permalink / raw)
  To: Mike Ximing Chen
  Cc: netdev, davem, kuba, arnd, dan.j.williams, pierre-louis.bossart,
	Gage Eads

On Wed, Feb 10, 2021 at 11:54:06AM -0600, Mike Ximing Chen wrote:
> --- a/drivers/misc/dlb/dlb_hw_types.h
> +++ b/drivers/misc/dlb/dlb_hw_types.h
> @@ -5,6 +5,13 @@
>  #define __DLB_HW_TYPES_H
>  
>  #include <linux/io.h>
> +#include <linux/types.h>
> +
> +#include "dlb_bitmap.h"
> +
> +#define BITS_SET(x, val, mask)	(x = ((x) & ~(mask))     \
> +				 | (((val) << (mask##_LOC)) & (mask)))
> +#define BITS_GET(x, mask)       (((x) & (mask)) >> (mask##_LOC))

Why not use the built-in kernel functions for this?  Why are you
creating your own?



> -static void
> -dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
> +static void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)

Why reformat code here, and not do it right the first time around?

>  {
>  	pcim_iounmap(pdev, dlb->hw.csr_kva);
>  	pcim_iounmap(pdev, dlb->hw.func_kva);
>  }
>  
> -static int
> -dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
> +static int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)

Same here.

>  {
>  	dlb->hw.func_kva = pcim_iomap_table(pdev)[DLB_FUNC_BAR];
>  	dlb->hw.func_phys_addr = pci_resource_start(pdev, DLB_FUNC_BAR);
> @@ -40,6 +42,59 @@ dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
>  	return 0;
>  }
>  
> +/*******************************/
> +/****** Driver management ******/
> +/*******************************/
> +
> +static int dlb_pf_init_driver_state(struct dlb *dlb)
> +{
> +	mutex_init(&dlb->resource_mutex);
> +
> +	return 0;

If this can not fail, why is this not just void?

> diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
> new file mode 100644
> index 000000000000..2229813d9c45
> --- /dev/null
> +++ b/drivers/misc/dlb/dlb_resource.h

Why do you have lots of little .h files and not just one simple .h file
for the driver?  That makes it much easier to maintain over time, right?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls
  2021-02-10 17:54 ` [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls Mike Ximing Chen
@ 2021-03-09  9:26   ` Greg KH
  2021-03-10  1:34     ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-09  9:26 UTC (permalink / raw)
  To: Mike Ximing Chen
  Cc: netdev, davem, kuba, arnd, dan.j.williams, pierre-louis.bossart,
	Gage Eads

On Wed, Feb 10, 2021 at 11:54:07AM -0600, Mike Ximing Chen wrote:
> +/*
> + * DLB_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
> + *	its hardware resources. This command returns the newly created domain
> + *	ID and a file descriptor for accessing the domain.
> + *
> + * Output parameters:
> + * @response.status: Detailed error code. In certain cases, such as if the
> + *	ioctl request arg is invalid, the driver won't set status.
> + * @response.id: domain ID.
> + * @domain_fd: file descriptor for performing the domain's ioctl operations
> + * @padding0: Reserved for future use.

For all of your paddingX fields, you MUST check that they are set to 0
by userspace now, otherwise they will never be able to actually be used
in the future.

But why do you need them at all anyway?  If something needs to be
changed in the future, just add a new ioctl, don't modify something that
is already working.  Are you _SURE_ you need this type of functionality?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 05/20] dlb: add scheduling domain configuration
  2021-02-10 17:54 ` [PATCH v10 05/20] dlb: add scheduling domain configuration Mike Ximing Chen
@ 2021-03-09  9:28   ` Greg KH
  2021-03-10  1:35     ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-09  9:28 UTC (permalink / raw)
  To: Mike Ximing Chen
  Cc: netdev, davem, kuba, arnd, dan.j.williams, pierre-louis.bossart,
	Gage Eads

On Wed, Feb 10, 2021 at 11:54:08AM -0600, Mike Ximing Chen wrote:
> +/**
> + * dlb_bitmap_clear_range() - clear a range of bitmap entries
> + * @bitmap: pointer to dlb_bitmap structure.
> + * @bit: starting bit index.
> + * @len: length of the range.
> + *
> + * Return:
> + * Returns 0 upon success, < 0 otherwise.
> + *
> + * Errors:
> + * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
> + *	    length.
> + */
> +static inline int dlb_bitmap_clear_range(struct dlb_bitmap *bitmap,
> +					 unsigned int bit,
> +					 unsigned int len)
> +{
> +	if (!bitmap || !bitmap->map)
> +		return -EINVAL;
> +
> +	if (bitmap->len <= bit)
> +		return -EINVAL;
> +
> +	bitmap_clear(bitmap->map, bit, len);
> +
> +	return 0;
> +}

Why isn't logic like this just added to the lib/bitmap.c file?

> +/**
> + * dlb_bitmap_find_set_bit_range() - find an range of set bits
> + * @bitmap: pointer to dlb_bitmap structure.
> + * @len: length of the range.
> + *
> + * This function looks for a range of set bits of length @len.
> + *
> + * Return:
> + * Returns the base bit index upon success, < 0 otherwise.
> + *
> + * Errors:
> + * ENOENT - unable to find a length *len* range of set bits.
> + * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
> + */
> +static inline int dlb_bitmap_find_set_bit_range(struct dlb_bitmap *bitmap,
> +						unsigned int len)
> +{
> +	struct dlb_bitmap *complement_mask = NULL;
> +	int ret;
> +
> +	if (!bitmap || !bitmap->map || len == 0)
> +		return -EINVAL;
> +
> +	if (bitmap->len < len)
> +		return -ENOENT;
> +
> +	ret = dlb_bitmap_alloc(&complement_mask, bitmap->len);
> +	if (ret)
> +		return ret;
> +
> +	bitmap_zero(complement_mask->map, complement_mask->len);
> +
> +	bitmap_complement(complement_mask->map, bitmap->map, bitmap->len);
> +
> +	ret = bitmap_find_next_zero_area(complement_mask->map,
> +					 complement_mask->len,
> +					 0,
> +					 len,
> +					 0);
> +
> +	dlb_bitmap_free(complement_mask);
> +
> +	/* No set bit range of length len? */
> +	return (ret >= (int)bitmap->len) ? -ENOENT : ret;
> +}

Same here, why not put this in the core kernel instead of a tiny random
driver like this?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 14/20] dlb: add start domain ioctl
  2021-02-10 17:54 ` [PATCH v10 14/20] dlb: add start domain ioctl Mike Ximing Chen
@ 2021-03-09  9:29   ` Greg KH
  2021-03-10  2:45     ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-09  9:29 UTC (permalink / raw)
  To: Mike Ximing Chen
  Cc: netdev, davem, kuba, arnd, dan.j.williams, pierre-louis.bossart,
	Gage Eads

On Wed, Feb 10, 2021 at 11:54:17AM -0600, Mike Ximing Chen wrote:
> Add ioctl to start a domain. Once a scheduling domain and its resources
> have been configured, this ioctl is called to allow the domain's ports to
> begin enqueueing to the device. Once started, the domain's resources cannot
> be configured again until after the domain is reset.
> 
> This ioctl instructs the DLB device to start load-balancing operations.
> It corresponds to rte_event_dev_start() function in DPDK' eventdev library.
> 
> Signed-off-by: Gage Eads <gage.eads@intel.com>
> Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> Reviewed-by: Björn Töpel <bjorn.topel@intel.com>
> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> ---
>  drivers/misc/dlb/dlb_ioctl.c    |   3 +
>  drivers/misc/dlb/dlb_main.h     |   4 ++
>  drivers/misc/dlb/dlb_pf_ops.c   |   9 +++
>  drivers/misc/dlb/dlb_resource.c | 116 ++++++++++++++++++++++++++++++++
>  drivers/misc/dlb/dlb_resource.h |   4 ++
>  include/uapi/linux/dlb.h        |  22 ++++++
>  6 files changed, 158 insertions(+)
> 
> diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
> index 6a311b969643..9b05344f03c8 100644
> --- a/drivers/misc/dlb/dlb_ioctl.c
> +++ b/drivers/misc/dlb/dlb_ioctl.c
> @@ -51,6 +51,7 @@ DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
>  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
>  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
>  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
> +DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
>  
>  /*
>   * Port creation ioctls don't use the callback template macro.
> @@ -322,6 +323,8 @@ long dlb_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
>  		return dlb_domain_ioctl_get_dir_port_pp_fd(dlb, dom, arg);
>  	case DLB_IOC_GET_DIR_PORT_CQ_FD:
>  		return dlb_domain_ioctl_get_dir_port_cq_fd(dlb, dom, arg);
> +	case DLB_IOC_START_DOMAIN:
> +		return dlb_domain_ioctl_start_domain(dlb, dom, arg);
>  	default:
>  		return -ENOTTY;
>  	}
> diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
> index 477974e1a178..2f3096a45b1e 100644
> --- a/drivers/misc/dlb/dlb_main.h
> +++ b/drivers/misc/dlb/dlb_main.h
> @@ -63,6 +63,10 @@ struct dlb_device_ops {
>  			       struct dlb_create_dir_port_args *args,
>  			       uintptr_t cq_dma_base,
>  			       struct dlb_cmd_response *resp);
> +	int (*start_domain)(struct dlb_hw *hw,
> +			    u32 domain_id,
> +			    struct dlb_start_domain_args *args,
> +			    struct dlb_cmd_response *resp);
>  	int (*get_num_resources)(struct dlb_hw *hw,
>  				 struct dlb_get_num_resources_args *args);
>  	int (*reset_domain)(struct dlb_hw *hw, u32 domain_id);
> diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
> index 02a188aa5a60..ce9d29b94a55 100644
> --- a/drivers/misc/dlb/dlb_pf_ops.c
> +++ b/drivers/misc/dlb/dlb_pf_ops.c
> @@ -160,6 +160,14 @@ dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
>  				       resp, false, 0);
>  }
>  
> +static int
> +dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
> +		    struct dlb_start_domain_args *args,
> +		    struct dlb_cmd_response *resp)
> +{
> +	return dlb_hw_start_domain(hw, id, args, resp, false, 0);
> +}
> +
>  static int dlb_pf_get_num_resources(struct dlb_hw *hw,
>  				    struct dlb_get_num_resources_args *args)
>  {
> @@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
>  	.create_dir_queue = dlb_pf_create_dir_queue,
>  	.create_ldb_port = dlb_pf_create_ldb_port,
>  	.create_dir_port = dlb_pf_create_dir_port,
> +	.start_domain = dlb_pf_start_domain,

Why do you have a "callback" when you only ever call one function?  Why
is that needed at all?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 03/20] dlb: add resource and device initialization
  2021-03-09  9:24   ` Greg KH
@ 2021-03-10  1:33     ` Chen, Mike Ximing
  2021-03-10  8:13       ` Greg KH
  0 siblings, 1 reply; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10  1:33 UTC (permalink / raw)
  To: Greg KH
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Gage Eads


> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> 
> On Wed, Feb 10, 2021 at 11:54:06AM -0600, Mike Ximing Chen wrote:
> > +
> > +#include "dlb_bitmap.h"
> > +
> > +#define BITS_SET(x, val, mask)	(x = ((x) & ~(mask))     \
> > +				 | (((val) << (mask##_LOC)) & (mask)))
> > +#define BITS_GET(x, mask)       (((x) & (mask)) >> (mask##_LOC))
> 
> Why not use the built-in kernel functions for this?  Why are you
> creating your own?
>
FIELD_GET(_mask, _val) and FIELD_PREP(_mask, _val) in include/linux/bitfield.h
are similar to our BITS_GET() and BITS_SET().  However in our case, mask##_LOC
is a known constant defined in dlb_regs.h,  so we don't need to use 
_buildin_ffs(mask) to calculate the location of mask as FIELD_GET() and FIELD_PREP()
do.  We can still use FIELD_GET and FIELD_PREP, but our macros are a little more 
efficient. Would it be OK to keep them?
 
> 
> 
> > -static void
> > -dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
> > +static void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
> 
> Why reformat code here, and not do it right the first time around?
> 
Sorry,  this should not happen. Will fix it.

> > +/*******************************/
> > +/****** Driver management ******/
> > +/*******************************/
> > +
> > +static int dlb_pf_init_driver_state(struct dlb *dlb)
> > +{
> > +	mutex_init(&dlb->resource_mutex);
> > +
> > +	return 0;
> 
> If this can not fail, why is this not just void?

Sure, will change it void.

> 
> > diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
> > new file mode 100644
> > index 000000000000..2229813d9c45
> > --- /dev/null
> > +++ b/drivers/misc/dlb/dlb_resource.h
> 
> Why do you have lots of little .h files and not just one simple .h file
> for the driver?  That makes it much easier to maintain over time, right?
> 
I combined a couple of header files in this version.  dlb_regs.h is pretty big (3640 lines), and is
generated by SW. I will merge other .h files into one.

Thanks
Mike 

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls
  2021-03-09  9:26   ` Greg KH
@ 2021-03-10  1:34     ` Chen, Mike Ximing
  0 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10  1:34 UTC (permalink / raw)
  To: Greg KH
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Brandeburg, Jesse

> From: Greg KH <gregkh@linuxfoundation.org>
> On Wed, Feb 10, 2021 at 11:54:07AM -0600, Mike Ximing Chen wrote:
> > +/*
> > + * DLB_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and
> reserve
> > + *	its hardware resources. This command returns the newly created domain
> > + *	ID and a file descriptor for accessing the domain.
> > + *
> > + * Output parameters:
> > + * @response.status: Detailed error code. In certain cases, such as if the
> > + *	ioctl request arg is invalid, the driver won't set status.
> > + * @response.id: domain ID.
> > + * @domain_fd: file descriptor for performing the domain's ioctl operations
> > + * @padding0: Reserved for future use.
> 
> For all of your paddingX fields, you MUST check that they are set to 0
> by userspace now, otherwise they will never be able to actually be used
> in the future.
> 
> But why do you need them at all anyway?  If something needs to be
> changed in the future, just add a new ioctl, don't modify something that
> is already working.  Are you _SURE_ you need this type of functionality?
> 
OK, we don't need the padding fields then, and will remove them.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 05/20] dlb: add scheduling domain configuration
  2021-03-09  9:28   ` Greg KH
@ 2021-03-10  1:35     ` Chen, Mike Ximing
  0 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10  1:35 UTC (permalink / raw)
  To: Greg KH
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Gage Eads



> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> 
> On Wed, Feb 10, 2021 at 11:54:08AM -0600, Mike Ximing Chen wrote:
> > +static inline int dlb_bitmap_clear_range(struct dlb_bitmap *bitmap,
> > +					 unsigned int bit,
> > +					 unsigned int len)
> > +{
> > +	if (!bitmap || !bitmap->map)
> > +		return -EINVAL;
> > +
> > +	if (bitmap->len <= bit)
> > +		return -EINVAL;
> > +
> > +	bitmap_clear(bitmap->map, bit, len);
> > +
> > +	return 0;
> > +}
> 
> Why isn't logic like this just added to the lib/bitmap.c file?
> 
> > +static inline int dlb_bitmap_find_set_bit_range(struct dlb_bitmap *bitmap,
> > +						unsigned int len)
> > +{
> > +	struct dlb_bitmap *complement_mask = NULL;
> > +	int ret;
> > +
> > +	if (!bitmap || !bitmap->map || len == 0)
> > +		return -EINVAL;
> > +
> > +	if (bitmap->len < len)
> > +		return -ENOENT;
> > +
> > +	ret = dlb_bitmap_alloc(&complement_mask, bitmap->len);
> > +	if (ret)
> > +		return ret;
> > +
> > +	bitmap_zero(complement_mask->map, complement_mask->len);
> > +
> > +	bitmap_complement(complement_mask->map, bitmap->map, bitmap->len);
> > +
> > +	ret = bitmap_find_next_zero_area(complement_mask->map,
> > +					 complement_mask->len,
> > +					 0,
> > +					 len,
> > +					 0);
> > +
> > +	dlb_bitmap_free(complement_mask);
> > +
> > +	/* No set bit range of length len? */
> > +	return (ret >= (int)bitmap->len) ? -ENOENT : ret;
> > +}
> 
> Same here, why not put this in the core kernel instead of a tiny random
> driver like this?
> 

OK, we will put them in include/Linux/bitmap.h.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 14/20] dlb: add start domain ioctl
  2021-03-09  9:29   ` Greg KH
@ 2021-03-10  2:45     ` Chen, Mike Ximing
  2021-03-10  8:14       ` Greg KH
  0 siblings, 1 reply; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10  2:45 UTC (permalink / raw)
  To: Greg KH
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Brandeburg, Jesse



> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> On Wed, Feb 10, 2021 at 11:54:17AM -0600, Mike Ximing Chen wrote:
> >
> > diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
> > index 6a311b969643..9b05344f03c8 100644
> > --- a/drivers/misc/dlb/dlb_ioctl.c
> > +++ b/drivers/misc/dlb/dlb_ioctl.c
> > @@ -51,6 +51,7 @@
> DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
> >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
> >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
> >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
> > +DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
> >
> > --- a/drivers/misc/dlb/dlb_pf_ops.c
> > +++ b/drivers/misc/dlb/dlb_pf_ops.c
> > @@ -160,6 +160,14 @@ dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
> >  				       resp, false, 0);
> >  }
> >
> > +static int
> > +dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
> > +		    struct dlb_start_domain_args *args,
> > +		    struct dlb_cmd_response *resp)
> > +{
> > +	return dlb_hw_start_domain(hw, id, args, resp, false, 0);
> > +}
> > +
> >  static int dlb_pf_get_num_resources(struct dlb_hw *hw,
> >  				    struct dlb_get_num_resources_args *args)
> >  {
> > @@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
> >  	.create_dir_queue = dlb_pf_create_dir_queue,
> >  	.create_ldb_port = dlb_pf_create_ldb_port,
> >  	.create_dir_port = dlb_pf_create_dir_port,
> > +	.start_domain = dlb_pf_start_domain,
> 
> Why do you have a "callback" when you only ever call one function?  Why
> is that needed at all?
> 
In our next submission, we are going to add virtual function (VF) support. The
callbacks for VFs are different from those for PF which is what we support in this
submission. We can defer the introduction of  the callback structure to when we 
add the VF support. But since we have many callback functions, that approach
will generate many changes in then "existing" code. We thought that putting
the callback structure in place now would make the job of adding VF support easier.
Is it OK?

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 03/20] dlb: add resource and device initialization
  2021-03-10  1:33     ` Chen, Mike Ximing
@ 2021-03-10  8:13       ` Greg KH
  2021-03-10 20:26         ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-10  8:13 UTC (permalink / raw)
  To: Chen, Mike Ximing
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Gage Eads

On Wed, Mar 10, 2021 at 01:33:24AM +0000, Chen, Mike Ximing wrote:
> 
> > -----Original Message-----
> > From: Greg KH <gregkh@linuxfoundation.org>
> > 
> > On Wed, Feb 10, 2021 at 11:54:06AM -0600, Mike Ximing Chen wrote:
> > > +
> > > +#include "dlb_bitmap.h"
> > > +
> > > +#define BITS_SET(x, val, mask)	(x = ((x) & ~(mask))     \
> > > +				 | (((val) << (mask##_LOC)) & (mask)))
> > > +#define BITS_GET(x, mask)       (((x) & (mask)) >> (mask##_LOC))
> > 
> > Why not use the built-in kernel functions for this?  Why are you
> > creating your own?
> >
> FIELD_GET(_mask, _val) and FIELD_PREP(_mask, _val) in include/linux/bitfield.h
> are similar to our BITS_GET() and BITS_SET().  However in our case, mask##_LOC
> is a known constant defined in dlb_regs.h,  so we don't need to use 
> _buildin_ffs(mask) to calculate the location of mask as FIELD_GET() and FIELD_PREP()
> do.  We can still use FIELD_GET and FIELD_PREP, but our macros are a little more 
> efficient. Would it be OK to keep them?

No, please use the kernel-wide proper functions, there's no need for
single tiny driver to be "special" in this regard.  If somehow the
in-kernel functions are not sufficient, it's always better to fix them
up than to go your own way here.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 14/20] dlb: add start domain ioctl
  2021-03-10  2:45     ` Chen, Mike Ximing
@ 2021-03-10  8:14       ` Greg KH
  2021-03-10 20:19         ` Dan Williams
  2021-03-10 20:26         ` Chen, Mike Ximing
  0 siblings, 2 replies; 50+ messages in thread
From: Greg KH @ 2021-03-10  8:14 UTC (permalink / raw)
  To: Chen, Mike Ximing
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Brandeburg, Jesse

On Wed, Mar 10, 2021 at 02:45:10AM +0000, Chen, Mike Ximing wrote:
> 
> 
> > -----Original Message-----
> > From: Greg KH <gregkh@linuxfoundation.org>
> > On Wed, Feb 10, 2021 at 11:54:17AM -0600, Mike Ximing Chen wrote:
> > >
> > > diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
> > > index 6a311b969643..9b05344f03c8 100644
> > > --- a/drivers/misc/dlb/dlb_ioctl.c
> > > +++ b/drivers/misc/dlb/dlb_ioctl.c
> > > @@ -51,6 +51,7 @@
> > DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
> > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
> > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
> > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
> > > +DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
> > >
> > > --- a/drivers/misc/dlb/dlb_pf_ops.c
> > > +++ b/drivers/misc/dlb/dlb_pf_ops.c
> > > @@ -160,6 +160,14 @@ dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
> > >  				       resp, false, 0);
> > >  }
> > >
> > > +static int
> > > +dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
> > > +		    struct dlb_start_domain_args *args,
> > > +		    struct dlb_cmd_response *resp)
> > > +{
> > > +	return dlb_hw_start_domain(hw, id, args, resp, false, 0);
> > > +}
> > > +
> > >  static int dlb_pf_get_num_resources(struct dlb_hw *hw,
> > >  				    struct dlb_get_num_resources_args *args)
> > >  {
> > > @@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
> > >  	.create_dir_queue = dlb_pf_create_dir_queue,
> > >  	.create_ldb_port = dlb_pf_create_ldb_port,
> > >  	.create_dir_port = dlb_pf_create_dir_port,
> > > +	.start_domain = dlb_pf_start_domain,
> > 
> > Why do you have a "callback" when you only ever call one function?  Why
> > is that needed at all?
> > 
> In our next submission, we are going to add virtual function (VF) support. The
> callbacks for VFs are different from those for PF which is what we support in this
> submission. We can defer the introduction of  the callback structure to when we 
> add the VF support. But since we have many callback functions, that approach
> will generate many changes in then "existing" code. We thought that putting
> the callback structure in place now would make the job of adding VF support easier.
> Is it OK?

No, do not add additional complexity when it is not needed.  It causes
much more review work and I and no one else have any idea that
"something might be coming in the future", so please do not make our
lives harder.

Make it simple, and work, now.  You can always add additional changes
later, if it is ever needed.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
                   ` (19 preceding siblings ...)
  2021-02-10 17:54 ` [PATCH v10 20/20] dlb: queue map/unmap workqueue Mike Ximing Chen
@ 2021-03-10  9:02 ` Greg KH
  2021-03-12  7:18   ` Dan Williams
  20 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-10  9:02 UTC (permalink / raw)
  To: Mike Ximing Chen
  Cc: netdev, davem, kuba, arnd, dan.j.williams, pierre-louis.bossart

On Wed, Feb 10, 2021 at 11:54:03AM -0600, Mike Ximing Chen wrote:
> Intel DLB is an accelerator for the event-driven programming model of
> DPDK's Event Device Library[2]. The library is used in packet processing
> pipelines that arrange for multi-core scalability, dynamic load-balancing,
> and variety of packet distribution and synchronization schemes

The more that I look at this driver, the more I think this is a "run
around" the networking stack.  Why are you all adding kernel code to
support DPDK which is an out-of-kernel networking stack?  We can't
support that at all.

Why not just use the normal networking functionality instead of this
custom char-device-node-monstrosity?

What is missing from todays kernel networking code that requires this
run-around?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 14/20] dlb: add start domain ioctl
  2021-03-10  8:14       ` Greg KH
@ 2021-03-10 20:19         ` Dan Williams
  2021-03-10 20:26         ` Chen, Mike Ximing
  1 sibling, 0 replies; 50+ messages in thread
From: Dan Williams @ 2021-03-10 20:19 UTC (permalink / raw)
  To: Greg KH
  Cc: Chen, Mike Ximing, netdev, davem, kuba, arnd,
	pierre-louis.bossart, Brandeburg, Jesse

On Wed, Mar 10, 2021 at 12:14 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Mar 10, 2021 at 02:45:10AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Greg KH <gregkh@linuxfoundation.org>
> > > On Wed, Feb 10, 2021 at 11:54:17AM -0600, Mike Ximing Chen wrote:
> > > >
> > > > diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
> > > > index 6a311b969643..9b05344f03c8 100644
> > > > --- a/drivers/misc/dlb/dlb_ioctl.c
> > > > +++ b/drivers/misc/dlb/dlb_ioctl.c
> > > > @@ -51,6 +51,7 @@
> > > DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
> > > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
> > > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
> > > >  DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
> > > > +DLB_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
> > > >
> > > > --- a/drivers/misc/dlb/dlb_pf_ops.c
> > > > +++ b/drivers/misc/dlb/dlb_pf_ops.c
> > > > @@ -160,6 +160,14 @@ dlb_pf_create_dir_port(struct dlb_hw *hw, u32 id,
> > > >                                  resp, false, 0);
> > > >  }
> > > >
> > > > +static int
> > > > +dlb_pf_start_domain(struct dlb_hw *hw, u32 id,
> > > > +             struct dlb_start_domain_args *args,
> > > > +             struct dlb_cmd_response *resp)
> > > > +{
> > > > + return dlb_hw_start_domain(hw, id, args, resp, false, 0);
> > > > +}
> > > > +
> > > >  static int dlb_pf_get_num_resources(struct dlb_hw *hw,
> > > >                               struct dlb_get_num_resources_args *args)
> > > >  {
> > > > @@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
> > > >   .create_dir_queue = dlb_pf_create_dir_queue,
> > > >   .create_ldb_port = dlb_pf_create_ldb_port,
> > > >   .create_dir_port = dlb_pf_create_dir_port,
> > > > + .start_domain = dlb_pf_start_domain,
> > >
> > > Why do you have a "callback" when you only ever call one function?  Why
> > > is that needed at all?
> > >
> > In our next submission, we are going to add virtual function (VF) support. The
> > callbacks for VFs are different from those for PF which is what we support in this
> > submission. We can defer the introduction of  the callback structure to when we
> > add the VF support. But since we have many callback functions, that approach
> > will generate many changes in then "existing" code. We thought that putting
> > the callback structure in place now would make the job of adding VF support easier.
> > Is it OK?
>
> No, do not add additional complexity when it is not needed.  It causes
> much more review work and I and no one else have any idea that
> "something might be coming in the future", so please do not make our
> lives harder.
>
> Make it simple, and work, now.  You can always add additional changes
> later, if it is ever needed.
>

Good points Greg, the internal reviews missed this, let me take
another once over before v11.

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 03/20] dlb: add resource and device initialization
  2021-03-10  8:13       ` Greg KH
@ 2021-03-10 20:26         ` Chen, Mike Ximing
  0 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10 20:26 UTC (permalink / raw)
  To: Greg KH; +Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart


> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Wednesday, March 10, 2021 3:13 AM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: netdev@vger.kernel.org; davem@davemloft.net; kuba@kernel.org;
> arnd@arndb.de; Williams, Dan J <dan.j.williams@intel.com>; pierre-
> louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> Subject: Re: [PATCH v10 03/20] dlb: add resource and device initialization
> 
> On Wed, Mar 10, 2021 at 01:33:24AM +0000, Chen, Mike Ximing wrote:
> >
> > > -----Original Message-----
> > > From: Greg KH <gregkh@linuxfoundation.org>
> > >
> > > On Wed, Feb 10, 2021 at 11:54:06AM -0600, Mike Ximing Chen wrote:
> > > > +
> > > > +#include "dlb_bitmap.h"
> > > > +
> > > > +#define BITS_SET(x, val, mask)	(x = ((x) & ~(mask))     \
> > > > +				 | (((val) << (mask##_LOC)) & (mask)))
> > > > +#define BITS_GET(x, mask)       (((x) & (mask)) >> (mask##_LOC))
> > >
> > > Why not use the built-in kernel functions for this?  Why are you
> > > creating your own?
> > >
> > FIELD_GET(_mask, _val) and FIELD_PREP(_mask, _val) in include/linux/bitfield.h
> > are similar to our BITS_GET() and BITS_SET().  However in our case, mask##_LOC
> > is a known constant defined in dlb_regs.h,  so we don't need to use
> > _buildin_ffs(mask) to calculate the location of mask as FIELD_GET() and
> FIELD_PREP()
> > do.  We can still use FIELD_GET and FIELD_PREP, but our macros are a little more
> > efficient. Would it be OK to keep them?
> 
> No, please use the kernel-wide proper functions, there's no need for
> single tiny driver to be "special" in this regard.  If somehow the
> in-kernel functions are not sufficient, it's always better to fix them
> up than to go your own way here.
> 

OK. I will use FIELD_GET() and FIELD_PREP() macros in the next revision.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 14/20] dlb: add start domain ioctl
  2021-03-10  8:14       ` Greg KH
  2021-03-10 20:19         ` Dan Williams
@ 2021-03-10 20:26         ` Chen, Mike Ximing
  1 sibling, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-10 20:26 UTC (permalink / raw)
  To: Greg KH
  Cc: netdev, davem, kuba, arnd, Williams, Dan J, pierre-louis.bossart,
	Brandeburg, Jesse


> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> 
> On Wed, Mar 10, 2021 at 02:45:10AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Greg KH <gregkh@linuxfoundation.org>
> > > On Wed, Feb 10, 2021 at 11:54:17AM -0600, Mike Ximing Chen wrote:
> > > >
> > > >  {
> > > > @@ -232,6 +240,7 @@ struct dlb_device_ops dlb_pf_ops = {
> > > >  	.create_dir_queue = dlb_pf_create_dir_queue,
> > > >  	.create_ldb_port = dlb_pf_create_ldb_port,
> > > >  	.create_dir_port = dlb_pf_create_dir_port,
> > > > +	.start_domain = dlb_pf_start_domain,
> > >
> > > Why do you have a "callback" when you only ever call one function?  Why
> > > is that needed at all?
> > >
> > In our next submission, we are going to add virtual function (VF) support. The
> > callbacks for VFs are different from those for PF which is what we support in this
> > submission. We can defer the introduction of  the callback structure to when we
> > add the VF support. But since we have many callback functions, that approach
> > will generate many changes in then "existing" code. We thought that putting
> > the callback structure in place now would make the job of adding VF support easier.
> > Is it OK?
> 
> No, do not add additional complexity when it is not needed.  It causes
> much more review work and I and no one else have any idea that
> "something might be coming in the future", so please do not make our
> lives harder.
> 
> Make it simple, and work, now.  You can always add additional changes
> later, if it is ever needed.
> 

Sure. We will remove the callback structure from this patch set.

Thanks for reviewing

Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-10  9:02 ` [PATCH v10 00/20] dlb: introduce DLB device driver Greg KH
@ 2021-03-12  7:18   ` Dan Williams
  2021-03-12 21:55     ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Dan Williams @ 2021-03-12  7:18 UTC (permalink / raw)
  To: Greg KH
  Cc: Mike Ximing Chen, Netdev, David Miller, Jakub Kicinski,
	Arnd Bergmann, Pierre-Louis Bossart

On Wed, Mar 10, 2021 at 1:02 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Feb 10, 2021 at 11:54:03AM -0600, Mike Ximing Chen wrote:
> > Intel DLB is an accelerator for the event-driven programming model of
> > DPDK's Event Device Library[2]. The library is used in packet processing
> > pipelines that arrange for multi-core scalability, dynamic load-balancing,
> > and variety of packet distribution and synchronization schemes
>
> The more that I look at this driver, the more I think this is a "run
> around" the networking stack.  Why are you all adding kernel code to
> support DPDK which is an out-of-kernel networking stack?  We can't
> support that at all.
>
> Why not just use the normal networking functionality instead of this
> custom char-device-node-monstrosity?

Hey Greg,

I've come to find out that this driver does not bypass kernel
networking, and the kernel functionality I thought it bypassed, IPC /
Scheduling, is not even in the picture in the non-accelerated case. So
given you and I are both confused by this submission that tells me
that the problem space needs to be clarified and assumptions need to
be enumerated.

> What is missing from todays kernel networking code that requires this
> run-around?

Yes, first and foremost Mike, what are the kernel infrastructure gaps
and pain points that led up to this proposal?

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-12  7:18   ` Dan Williams
@ 2021-03-12 21:55     ` Chen, Mike Ximing
  2021-03-13  1:39       ` Dan Williams
  0 siblings, 1 reply; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-12 21:55 UTC (permalink / raw)
  To: Williams, Dan J, Greg KH
  Cc: Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse

[-- Attachment #1: Type: text/plain, Size: 4028 bytes --]



> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Friday, March 12, 2021 2:18 AM
> To: Greg KH <gregkh@linuxfoundation.org>
> Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Netdev <netdev@vger.kernel.org>; David Miller
> <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-
> Louis Bossart <pierre-louis.bossart@linux.intel.com>
> Subject: Re: [PATCH v10 00/20] dlb: introduce DLB device driver
> 
> On Wed, Mar 10, 2021 at 1:02 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Feb 10, 2021 at 11:54:03AM -0600, Mike Ximing Chen wrote:
> > > Intel DLB is an accelerator for the event-driven programming model of
> > > DPDK's Event Device Library[2]. The library is used in packet processing
> > > pipelines that arrange for multi-core scalability, dynamic load-balancing,
> > > and variety of packet distribution and synchronization schemes
> >
> > The more that I look at this driver, the more I think this is a "run
> > around" the networking stack.  Why are you all adding kernel code to
> > support DPDK which is an out-of-kernel networking stack?  We can't
> > support that at all.
> >
> > Why not just use the normal networking functionality instead of this
> > custom char-device-node-monstrosity?
> 
> Hey Greg,
> 
> I've come to find out that this driver does not bypass kernel
> networking, and the kernel functionality I thought it bypassed, IPC /
> Scheduling, is not even in the picture in the non-accelerated case. So
> given you and I are both confused by this submission that tells me
> that the problem space needs to be clarified and assumptions need to
> be enumerated.
> 
> > What is missing from todays kernel networking code that requires this
> > run-around?
> 
> Yes, first and foremost Mike, what are the kernel infrastructure gaps
> and pain points that led up to this proposal?

Hi Greg/Dan,

Sorry for the confusion. The cover letter and document did not articulate 
clearly the problem being solved by DLB. We will update the document in
the next revision.

In a brief description, Intel DLB is an accelerator that replaces shared memory
queuing systems. Large modern server-class CPUs,  with local caches
for each core, tend to incur costly cache misses, cross core snoops
and contentions.  The impact becomes noticeable at high (messages/sec) 
rates, such as are seen in high throughput packet processing and HPC 
applications. DLB is used in high rate pipelines that require a variety of packet
distribution & synchronization schemes.  It can be leveraged to accelerate
user space libraries, such as DPDK eventdev. It could show similar benefits in 
frameworks such as PADATA in the Kernel - if the messaging rate is sufficiently
high. As can be seen in the following diagram,  DLB operations come into the 
picture only after packets are received by Rx core from the networking 
devices. WCs are the worker cores which process packets distributed by DLB. 
(In case the diagram gets mis-formatted,  please see attached file).


                              WC1              WC4
 +-----+   +----+   +---+  /      \  +---+  /      \  +---+   +----+   +-----+
 |NIC  |   |Rx  |   |DLB| /        \ |DLB| /        \ |DLB|   |Tx  |   |NIC  |
 |Ports|---|Core|---|   |-----WC2----|   |-----WC5----|   |---|Core|---|Ports|
 +-----+   -----+   +---+ \        / +---+ \        / +---+   +----+   ------+
                           \      /         \      /
                              WC3              WC6 

At its heart DLB consists of resources than can be assigned to 
VDEVs/applications in a flexible manner, such as ports, queues, credits to use
queues, sequence numbers, etc. We support up to 16/32 VF/VDEVs (depending
on version) with SRIOV and SIOV. Role of the kernel driver includes VDEV 
Composition (vdcm module), functional level reset, live migration, error 
handling, power management, and etc..

Thanks
Mike

[-- Attachment #2: dlb_drawing.txt --]
[-- Type: text/plain, Size: 542 bytes --]





                              WC1              WC4          
 +-----+   +----+   +---+  /      \  +---+  /      \  +---+   +----+   +-----+
 |NIC  |   |Rx  |   |DLB| /        \ |DLB| /        \ |DLB|   |Tx  |   |NIC  |
 |Ports|---|Core|---|   |-----WC2----|   |-----WC5----|   |---|Core|---|Ports| 
 +-----+   -----+   +---+ \        / +---+ \        / +---+   +----+   ------+
                           \      /         \      /                           
                              WC3              WC6                             

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-12 21:55     ` Chen, Mike Ximing
@ 2021-03-13  1:39       ` Dan Williams
  2021-03-15 20:04         ` Chen, Mike Ximing
                           ` (2 more replies)
  0 siblings, 3 replies; 50+ messages in thread
From: Dan Williams @ 2021-03-13  1:39 UTC (permalink / raw)
  To: Chen, Mike Ximing
  Cc: Greg KH, Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse

On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing
<mike.ximing.chen@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Dan Williams <dan.j.williams@intel.com>
> > Sent: Friday, March 12, 2021 2:18 AM
> > To: Greg KH <gregkh@linuxfoundation.org>
> > Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Netdev <netdev@vger.kernel.org>; David Miller
> > <davem@davemloft.net>; Jakub Kicinski <kuba@kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-
> > Louis Bossart <pierre-louis.bossart@linux.intel.com>
> > Subject: Re: [PATCH v10 00/20] dlb: introduce DLB device driver
> >
> > On Wed, Mar 10, 2021 at 1:02 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Feb 10, 2021 at 11:54:03AM -0600, Mike Ximing Chen wrote:
> > > > Intel DLB is an accelerator for the event-driven programming model of
> > > > DPDK's Event Device Library[2]. The library is used in packet processing
> > > > pipelines that arrange for multi-core scalability, dynamic load-balancing,
> > > > and variety of packet distribution and synchronization schemes
> > >
> > > The more that I look at this driver, the more I think this is a "run
> > > around" the networking stack.  Why are you all adding kernel code to
> > > support DPDK which is an out-of-kernel networking stack?  We can't
> > > support that at all.
> > >
> > > Why not just use the normal networking functionality instead of this
> > > custom char-device-node-monstrosity?
> >
> > Hey Greg,
> >
> > I've come to find out that this driver does not bypass kernel
> > networking, and the kernel functionality I thought it bypassed, IPC /
> > Scheduling, is not even in the picture in the non-accelerated case. So
> > given you and I are both confused by this submission that tells me
> > that the problem space needs to be clarified and assumptions need to
> > be enumerated.
> >
> > > What is missing from todays kernel networking code that requires this
> > > run-around?
> >
> > Yes, first and foremost Mike, what are the kernel infrastructure gaps
> > and pain points that led up to this proposal?
>
> Hi Greg/Dan,
>
> Sorry for the confusion. The cover letter and document did not articulate
> clearly the problem being solved by DLB. We will update the document in
> the next revision.

I'm not sure this answers Greg question about what is missing from
today's kernel implementation?

> In a brief description, Intel DLB is an accelerator that replaces shared memory
> queuing systems. Large modern server-class CPUs,  with local caches
> for each core, tend to incur costly cache misses, cross core snoops
> and contentions.  The impact becomes noticeable at high (messages/sec)
> rates, such as are seen in high throughput packet processing and HPC
> applications. DLB is used in high rate pipelines that require a variety of packet
> distribution & synchronization schemes.  It can be leveraged to accelerate
> user space libraries, such as DPDK eventdev. It could show similar benefits in
> frameworks such as PADATA in the Kernel - if the messaging rate is sufficiently
> high.

Where is PADATA limited by distribution and synchronization overhead?
It's meant for parallelizable work that has minimal communication
between the work units, ordering is about it's only synchronization
overhead, not messaging. It's used for ipsec crypto and page init.
Even potential future bulk work usages that might benefit from PADATA
like like md-raid, ksm, or kcopyd do not have any messaging overhead.

> As can be seen in the following diagram,  DLB operations come into the
> picture only after packets are received by Rx core from the networking
> devices. WCs are the worker cores which process packets distributed by DLB.
> (In case the diagram gets mis-formatted,  please see attached file).
>
>
>                               WC1              WC4
>  +-----+   +----+   +---+  /      \  +---+  /      \  +---+   +----+   +-----+
>  |NIC  |   |Rx  |   |DLB| /        \ |DLB| /        \ |DLB|   |Tx  |   |NIC  |
>  |Ports|---|Core|---|   |-----WC2----|   |-----WC5----|   |---|Core|---|Ports|
>  +-----+   -----+   +---+ \        / +---+ \        / +---+   +----+   ------+
>                            \      /         \      /
>                               WC3              WC6
>
> At its heart DLB consists of resources than can be assigned to
> VDEVs/applications in a flexible manner, such as ports, queues, credits to use
> queues, sequence numbers, etc.

All of those objects are managed in userspace today in the unaccelerated case?

> We support up to 16/32 VF/VDEVs (depending
> on version) with SRIOV and SIOV. Role of the kernel driver includes VDEV
> Composition (vdcm module), functional level reset, live migration, error
> handling, power management, and etc..

Need some more specificity here. What about those features requires
the kernel to get involved with a DLB2 specific ABI to manage ports,
queues, credits, sequence numbers, etc...?

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-13  1:39       ` Dan Williams
@ 2021-03-15 20:04         ` Chen, Mike Ximing
  2021-03-15 20:08         ` Chen, Mike Ximing
  2021-03-15 20:18         ` Chen, Mike Ximing
  2 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-15 20:04 UTC (permalink / raw)
  To: Williams, Dan J
  Cc: Greg KH, Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse

> From: Dan Williams <dan.j.williams@intel.com>
> On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing <mike.ximing.chen@intel.com> wrote:
> >
> > In a brief description, Intel DLB is an accelerator that replaces
> > shared memory queuing systems. Large modern server-class CPUs,  with
> > local caches for each core, tend to incur costly cache misses, cross
> > core snoops and contentions.  The impact becomes noticeable at high
> > (messages/sec) rates, such as are seen in high throughput packet
> > processing and HPC applications. DLB is used in high rate pipelines
> > that require a variety of packet distribution & synchronization
> > schemes.  It can be leveraged to accelerate user space libraries, such
> > as DPDK eventdev. It could show similar benefits in frameworks such as
> > PADATA in the Kernel - if the messaging rate is sufficiently high.
> 
> Where is PADATA limited by distribution and synchronization overhead?
> It's meant for parallelizable work that has minimal communication between the work units, ordering is
> about it's only synchronization overhead, not messaging. It's used for ipsec crypto and page init.
> Even potential future bulk work usages that might benefit from PADATA like like md-raid, ksm, or kcopyd
> do not have any messaging overhead.
> 
In the our PADATA investigation, the improvements are primarily from ordering overhead.
Parallel scheduling is offloaded by DLB orderd parallel queue.
Serialization (re-order) is offloaded by DLB directed queue.
We see significant throughput increases in crypto tests using tcrypt.  In our test configuration, preliminary results show that the dlb accelerated case encrypts at 2.4x (packets/s), and decrypts at 2.6x of the unaccelerated case.

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-13  1:39       ` Dan Williams
  2021-03-15 20:04         ` Chen, Mike Ximing
@ 2021-03-15 20:08         ` Chen, Mike Ximing
  2021-03-15 20:18         ` Chen, Mike Ximing
  2 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-15 20:08 UTC (permalink / raw)
  To: Williams, Dan J
  Cc: Greg KH, Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse

> From: Dan Williams <dan.j.williams@intel.com>
> On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing <mike.ximing.chen@intel.com> wrote:
> >
> > At its heart DLB consists of resources than can be assigned to
> > VDEVs/applications in a flexible manner, such as ports, queues,
> > credits to use queues, sequence numbers, etc.
> 
> All of those objects are managed in userspace today in the unaccelerated case?
> 

Yes, in the unaccelerated case, the software queue manager is generally implemented in the user space (except for cases like padata), so the resources are managed in the user space as well.
With a hardware DLB module, these resources will be managed by the kernel driver for VF and VDEV supports.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-13  1:39       ` Dan Williams
  2021-03-15 20:04         ` Chen, Mike Ximing
  2021-03-15 20:08         ` Chen, Mike Ximing
@ 2021-03-15 20:18         ` Chen, Mike Ximing
  2021-03-16  9:01           ` Greg KH
  2 siblings, 1 reply; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-03-15 20:18 UTC (permalink / raw)
  To: Williams, Dan J
  Cc: Greg KH, Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse

> From: Dan Williams <dan.j.williams@intel.com>
> On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing <mike.ximing.chen@intel.com> wrote:
> >
> > We support up to 16/32 VF/VDEVs (depending on version) with SRIOV and
> > SIOV. Role of the kernel driver includes VDEV Composition (vdcm
> > module), functional level reset, live migration, error handling, power
> > management, and etc..
> 
> Need some more specificity here. What about those features requires the kernel to get involved with a
> DLB2 specific ABI to manage ports, queues, credits, sequence numbers, etc...?

Role of the dlb kernel driver:

VDEV Composition
For example writing 1024 to the VDEV_CREDITS[0] register will allocate 1024 credits to VDEV 0. In this way, VFs or VDEVs can be composed  as mini-versions of the full device.
VDEV composition will leverage vfio-mdev to create the VDEV devices while the KMD will implement the VDCM.

Dynamic Composition
Such composition can be dynamic – the PF/VF interface supports scenarios whereby, for example, an application may wish to boost its credit allocation – can I have 100 more credits?

Functional Level Reset
Much of the internal storage is RAM based and not resettable by hardware schemes. There are also internal SRAM  based control structures (BCAM) that have to be flushed. 
The planned way to do this is, roughly:
  -- Kernel driver disables access from the associated ports  (to prevent any SW access, the application should be deadso this is a precaution).
  -- Kernel masquerades as the application to drain all data from internal queues. It can poll some internal counters to verify everything is fully drained.
  -- Only at this point can the resources associated with the VDEV be returned to the pool of available resources for handing to another application/VDEV.

Migration
Requirement is fairly similar to FLR. A VDEV has to be manually drained and reconstituted on another server, Kernel driver is responsible on both sides.

Error Handling
Errors include “Credit Excursions” where a VDEV attempts to use more of the internal capacity (credits) than has been allocated. In such a case, 
the data is dropped and an interrupt generated. All such interrupts are directed to the PF driver, which may simply forward them to a VF (via the PF/VF comms mechanism).

Power Management
The kernel driver keeps the device in D3Hot when not in use. The driver transitions the device to D0 when the first device file is opened or a VF or VDEV is created, 
and keeps it in that state until there are no open device files, memory mappings, or VFs/VDEVs.

Ioctl interface
Kernel driver provides ioctl interface for user applications to setup and configure dlb domains, ports, queues, scheduling types, credits, 
sequence numbers, and links between ports and queues.  Applications also use the interface to start, stop and inquire the dlb operations.

Thanks
Mike


^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-15 20:18         ` Chen, Mike Ximing
@ 2021-03-16  9:01           ` Greg KH
  2021-05-12 19:07             ` Dan Williams
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-03-16  9:01 UTC (permalink / raw)
  To: Chen, Mike Ximing
  Cc: Williams, Dan J, Netdev, David Miller, Jakub Kicinski,
	Arnd Bergmann, Pierre-Louis Bossart, Brandeburg, Jesse

On Mon, Mar 15, 2021 at 08:18:10PM +0000, Chen, Mike Ximing wrote:
> > From: Dan Williams <dan.j.williams@intel.com>
> > On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing <mike.ximing.chen@intel.com> wrote:
> > >
> > > We support up to 16/32 VF/VDEVs (depending on version) with SRIOV and
> > > SIOV. Role of the kernel driver includes VDEV Composition (vdcm
> > > module), functional level reset, live migration, error handling, power
> > > management, and etc..
> > 
> > Need some more specificity here. What about those features requires the kernel to get involved with a
> > DLB2 specific ABI to manage ports, queues, credits, sequence numbers, etc...?
> 
> Role of the dlb kernel driver:
> 
> VDEV Composition
> For example writing 1024 to the VDEV_CREDITS[0] register will allocate 1024 credits to VDEV 0. In this way, VFs or VDEVs can be composed  as mini-versions of the full device.
> VDEV composition will leverage vfio-mdev to create the VDEV devices while the KMD will implement the VDCM.

What is a vdev?

What is KMD?

What is VDCM?

What is VF?

And how does this all work?

> Dynamic Composition
> Such composition can be dynamic – the PF/VF interface supports scenarios whereby, for example, an application may wish to boost its credit allocation – can I have 100 more credits?

What applications?  What "credits"  For what resources?

> Functional Level Reset
> Much of the internal storage is RAM based and not resettable by hardware schemes. There are also internal SRAM  based control structures (BCAM) that have to be flushed. 
> The planned way to do this is, roughly:
>   -- Kernel driver disables access from the associated ports  (to prevent any SW access, the application should be deadso this is a precaution).

What is a "port" here?


>   -- Kernel masquerades as the application to drain all data from internal queues. It can poll some internal counters to verify everything is fully drained.

What queues?

Why would the kernel mess with userspace data?

>   -- Only at this point can the resources associated with the VDEV be returned to the pool of available resources for handing to another application/VDEV.

What is a VDEV and how does an application be "associated with it"?

> Migration
> Requirement is fairly similar to FLR. A VDEV has to be manually drained and reconstituted on another server, Kernel driver is responsible on both sides.

What is FLR?

> Error Handling
> Errors include “Credit Excursions” where a VDEV attempts to use more of the internal capacity (credits) than has been allocated. In such a case, 
> the data is dropped and an interrupt generated. All such interrupts are directed to the PF driver, which may simply forward them to a VF (via the PF/VF comms mechanism).

What data is going where?

> Power Management
> The kernel driver keeps the device in D3Hot when not in use. The driver transitions the device to D0 when the first device file is opened or a VF or VDEV is created, 
> and keeps it in that state until there are no open device files, memory mappings, or VFs/VDEVs.

That's just normal power management for any device, why is this anything
special?

> Ioctl interface
> Kernel driver provides ioctl interface for user applications to setup and configure dlb domains, ports, queues, scheduling types, credits, 
> sequence numbers, and links between ports and queues.  Applications also use the interface to start, stop and inquire the dlb operations.

What applications use any of this?  What userspace implementation today
interacts with this?  Where is that code located?

Too many TLAs here, I have even less of an understanding of what this
driver is supposed to be doing, and what this hardware is now than
before.

And here I thought I understood hardware devices, and if I am confused,
I pity anyone else looking at this code...

You all need to get some real documentation together to explain
everything here in terms that anyone can understand.  Without that, this
code is going nowhere.

good luck!

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-03-16  9:01           ` Greg KH
@ 2021-05-12 19:07             ` Dan Williams
  2021-05-14 14:33               ` Greg KH
  0 siblings, 1 reply; 50+ messages in thread
From: Dan Williams @ 2021-05-12 19:07 UTC (permalink / raw)
  To: Greg KH
  Cc: Chen, Mike Ximing, Netdev, David Miller, Jakub Kicinski,
	Arnd Bergmann, Pierre-Louis Bossart, Brandeburg, Jesse, KVM list,
	Raj, Ashok

[ add kvm@vger.kernel.org for VFIO discussion ]


On Tue, Mar 16, 2021 at 2:01 AM Greg KH <gregkh@linuxfoundation.org> wrote:
[..]
> > Ioctl interface
> > Kernel driver provides ioctl interface for user applications to setup and configure dlb domains, ports, queues, scheduling types, credits,
> > sequence numbers, and links between ports and queues.  Applications also use the interface to start, stop and inquire the dlb operations.
>
> What applications use any of this?  What userspace implementation today
> interacts with this?  Where is that code located?
>
> Too many TLAs here, I have even less of an understanding of what this
> driver is supposed to be doing, and what this hardware is now than
> before.
>
> And here I thought I understood hardware devices, and if I am confused,
> I pity anyone else looking at this code...
>
> You all need to get some real documentation together to explain
> everything here in terms that anyone can understand.  Without that, this
> code is going nowhere.

Hi Greg,

So, for the last few weeks Mike and company have patiently waded
through my questions and now I think we are at a point to work through
the upstream driver architecture options and tradeoffs. You were not
alone in struggling to understand what this device does because it is
unlike any other accelerator Linux has ever considered. It shards /
load balances a data stream for processing by CPU threads. This is
typically a network appliance function / protocol, but could also be
any other generic thread pool like the kernel's padata. It saves the
CPU cycles spent load balancing work items and marshaling them through
a thread pool pipeline. For example, in DPDK applications, DLB2 frees
up entire cores that would otherwise be consumed with scheduling and
work distribution. A separate proof-of-concept, using DLB2 to
accelerate the kernel's "padata" thread pool for a crypto workload,
demonstrated ~150% higher throughput with hardware employed to manage
work distribution and result ordering. Yes, you need a sufficiently
high touch / high throughput protocol before the software load
balancing overhead coordinating CPU threads starts to dominate the
performance, but there are some specific workloads willing to switch
to this regime.

The primary consumer to date has been as a backend for the event
handling in the userspace networking stack, DPDK. DLB2 has an existing
polled-mode-userspace driver for that use case. So I said, "great,
just add more features to that userspace driver and you're done". In
fact there was DLB1 hardware that also had a polled-mode-userspace
driver. So, the next question is "what's changed in DLB2 where a
userspace driver is no longer suitable?". The new use case for DLB2 is
new hardware support for a host driver to carve up device resources
into smaller sets (vfio-mdevs) that can be assigned to guests (Intel
calls this new hardware capability SIOV: Scalable IO Virtualization).

Hardware resource management is difficult to handle in userspace
especially when bare-metal hardware events need to coordinate with
guest-VM device instances. This includes a mailbox interface for the
guest VM to negotiate resources with the host driver. Another more
practical roadblock for a "DLB2 in userspace" proposal is the fact
that it implements what are in-effect software-defined-interrupts to
go beyond the scalability limits of PCI MSI-x (Intel calls this
Interrupt Message Store: IMS). So even if hardware resource management
was awkwardly plumbed into a userspace daemon there would still need
to be kernel enabling for device-specific extensions to
drivers/vfio/pci/vfio_pci_intrs.c for it to understand the IMS
interrupts of DLB2 in addition to PCI MSI-x.

While that still might be solvable in userspace if you squint at it, I
don't think Linux end users are served by pushing all of hardware
resource management to userspace. VFIO is mostly built to pass entire
PCI devices to guests, or in coordination with a kernel driver to
describe a subset of the hardware to a virtual-device (vfio-mdev)
interface. The rub here is that to date kernel drivers using VFIO to
provision mdevs have some existing responsibilities to the core kernel
like a network driver or DMA offload driver. The DLB2 driver offers no
such service to the kernel for its primary role of accelerating a
userspace data-plane. I am assuming here that  the padata
proof-of-concept is interesting, but not a compelling reason to ship a
driver compared to giving end users competent kernel-driven
hardware-resource assignment for deploying DLB2 virtual instances into
guest VMs.

My "just continue in userspace" suggestion has no answer for the IMS
interrupt and reliable hardware resource management support
requirements. If you're with me so far we can go deeper into the
details, but in answer to your previous questions most of the TLAs
were from the land of "SIOV" where the VFIO community should be
brought in to review. The driver is mostly a configuration plane where
the fast path data-plane is entirely in userspace. That configuration
plane needs to manage hardware events and resourcing on behalf of
guest VMs running on a partitioned subset of the device. There are
worthwhile questions about whether some of the uapi can be refactored
to common modules like uacce, but I think we need to get to a first
order understanding on what DLB2 is and why the kernel has a role
before diving into the uapi discussion.

Any clearer?

So, in summary drivers/misc/ appears to be the first stop in the
review since a host driver needs to be established to start the VFIO
enabling campaign. With my community hat on, I think requiring
standalone host drivers is healthier for Linux than broaching the
subject of VFIO-only drivers. Even if, as in this case, the initial
host driver is mostly implementing a capability that could be achieved
with a userspace driver.

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-05-12 19:07             ` Dan Williams
@ 2021-05-14 14:33               ` Greg KH
  2021-07-16  1:04                 ` Chen, Mike Ximing
  0 siblings, 1 reply; 50+ messages in thread
From: Greg KH @ 2021-05-14 14:33 UTC (permalink / raw)
  To: Dan Williams
  Cc: Chen, Mike Ximing, Netdev, David Miller, Jakub Kicinski,
	Arnd Bergmann, Pierre-Louis Bossart, Brandeburg, Jesse, KVM list,
	Raj, Ashok

On Wed, May 12, 2021 at 12:07:31PM -0700, Dan Williams wrote:
> [ add kvm@vger.kernel.org for VFIO discussion ]
> 
> 
> On Tue, Mar 16, 2021 at 2:01 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> [..]
> > > Ioctl interface
> > > Kernel driver provides ioctl interface for user applications to setup and configure dlb domains, ports, queues, scheduling types, credits,
> > > sequence numbers, and links between ports and queues.  Applications also use the interface to start, stop and inquire the dlb operations.
> >
> > What applications use any of this?  What userspace implementation today
> > interacts with this?  Where is that code located?
> >
> > Too many TLAs here, I have even less of an understanding of what this
> > driver is supposed to be doing, and what this hardware is now than
> > before.
> >
> > And here I thought I understood hardware devices, and if I am confused,
> > I pity anyone else looking at this code...
> >
> > You all need to get some real documentation together to explain
> > everything here in terms that anyone can understand.  Without that, this
> > code is going nowhere.
> 
> Hi Greg,
> 
> So, for the last few weeks Mike and company have patiently waded
> through my questions and now I think we are at a point to work through
> the upstream driver architecture options and tradeoffs. You were not
> alone in struggling to understand what this device does because it is
> unlike any other accelerator Linux has ever considered. It shards /
> load balances a data stream for processing by CPU threads. This is
> typically a network appliance function / protocol, but could also be
> any other generic thread pool like the kernel's padata. It saves the
> CPU cycles spent load balancing work items and marshaling them through
> a thread pool pipeline. For example, in DPDK applications, DLB2 frees
> up entire cores that would otherwise be consumed with scheduling and
> work distribution. A separate proof-of-concept, using DLB2 to
> accelerate the kernel's "padata" thread pool for a crypto workload,
> demonstrated ~150% higher throughput with hardware employed to manage
> work distribution and result ordering. Yes, you need a sufficiently
> high touch / high throughput protocol before the software load
> balancing overhead coordinating CPU threads starts to dominate the
> performance, but there are some specific workloads willing to switch
> to this regime.
> 
> The primary consumer to date has been as a backend for the event
> handling in the userspace networking stack, DPDK. DLB2 has an existing
> polled-mode-userspace driver for that use case. So I said, "great,
> just add more features to that userspace driver and you're done". In
> fact there was DLB1 hardware that also had a polled-mode-userspace
> driver. So, the next question is "what's changed in DLB2 where a
> userspace driver is no longer suitable?". The new use case for DLB2 is
> new hardware support for a host driver to carve up device resources
> into smaller sets (vfio-mdevs) that can be assigned to guests (Intel
> calls this new hardware capability SIOV: Scalable IO Virtualization).
> 
> Hardware resource management is difficult to handle in userspace
> especially when bare-metal hardware events need to coordinate with
> guest-VM device instances. This includes a mailbox interface for the
> guest VM to negotiate resources with the host driver. Another more
> practical roadblock for a "DLB2 in userspace" proposal is the fact
> that it implements what are in-effect software-defined-interrupts to
> go beyond the scalability limits of PCI MSI-x (Intel calls this
> Interrupt Message Store: IMS). So even if hardware resource management
> was awkwardly plumbed into a userspace daemon there would still need
> to be kernel enabling for device-specific extensions to
> drivers/vfio/pci/vfio_pci_intrs.c for it to understand the IMS
> interrupts of DLB2 in addition to PCI MSI-x.
> 
> While that still might be solvable in userspace if you squint at it, I
> don't think Linux end users are served by pushing all of hardware
> resource management to userspace. VFIO is mostly built to pass entire
> PCI devices to guests, or in coordination with a kernel driver to
> describe a subset of the hardware to a virtual-device (vfio-mdev)
> interface. The rub here is that to date kernel drivers using VFIO to
> provision mdevs have some existing responsibilities to the core kernel
> like a network driver or DMA offload driver. The DLB2 driver offers no
> such service to the kernel for its primary role of accelerating a
> userspace data-plane. I am assuming here that  the padata
> proof-of-concept is interesting, but not a compelling reason to ship a
> driver compared to giving end users competent kernel-driven
> hardware-resource assignment for deploying DLB2 virtual instances into
> guest VMs.
> 
> My "just continue in userspace" suggestion has no answer for the IMS
> interrupt and reliable hardware resource management support
> requirements. If you're with me so far we can go deeper into the
> details, but in answer to your previous questions most of the TLAs
> were from the land of "SIOV" where the VFIO community should be
> brought in to review. The driver is mostly a configuration plane where
> the fast path data-plane is entirely in userspace. That configuration
> plane needs to manage hardware events and resourcing on behalf of
> guest VMs running on a partitioned subset of the device. There are
> worthwhile questions about whether some of the uapi can be refactored
> to common modules like uacce, but I think we need to get to a first
> order understanding on what DLB2 is and why the kernel has a role
> before diving into the uapi discussion.
> 
> Any clearer?

A bit, yes, thanks.

> So, in summary drivers/misc/ appears to be the first stop in the
> review since a host driver needs to be established to start the VFIO
> enabling campaign. With my community hat on, I think requiring
> standalone host drivers is healthier for Linux than broaching the
> subject of VFIO-only drivers. Even if, as in this case, the initial
> host driver is mostly implementing a capability that could be achieved
> with a userspace driver.

Ok, then how about a much "smaller" kernel driver for all of this, and a
whole lot of documentation to describe what is going on and what all of
the TLAs are.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 50+ messages in thread

* RE: [PATCH v10 00/20] dlb: introduce DLB device driver
  2021-05-14 14:33               ` Greg KH
@ 2021-07-16  1:04                 ` Chen, Mike Ximing
  0 siblings, 0 replies; 50+ messages in thread
From: Chen, Mike Ximing @ 2021-07-16  1:04 UTC (permalink / raw)
  To: Greg KH, Williams, Dan J
  Cc: Netdev, David Miller, Jakub Kicinski, Arnd Bergmann,
	Pierre-Louis Bossart, Brandeburg, Jesse, KVM list, Raj, Ashok,
	Linux Kernel Mailing List

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Friday, May 14, 2021 10:33 AM
> To: Williams, Dan J <dan.j.williams@intel.com>
> > Hi Greg,
> >
> > So, for the last few weeks Mike and company have patiently waded
> > through my questions and now I think we are at a point to work through
> > the upstream driver architecture options and tradeoffs. You were not
> > alone in struggling to understand what this device does because it is
> > unlike any other accelerator Linux has ever considered. It shards /
> > load balances a data stream for processing by CPU threads. This is
> > typically a network appliance function / protocol, but could also be
> > any other generic thread pool like the kernel's padata. It saves the
> > CPU cycles spent load balancing work items and marshaling them through
> > a thread pool pipeline. For example, in DPDK applications, DLB2 frees
> > up entire cores that would otherwise be consumed with scheduling and
> > work distribution. A separate proof-of-concept, using DLB2 to
> > accelerate the kernel's "padata" thread pool for a crypto workload,
> > demonstrated ~150% higher throughput with hardware employed to manage
> > work distribution and result ordering. Yes, you need a sufficiently
> > high touch / high throughput protocol before the software load
> > balancing overhead coordinating CPU threads starts to dominate the
> > performance, but there are some specific workloads willing to switch
> > to this regime.
> >
> > The primary consumer to date has been as a backend for the event
> > handling in the userspace networking stack, DPDK. DLB2 has an existing
> > polled-mode-userspace driver for that use case. So I said, "great,
> > just add more features to that userspace driver and you're done". In
> > fact there was DLB1 hardware that also had a polled-mode-userspace
> > driver. So, the next question is "what's changed in DLB2 where a
> > userspace driver is no longer suitable?". The new use case for DLB2 is
> > new hardware support for a host driver to carve up device resources
> > into smaller sets (vfio-mdevs) that can be assigned to guests (Intel
> > calls this new hardware capability SIOV: Scalable IO Virtualization).
> >
> > Hardware resource management is difficult to handle in userspace
> > especially when bare-metal hardware events need to coordinate with
> > guest-VM device instances. This includes a mailbox interface for the
> > guest VM to negotiate resources with the host driver. Another more
> > practical roadblock for a "DLB2 in userspace" proposal is the fact
> > that it implements what are in-effect software-defined-interrupts to
> > go beyond the scalability limits of PCI MSI-x (Intel calls this
> > Interrupt Message Store: IMS). So even if hardware resource management
> > was awkwardly plumbed into a userspace daemon there would still need
> > to be kernel enabling for device-specific extensions to
> > drivers/vfio/pci/vfio_pci_intrs.c for it to understand the IMS
> > interrupts of DLB2 in addition to PCI MSI-x.
> >
> > While that still might be solvable in userspace if you squint at it, I
> > don't think Linux end users are served by pushing all of hardware
> > resource management to userspace. VFIO is mostly built to pass entire
> > PCI devices to guests, or in coordination with a kernel driver to
> > describe a subset of the hardware to a virtual-device (vfio-mdev)
> > interface. The rub here is that to date kernel drivers using VFIO to
> > provision mdevs have some existing responsibilities to the core kernel
> > like a network driver or DMA offload driver. The DLB2 driver offers no
> > such service to the kernel for its primary role of accelerating a
> > userspace data-plane. I am assuming here that  the padata
> > proof-of-concept is interesting, but not a compelling reason to ship a
> > driver compared to giving end users competent kernel-driven
> > hardware-resource assignment for deploying DLB2 virtual instances into
> > guest VMs.
> >
> > My "just continue in userspace" suggestion has no answer for the IMS
> > interrupt and reliable hardware resource management support
> > requirements. If you're with me so far we can go deeper into the
> > details, but in answer to your previous questions most of the TLAs
> > were from the land of "SIOV" where the VFIO community should be
> > brought in to review. The driver is mostly a configuration plane where
> > the fast path data-plane is entirely in userspace. That configuration
> > plane needs to manage hardware events and resourcing on behalf of
> > guest VMs running on a partitioned subset of the device. There are
> > worthwhile questions about whether some of the uapi can be refactored
> > to common modules like uacce, but I think we need to get to a first
> > order understanding on what DLB2 is and why the kernel has a role
> > before diving into the uapi discussion.
> >
> > Any clearer?
> 
> A bit, yes, thanks.
> 
> > So, in summary drivers/misc/ appears to be the first stop in the
> > review since a host driver needs to be established to start the VFIO
> > enabling campaign. With my community hat on, I think requiring
> > standalone host drivers is healthier for Linux than broaching the
> > subject of VFIO-only drivers. Even if, as in this case, the initial
> > host driver is mostly implementing a capability that could be achieved
> > with a userspace driver.
> 
> Ok, then how about a much "smaller" kernel driver for all of this, and a whole lot of documentation to
> describe what is going on and what all of the TLAs are.
> 
> thanks,
> 
> greg k-h

Hi Greg,

tl;dr: We have been looking into various options to reduce the kernel driver size and ABI surface, such as moving more responsibility to user space, reusing existing kernel modules (uacce, for example), and converting functionality from ioctl to sysfs. End result 10 ioctls will be replaced by sysfs, the rest of them (20 ioctls) will be replaced by configfs. Some concepts are moved to device-special files rather than ioctls that produce file descriptors.

Details:
We investigated the possibility of using uacce (https://www.kernel.org/doc/html/latest/misc-devices/uacce.html) in our kernel driver.  The uacce interface fits well with accelerators that process user data with known source and destination addresses. For a DLB (Dynamic Load Balancer), however,  the destination port depends on the system load and is unknown to the application. While uacce exposes  "queues" to user, the dlb driver has to handle much complicated resource managements, such as credits, ports, queues and domains. We would have to add a lot of more concepts and code, which are not useful for other accelerators,  in uacce to make it working for DLB. This may also lead to a bigger code size over all.

We also took a another look at moving resource management functionality from kernel space to user space. Much of kernel driver supports both PF (Physical Function) on host and VFs (Virtual Functions) on VMs. Since only the PF on the host has permissions to setup resource and configure the DLB HW, all the requests on VFs are forwarded to PF via the VF-PF mail boxes, which are handled by the kernel driver. The driver also maintains various virtual id to physical id translations (for VFs, ports, queues, etc), and provides virtual-to-physical id mapping info DLB HW so that an application in VM can access the resources with virtual IDs only. Because of the VF/VDEV support, we have to keep the resource management, which is more than one half of the code size, in the driver.

To simplify the user interface, we explored the ways to reduce/eliminate ioctl interface, and found that we can utilize configfs for many of the DLB functionalities. Our current plan is to replace all the ioctls in the driver with sysfs and configfs. We will use configfs for most of setup and configuration for both physical function and virtual functions. This may not reduce the overall driver size greatly, but it will lessen much of ABI maintenance burden (with the elimination of ioctls).  I hope this is something that is in line with what you like to see for the driver.

Thanks
Mike

^ permalink raw reply	[flat|nested] 50+ messages in thread

* [PATCH v10 00/20] dlb: introduce DLB device driver
@ 2021-01-27 22:56 Mike Ximing Chen
  0 siblings, 0 replies; 50+ messages in thread
From: Mike Ximing Chen @ 2021-01-27 22:56 UTC (permalink / raw)
  To: linux-kernel; +Cc: arnd, gregkh, dan.j.williams, pierre-louis.bossart

Introduce a new misc device driver for the Intel(r) Dynamic Load Balancer
(Intel(r) DLB). The Intel DLB is a PCIe device that provides
load-balanced, prioritized scheduling of core-to-core communication.

Intel DLB is an accelerator for the event-driven programming model of
DPDK's Event Device Library[2]. The library is used in packet processing
pipelines that arrange for multi-core scalability, dynamic load-balancing,
and variety of packet distribution and synchronization schemes

These distribution schemes include "parallel" (packets are load-balanced
across multiple cores and processed in parallel), "ordered" (similar to
"parallel" but packets are reordered into ingress order by the device), and
"atomic" (packet flows are scheduled to a single core at a time such that
locks are not required to access per-flow data, and dynamically migrated to
ensure load-balance).

This submission supports Intel DLB 2.0 only.

The Intel DLB consists of queues and arbiters that connect producer
cores and consumer cores. The device implements load-balanced queueing
features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
  distribution.

The fundamental unit of communication through the device is a queue entry
(QE), which consists of 8B of data and 8B of metadata (destination queue,
priority, etc.). The data field can be any type that fits within 8B.

A core's interface to the device, a "port," consists of a memory-mappable
region through which the core enqueues a queue entry, and an in-memory
queue (the "consumer queue") to which the device schedules QEs. Each QE
is enqueued to a device-managed queue, and from there scheduled to a port.
Software specifies the "linking" of queues and ports; i.e. which ports the
device is allowed to schedule to for a given queue. The device uses a
credit scheme to prevent overflow of the on-device queue storage.

Applications can interface directly with the device by mapping the port's
memory and MMIO regions into the application's address space for enqueue
and dequeue operations, but call into the kernel driver for configuration
operations. An application can also be polling- or interrupt-driven;
Intel DLB supports both modes of operation.

Device resources -- i.e. ports, queues, and credits -- are contained within
a scheduling domain. Scheduling domains are isolated from one another; a
port can only enqueue to and dequeue from queues within its scheduling
domain. A scheduling domain's resources are configured through a scheduling
domain file, which is acquired through an ioctl.

Intel DLB supports SR-IOV and Scalable IOV, and allows for a flexible
division of its resources among the PF and its virtual devices. The virtual
devices are incapable of configuring the device directly; they use a
hardware mailbox to proxy configuration requests to the PF driver. This
driver supports both PF and virtual devices, as there is significant code
re-use between the two, with device-specific behavior handled through a
callback interface.  Virtualization support will be added in a later patch
set.

The dlb driver uses ioctls as its primary interface (it makes use of sysfs
as well, to a lesser extent). The dlb device file supports a different
ioctl interface than the scheduling domain file; the dlb device file
is used for device-wide operations (including scheduling domain creation),
and the scheduling domain file supports operations on the scheduling
domain's resources (primarily resource configuration). Scheduling domains
are created dynamically (using a dlb device file ioctl) by user-space
software, and the scheduling domain file is created from an anonymous file
that is installed in the ioctl's calling process's file descriptor table.

[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html

v10:
- Addressed an issue reported by kernel test robot <lkp@intel.com>
-- Add "WITH Linux-syscall-note" to the SPDX-License-Identifier in uapi
   header file dlb.h.

v9:
- Addressed all of Greg's feecback on v8, including
-- Remove function name (__func__) from dev_err() messages, that could spam log.
-- Replace list and function pointer calls in dlb_ioctl() with switch-case
   and real function calls for ioctl.
-- Drop the compat_ptr_ioctl in dlb_ops (struct file_operations).
-- Change ioctl magic number for DLB to unused 0x81 (from 'h').
-- Remove all placeholder/dummy functions in the patch set.
-- Re-arrange the comments in dlb.h so that the order is consistent with that
   of data structures referred.
-- Correct the comments on SPDX License and DLB versions in dlb.h.
-- Replace BIT_SET() and BITS_CLR() marcos with direct coding.   
-- Remove NULL pointer checking (f->private_data) in dlb_ioctl().
-- Use whole line whenever possible and not wrapping lines unnecessarily.
-- Remove __attribute__((unused)).
-- Merge dlb_ioctl.h and dlb_file.h into dlb_main.h

v8:
- Add a functional block diagram in dlb.rst 
- Modify change logs to reflect the links between patches and DPDK
  eventdev library.
- Add a check of power-of-2 for CQ depth.
- Move call to INIT_WORK() to dlb_open().
- Clean dlb workqueue by calling flush_scheduled_work().
- Add unmap_mapping_range() in dlb_port_close().

v7 (Intel internal version):
- Address all of Dan's feedback, including
-- Drop DLB 2.0 throughout the patch set, use DLB only.
-- Fix license and copyright statements
-- Use pcim_enable_device() and pcim_iomap_regions(), instead of
   unmanaged version.
-- Move cdev_add() to dlb_init() and add all devices at once.
-- Fix Makefile, using "+=" style.
-- Remove FLR description and mention movdir64/enqcmd usage in doc.
-- Make the permission for the domain same as that for device for
   ioctl access.
-- Use idr instead of ida.
-- Add a lock in dlb_close() to prevent driver unbinding while ioctl
   coomands are in progress.
-- Remove wrappers that are used for code sharing between kernel driver
   and DPDK. 
- Address Pierre-Louis' feedback, including
-- Clean the warinings from checkpatch
-- Fix the warnings from "make W=1"

v6 (Intel internal version):
- Change the module name to dlb(from dlb2), which currently supports Intel
  DLB 2.0 only.
- Address all of Pierre-Louis' feedback on v5, including
-- Consolidate the two near-identical for loops in dlb2_release_domain_memory().
-- Remove an unnecessary "port = NULL" initialization
-- Consistently use curly braces on the *_LIST_FOR macros
   when the for-loop contents spans multiple lines.
-- Add a comment to the definition of DLB2FS_MAGIC
-- Remove always true if statemnets
-- Move the get_cos_bw mutex unlock call earlier to shorten the critical
   section.
- Address all of Dan's feedbacks, including
-- Replace the unions for register bits access with bitmask and shifts
-- Centralize the "to/from" user memory copies for ioctl functions.
-- Review ioctl design against Documentation/process/botching-up-ioctls.rst
-- Remove wraper functions for memory barriers.
-- Use ilog() to simplify a switch code block.
-- Add base-commit to cover letter.

v5 (Intel internal version):
- Reduce the scope of the initial patch set (drop the last 8 patches)
- Further decompose some of the remaining patches into multiple patches.
- Address all of Pierre-Louis' feedback, including:
-- Move kerneldoc to *.c files
-- Fix SPDX comment style
-- Add BAR macros
-- Improve/clarify struct dlb2_dev and struct device variable naming
-- Add const where missing
-- Clarify existing comments and add new ones in various places
-- Remove unnecessary memsets and zero-initialization
-- Remove PM abstraction, fix missing pm_runtime_allow(), and don't
   update PM refcnt when port files are opened and closed.
-- Convert certain ternary operations into if-statements
-- Out-line the CQ depth valid check
-- De-duplicate the logic in dlb2_release_device_memory()
-- Limit use of devm functions to allocating/freeing struct dlb2
- Address Ira's comments on dlb2.rst and correct commit messages that
  don't use the imperative voice.

v4:
- Move PCI device ID definitions into dlb2_hw_types.h, drop the VF definition
- Remove dlb2_dev_list
- Remove open/close functions and fops structure (unused)
- Remove "(char *)" cast from PCI driver name
- Unwind init failures properly
- Remove ID alloc helper functions and call IDA interfaces directly instead

v3:
- Remove DLB2_PCI_REG_READ/WRITE macros

v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command

Mike Ximing Chen (20):
  dlb: add skeleton for DLB driver
  dlb: initialize device
  dlb: add resource and device initialization
  dlb: add device ioctl layer and first three ioctls
  dlb: add scheduling domain configuration
  dlb: add domain software reset
  dlb: add low-level register reset operations
  dlb: add runtime power-management support
  dlb: add queue create, reset, get-depth ioctls
  dlb: add register operations for queue management
  dlb: add ioctl to configure ports and query poll mode
  dlb: add register operations for port management
  dlb: add port mmap support
  dlb: add start domain ioctl
  dlb: add queue map, unmap, and pending unmap operations
  dlb: add port map/unmap state machine
  dlb: add static queue map register operations
  dlb: add dynamic queue map register operations
  dlb: add queue unmap register operations
  dlb: queue map/unmap workqueue

 Documentation/misc-devices/dlb.rst            |  259 +
 Documentation/misc-devices/index.rst          |    1 +
 .../userspace-api/ioctl/ioctl-number.rst      |    1 +
 MAINTAINERS                                   |    8 +
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/dlb/Kconfig                      |   18 +
 drivers/misc/dlb/Makefile                     |   11 +
 drivers/misc/dlb/dlb_bitmap.h                 |  210 +
 drivers/misc/dlb/dlb_file.c                   |  149 +
 drivers/misc/dlb/dlb_hw_types.h               |  311 +
 drivers/misc/dlb/dlb_ioctl.c                  |  498 ++
 drivers/misc/dlb/dlb_main.c                   |  614 ++
 drivers/misc/dlb/dlb_main.h                   |  178 +
 drivers/misc/dlb/dlb_pf_ops.c                 |  277 +
 drivers/misc/dlb/dlb_regs.h                   | 3640 +++++++++++
 drivers/misc/dlb/dlb_resource.c               | 5469 +++++++++++++++++
 drivers/misc/dlb/dlb_resource.h               |   94 +
 include/uapi/linux/dlb.h                      |  602 ++
 19 files changed, 12342 insertions(+)
 create mode 100644 Documentation/misc-devices/dlb.rst
 create mode 100644 drivers/misc/dlb/Kconfig
 create mode 100644 drivers/misc/dlb/Makefile
 create mode 100644 drivers/misc/dlb/dlb_bitmap.h
 create mode 100644 drivers/misc/dlb/dlb_file.c
 create mode 100644 drivers/misc/dlb/dlb_hw_types.h
 create mode 100644 drivers/misc/dlb/dlb_ioctl.c
 create mode 100644 drivers/misc/dlb/dlb_main.c
 create mode 100644 drivers/misc/dlb/dlb_main.h
 create mode 100644 drivers/misc/dlb/dlb_pf_ops.c
 create mode 100644 drivers/misc/dlb/dlb_regs.h
 create mode 100644 drivers/misc/dlb/dlb_resource.c
 create mode 100644 drivers/misc/dlb/dlb_resource.h
 create mode 100644 include/uapi/linux/dlb.h


base-commit: e71ba9452f0b5b2e8dc8aa5445198cd9214a6a62
-- 
2.17.1


^ permalink raw reply	[flat|nested] 50+ messages in thread

end of thread, other threads:[~2021-07-16  1:04 UTC | newest]

Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-10 17:54 [PATCH v10 00/20] dlb: introduce DLB device driver Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 01/20] dlb: add skeleton for DLB driver Mike Ximing Chen
2021-02-18  7:34   ` Chen, Mike Ximing
2021-02-18  7:52     ` gregkh
2021-02-18 15:37       ` Chen, Mike Ximing
2021-03-07 13:59       ` Chen, Mike Ximing
2021-02-10 17:54 ` [PATCH v10 02/20] dlb: initialize device Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 03/20] dlb: add resource and device initialization Mike Ximing Chen
2021-03-09  9:24   ` Greg KH
2021-03-10  1:33     ` Chen, Mike Ximing
2021-03-10  8:13       ` Greg KH
2021-03-10 20:26         ` Chen, Mike Ximing
2021-02-10 17:54 ` [PATCH v10 04/20] dlb: add device ioctl layer and first three ioctls Mike Ximing Chen
2021-03-09  9:26   ` Greg KH
2021-03-10  1:34     ` Chen, Mike Ximing
2021-02-10 17:54 ` [PATCH v10 05/20] dlb: add scheduling domain configuration Mike Ximing Chen
2021-03-09  9:28   ` Greg KH
2021-03-10  1:35     ` Chen, Mike Ximing
2021-02-10 17:54 ` [PATCH v10 06/20] dlb: add domain software reset Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 07/20] dlb: add low-level register reset operations Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 08/20] dlb: add runtime power-management support Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 09/20] dlb: add queue create, reset, get-depth ioctls Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 10/20] dlb: add register operations for queue management Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 11/20] dlb: add ioctl to configure ports and query poll mode Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 12/20] dlb: add register operations for port management Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 13/20] dlb: add port mmap support Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 14/20] dlb: add start domain ioctl Mike Ximing Chen
2021-03-09  9:29   ` Greg KH
2021-03-10  2:45     ` Chen, Mike Ximing
2021-03-10  8:14       ` Greg KH
2021-03-10 20:19         ` Dan Williams
2021-03-10 20:26         ` Chen, Mike Ximing
2021-02-10 17:54 ` [PATCH v10 15/20] dlb: add queue map, unmap, and pending unmap operations Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 16/20] dlb: add port map/unmap state machine Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 17/20] dlb: add static queue map register operations Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 18/20] dlb: add dynamic " Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 19/20] dlb: add queue unmap " Mike Ximing Chen
2021-02-10 17:54 ` [PATCH v10 20/20] dlb: queue map/unmap workqueue Mike Ximing Chen
2021-03-10  9:02 ` [PATCH v10 00/20] dlb: introduce DLB device driver Greg KH
2021-03-12  7:18   ` Dan Williams
2021-03-12 21:55     ` Chen, Mike Ximing
2021-03-13  1:39       ` Dan Williams
2021-03-15 20:04         ` Chen, Mike Ximing
2021-03-15 20:08         ` Chen, Mike Ximing
2021-03-15 20:18         ` Chen, Mike Ximing
2021-03-16  9:01           ` Greg KH
2021-05-12 19:07             ` Dan Williams
2021-05-14 14:33               ` Greg KH
2021-07-16  1:04                 ` Chen, Mike Ximing
  -- strict thread matches above, loose matches on Subject: below --
2021-01-27 22:56 Mike Ximing Chen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.