All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-10 18:22 ` Michal Wajdeczko
  0 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-10 18:22 UTC (permalink / raw)
  To: intel-xe, dri-devel
  Cc: Tvrtko Ursulin, Oded Gabbay, Rodrigo Vivi, Michal Wajdeczko

The Single Root I/O Virtualization (SR-IOV) extension to the PCI
Express (PCIe) specification suite is supported starting from 12th
generation of Intel Graphics processors.

This RFC aims to explain how do we want to add support for SR-IOV
to the new Xe driver and to propose related additions to the sysfs.

Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Daniel Vetter <daniel@ffwll.ch>
---
 Documentation/gpu/rfc/index.rst             |   5 +
 Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
 Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
 3 files changed, 698 insertions(+)
 create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
 create mode 100644 Documentation/gpu/rfc/xe_sriov.rst

diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
index e4f7b005138d..fc5bc447f30d 100644
--- a/Documentation/gpu/rfc/index.rst
+++ b/Documentation/gpu/rfc/index.rst
@@ -35,3 +35,8 @@ host such documentation:
 .. toctree::
 
    xe.rst
+
+.. toctree::
+   :maxdepth: 1
+
+   xe_sriov.rst
diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
new file mode 100644
index 000000000000..77748204dd83
--- /dev/null
+++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
@@ -0,0 +1,501 @@
+.. Documentation/ABI/testing/sysfs-driver-xe-sriov
+..
+.. Intel Xe driver ABI (SR-IOV extensions)
+..
+    The Single Root I/O Virtualization (SR-IOV) extension to
+    the PCI Express (PCIe) specification suite is supported
+    starting from 12th generation of Intel Graphics processors.
+
+    This document describes Xe driver specific additions.
+
+    For description of generic SR-IOV sysfs attributes see
+    "Documentation/ABI/testing/sysfs-bus-pci" document.
+
+    /sys/bus/pci/drivers/xe/BDF/
+    ├── sriov_auto_provisioning
+    │   ├── admin_mode
+    │   ├── enabled
+    │   ├── reset_defaults
+    │   ├── resources
+    │   │   ├── default_contexts_quota
+    │   │   ├── default_doorbells_quota
+    │   │   ├── default_ggtt_quota
+    │   │   └── default_lmem_quota
+    │   ├── scheduling
+    │   │   ├── default_exec_quantum_ms
+    │   │   └── default_preempt_timeout_us
+    │   └── monitoring
+    │       ├── default_cat_error_count
+    │       ├── default_doorbell_time_us
+    │       ├── default_engine_reset_count
+    │       ├── default_h2g_time_us
+    │       ├── default_irq_time_us
+    │       └── default_page_fault_count
+
+    /sys/bus/pci/drivers/xe/BDF/
+    ├── sriov_extensions
+    │   ├── monitoring_period_ms
+    │   ├── strict_scheduling_enabled
+    │   ├── pf
+    │   │   ├── device -> ../../../BDF
+    │   │   ├── priority
+    │   │   ├── tile0
+    │   │   │   ├── gt0
+    │   │   │   │   ├── exec_quantum_ms
+    │   │   │   │   ├── preempt_timeout_us
+    │   │   │   │   └── thresholds
+    │   │   │   │       ├── cat_error_count
+    │   │   │   │       ├── doorbell_time_us
+    │   │   │   │       ├── engine_reset_count
+    │   │   │   │       ├── h2g_time_us
+    │   │   │   │       ├── irq_time_us
+    │   │   │   │       └── page_fault_count
+    │   │   │   └── gtX
+    │   │   └── tileT
+    │   ├── vf1
+    │   │   ├── device -> ../../../BDF+1
+    │   │   ├── stop
+    │   │   ├── tile0
+    │   │   │   ├── ggtt_quota
+    │   │   │   ├── lmem_quota
+    │   │   │   ├── gt0
+    │   │   │   │   ├── contexts_quota
+    │   │   │   │   ├── doorbells_quota
+    │   │   │   │   ├── exec_quantum_ms
+    │   │   │   │   ├── preempt_timeout_us
+    │   │   │   │   └── thresholds
+    │   │   │   │       ├── cat_error_count
+    │   │   │   │       ├── doorbell_time_us
+    │   │   │   │       ├── engine_reset_count
+    │   │   │   │       ├── h2g_time_us
+    │   │   │   │       ├── irq_time_us
+    │   │   │   │       └── page_fault_count
+    │   │   │   └── gtX
+    │   │   └── tileT
+    │   └── vfN
+..
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory appears on the device when:
+
+		 - device supports SR-IOV, and
+		 - device is a Physical Function (PF), and
+		 - xe driver supports SR-IOV PF on given device, and
+		 - xe driver supports automatic VFs provisioning.
+
+		This directory is used as a root for all attributes related to
+		automatic provisioning of SR-IOV Physical Function (PF) and/or
+		Virtual Functions (VFs).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool (0, 1)
+
+		This file represents configuration flag for the automatic VFs
+		(un)provisioning that could be performed by the PF.
+
+		The default value is 1 (true).
+
+		This flag can be set to false, unless manual provisioning is not
+		applicable for given platform or it is not supported by current
+		PF implementation. In such cases -EPERM will be returned.
+
+		This flag will be automatically set to false when there will be
+		other attempts to change any of VF's resource provisioning.
+		See "sriov_extensions" section for details.
+
+		This flag can be set back to true if and only if all VFs are
+		fully unprovisioned, otherwise -EEXIST error will be returned.
+
+		false = "disabled"
+			When disabled, then PF will not attempt to do automatic
+			VFs provisioning when VFs are being enabled and will not
+			perform automatic unprovisioning of the VFs when VFs will
+			be disabled.
+
+		true = "enabled"
+			When enabled, then on VFs enabling PF will do automatic
+			VFs provisioning based on the default settings described
+			below.
+
+			If automatic VFs provisioning fails due to some reasons,
+			then VFs will not be enabled.
+
+			If enabled, all resources allocated during VFs enabling
+			will be released during VFs disabling (automatic unprovisioning).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool (0, 1)
+
+		This file represents configuration flag for the automatic VFs
+		provisioning that could be performed by the PF.
+
+		The default value depends on the platform type.
+
+		This flag can be changed any time, but will have no effect if
+		VFs are already provisioned.
+
+		If enabled (default on discrete platforms) then the PF will
+		retain only minimum hardcoded resources for its own use when
+		doing VFs automatic provisioning and will not use any default
+		values described below for its own configuration.
+
+		If disabled (default on integrated platforms) then the PF will
+		treat itself like yet another additional VF in all fair resource
+		allocations and will also try to apply default provisioning
+		values described below for its own configuration.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(WO) bool (1)
+
+		Writing to this file will reset all default provisioning parameters
+		listed below to the default values.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent default provisioning that should be used
+		for VFs automatic provisioning.
+
+		These values can be changed any time, but will have no effect if
+		VFs are already provisioned.
+
+		default_contexts_quota: (RW) integer 0..U32_MAX
+			The number of GuC context IDs to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
+
+		default_doorbells_quota: (RW) integer 0..U32_MAX
+			The number of GuC doorbells to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
+
+		default_ggtt_quota: (RW) integer 0..U32_MAX
+			The size of the GGTT address space (in bytes) to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
+
+		default_lmem_quota: (RW) integer 0..U32_MAX
+			The size of the LMEM (in bytes) to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
+
+		default_exec_quantum_ms: (RW) integer 0..U32_MAX
+			The GT execution quantum (in millisecs) assigned to the function.
+			The default value is 0 (infinify).
+			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
+
+		default_preempt_timeout_us: (RW) integer 0..U32_MAX
+			The GT preemption timeout (in microsecs) assigned to the function.
+			The default value is 0 (infinity).
+			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
+
+		default_cat_error_count: (RW) integer 0..U32_MAX
+		default_doorbell_time_us: (RW) integer 0..U32_MAX
+		default_engine_reset_count: (RW) integer 0..U32_MAX
+		default_h2g_time_us: (RW) integer 0..U32_MAX
+		default_irq_time_us: (RW) integer 0..U32_MAX
+		default_page_fault_count: (RW) integer 0..U32_MAX
+			The monitoring threshold to be set for the function.
+			The default value is 0 (don't monitor).
+			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory appears on Xe device when:
+
+		 - device supports SR-IOV, and
+		 - device is a Physical Function (PF), and
+		 - driver is enabled to support SR-IOV PF on given device.
+
+		This directory is used as a root for all attributes required to
+		manage both Physical Function (PF) and Virtual Functions (VFs).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool
+
+		This file represents a flag used to determine if scheduling
+		parameters should be respected even if there is no active
+		workloads submitted by the PF or VFs.
+
+		This flag is disabled by default, unless strict scheduling is
+		not applicable on given platform. In such case this file will
+		be read-only.
+
+		The change to this file may have no effect if VFs are not yet enabled.
+		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) integer
+
+		This file represents the configuration knob used by adverse event
+		monitoring. A value here is the period in millisecs during which
+		events are counted and the total is checked against a threshold.
+		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
+
+		Default is 0 (monitoring is disabled).
+
+		If monitoring capability is not available, then attempt to enable
+		will fail with -EPERM error. If monitoring can't be enabled in
+		GuC then write will fail with -EIO.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all attributes related to the SR-IOV
+		Physical Function (PF).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all attributes related to the SR-IOV
+		Virtual Function (VF).
+
+		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
+		The Xe driver implementaton follows that naming schema.
+
+		There will be "vf1", "vf2" up to "vfN" directories, where N matches
+		value of the PCI "sriov_totalvfs" attribute.
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all SR-IOV attributes related to the device tile.
+		The tile numbers (T) start from 0.
+
+		There is at least one "tile0/" directory present.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all SR-IOV attributes related to the device GT.
+		The GT numbers (X) start from 0.
+
+		There is at least one "gt0/" directory present.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(symbolic link)
+
+		Backlink to the PCI device entry representing given function.
+		For PF this link is always present.
+		For VF this link is present only for currently enabled VFs.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) string
+
+		This file represents a GuC Scheduler knob to override the default
+		round-robin or FIFO scheduler policies implemented by the GuC.
+
+		The default value is "peer".
+
+		This flag can be changed, unless such change is not applicable
+		for given platform or is not supported by current GuC firmware.
+		In such case this file could be read-only or will return -EPERM
+		on write attempt.
+
+		"immediate"
+			GuC will Schedule PF workloads immediately and PF
+			workloads only until the PF's work queues in GuC
+			are empty.
+
+		"lazy"
+			GuC will Schedule PF workloads at the next opportune
+			moment and PF workloads only until the PF work queues
+			in GuC are empty.
+
+		"peer"
+			GuC Scheduler will treat PF and VFs with equal priority.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(WO) bool (1)
+
+		Write to this file will force GuC to stop handle any requests from
+		this VF, but without triggering a FLR.
+		To recover, the full FLR must be issued using generic "device/reset".
+
+		This file allows to implement custom policy mechanism when VF is
+		misbehaving and triggering adverse events above defined thresholds.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent scheduling parameters of the functions.
+
+		These scheduling parameters can be changed even if VFs are enabled
+		and running, unless such change is not applicable on given platform
+		due to fixed hardware or firmware assignment.
+
+		exec_quantum_ms: (RW) integer 0..U32_MAX
+			The GT execution quantum in [ms] assigned to the function.
+			Requested quantum might be aligned per HW/FW requirements.
+
+			Default is 0 (unlimited).
+
+		preempt_timeout_us: (RW) integer 0..U32_MAX
+			The GT preemption timeout in [us] assigned to the function.
+			Requested timeout might be aligned per HW/FW requirements.
+
+			Default is 0 (unlimited).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent shared resource assigned to the functions.
+
+		These resource parameters can be changed, unless VF is already running,
+		or such change is not applicable on given platform due to fixed hardware
+		or firmware assignment.
+
+		Writes to these attributes may fail with:
+			-EPERM if change is not applicable on give HW/FW.
+			-E2BIG if value larger that HW/FW limit.
+			-EDQUOT if value is larger than maximum quota defined by the PF.
+			-ENOSPC if PF can't allocate required quota.
+			-EBUSY if the resource is currently in use by the VF.
+			-EIO if GuC refuses to change provisioning.
+
+		ggtt_quota: (RW) integer 0..U64_MAX
+			The size of the GGTT address space (in bytes) assigned to the VF.
+			The value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+		lmem_quota: (RW) integer 0..U64_MAX
+			The size of the Local Memory (in bytes) assigned to the VF.
+			The value might be aligned per HW/FW requirements.
+
+			This attribute is only available on discrete platforms.
+
+			Default is 0 (unprovisioned).
+
+		contexts_quota: (RW) 0..U16_MAX
+			The number of GuC submission contexts assigned to the VF.
+			This value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+		doorbells_quota: (RW) 0..U16_MAX
+			The number of GuC doorbells assigned to the VF.
+			This value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent threshold values used by the GuC to trigger
+		security events if adverse event monitoring is enabled.
+
+		These thresholds are checked every "monitoring_period_ms".
+		Refer to GuC ABI for details about each threshold category.
+
+		Default value for all thresholds is 0 (disabled).
+
+		cat_error_count: (RW) integer
+		doorbell_time_us: (RW) integer
+		engine_reset_count: (RW) integer
+		h2g_time_us: (RW) integer
+		irq_time_us: (RW) integer
+		page_fault_count: (RW) integer
diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
new file mode 100644
index 000000000000..574f6414eabb
--- /dev/null
+++ b/Documentation/gpu/rfc/xe_sriov.rst
@@ -0,0 +1,192 @@
+.. SPDX-License-Identifier: MIT
+
+========================
+Xe – SR-IOV Support Plan
+========================
+
+The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
+specification suite is supported starting from 12th generation of Intel Graphics
+processors.
+
+This document describes planned ABI of the new Xe driver (see xe.rst) that will
+provide flexible configuration and management options related to the SR-IOV.
+It will also highlight few most important changes to the Xe driver
+implementation to deal with Intel GPU SR-IOV specific requirements.
+
+
+SR-IOV Capability
+=================
+
+Due to SR-IOV complexity and required co-operation between hardware, firmware
+and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
+or fully functional.
+
+To control at the driver level which platform will provide support for SR-IOV,
+as we can't just rely on the PCI configuration data exposed by the hardware,
+we will introduce "has_sriov" flag to the struct xe_device_desc that describes
+a device capabilities that driver checks during the probe.
+
+Initially this flag will be set to disabled even on platforms that we plan to
+support. We will enable this flag only once we finish merging all required
+changes to the driver and related validated firmwares are also made available.
+
+
+SR-IOV Platforms
+================
+
+Initially we plan to add SR-IOV functionality to the following SDV platforms
+already supported by the Xe driver:
+
+ - TGL (up to 7 VFs)
+ - ADL (up to 7 VFs)
+ - MTL (up to 7 VFs)
+ - ATSM (up to 31 VFs)
+ - PVC (up to 63 VFs)
+
+Newer platforms will be supported later, but we hope that enabling will be
+much faster, as majority of the driver changes are either platform agnostic
+or are similar between earlier platforms (hence we start with SDVs).
+
+
+PF Mode
+=======
+
+Support in the driver for acting in Physical Function (PF) mode, i.e. mode
+that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
+enabled by default.
+
+However, due to potentially conflicting requirements for SR-IOV and other mega
+features, we might want to have an option to disable SR-IOV PF mode support at
+the driver load time.
+
+Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
+0 will force the driver to operate in the native (non-virtualized) mode.
+The same modparam could be used to limit number of supported Virtual Functions
+(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
+
+The name of this modparam corresponds to the existing PCI sysfs attribute, that
+by default exposes hardware capability.
+
+The default value of this param will allow to support all possible VFs as
+claimed by the hardware.
+
+This modparam will have no effect if driver is running on the VF device.
+
+
+VFs Enabling
+============
+
+To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
+the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
+provision VFs in our custom implementation of the "sriov_configure" hook from
+the struct pci_driver.
+
+If for some reason, including explicit request to disable SR-IOV PF mode using
+modparam, we will not be able to correctly support any VFs, driver will change
+number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
+to 0, thus preventing configuration of the VFs.
+
+
+VF Mode
+=======
+
+When driver is running on the VF device, then due to hardware enforcements,
+access to the privileged registers is not possible. To avoid relying on these
+registers, we plan to perform early detection if we are running on the VF
+device using dedicated VF_CAP(0x1901f8) register and then use global macro
+IS_SRIOV_VF(xe) to control the driver logic.
+
+To speed up merging of the required changes, we might first introduce dummy
+macro that is always set to false, to prepare driver to avoid some code paths
+before we finalize our VF mode detection and other VFs enabling changes.
+
+
+Resources
+=========
+
+Most of the hardware (or firmware) resources available on the Xe architecture,
+like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
+VFs and will require some provisioning steps to assign those resources for use
+by the VF.
+
+Until VFs are provisioned with resources, the PF driver will be able to use all
+resources, in the same way as it would be running in non-virtualized mode.
+
+If some resource (of part or region of it) is assigned to specific VF, then PF
+is not allowed to use that part or region of the resource, but can continue to
+use whatever is left available.
+
+Those resources are usually fully virtualized, so they will not require any
+special handling when used by the VF driver, except that VF driver must know
+the assigned quota.
+
+The most notable exception is the GGTT address space, as on some platforms,
+the VF driver must additionally know the real range that it can access.
+
+Once the resources were assigned to the VF use and the VF driver has started,
+then it is not allowed to change such provisioning, as that would break the
+VF driver. To make changes the VF driver, which was using these resources,
+must be unloaded (or the VM is terminated) and the VF device must be reset
+using the FLR.
+
+
+Scheduling
+==========
+
+The workloads from PF driver and VF drivers must be submitted to the hardware
+always by using the GuC submission mechanism. Unless VF has exclusive access
+to the GT then submissions from different VFs are time-sliced and controlled
+with additional "execution_quantum" and "preemption_timeout" parameters.
+
+In contrast to the resource provisioning, those scheduling parameters can be
+changed even if VF drivers are already running and are active.
+
+
+Automatic VFs Provisioning
+==========================
+
+To provide out-of-the box experience when user will be enabling VFs using
+generic "sriov_numvfs" attribute without requiring complex provisioning steps,
+the SR-IOV PF driver will implement automatic VFs resource provisioning.
+
+By default, all VFs will be allocated with the fair amount of the mandatory
+resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
+Such provisioning should be sufficient for most of the normal usages, when
+no strict SLA is required.
+
+The PF driver will also expose some additional sysfs files to allow adjusting
+this automatic VFs provisioning, like default values for most of the
+provisioning parameters that PF will then apply for each enabled VF.
+
+    Details about those extension can be found in
+    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
+
+
+Manual VFs Provisioning
+=======================
+
+If automatic VFs provisioning, which applies same configuration to every VF,
+is not sufficient or there is a need for advanced customization of some VF,
+the PF driver will also provide extended sysfs interface which will allow
+control every provisioning attribute to the lowest feasible level.
+
+It is expected that these low-level attributes will be mostly used by the
+advanced users or by the custom tools that will setup configurations that
+meet predefined and validated SLA as required by the customers.
+
+    Details about those extension can be found in
+    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
+
+
+VFs Monitoring
+==============
+
+In addition to the resource provisioning or changing scheduling parameters,
+the PF driver might also allow configure some monitoring parameters, like
+thresholds of adverse events or sample period, to track undesired behavior
+of the VFs that could impact the whole system.
+
+Once those thresholds are setup and sampling period is defined, the GuC will
+notify the PF driver about which VF is excessing the threshold and then PF is
+able to trigger the uevent to notify the administrator (or VMM) that could
+take some action against the VF.
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-10 18:22 ` Michal Wajdeczko
  0 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-10 18:22 UTC (permalink / raw)
  To: intel-xe, dri-devel; +Cc: Tvrtko Ursulin, Daniel Vetter, Rodrigo Vivi

The Single Root I/O Virtualization (SR-IOV) extension to the PCI
Express (PCIe) specification suite is supported starting from 12th
generation of Intel Graphics processors.

This RFC aims to explain how do we want to add support for SR-IOV
to the new Xe driver and to propose related additions to the sysfs.

Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Daniel Vetter <daniel@ffwll.ch>
---
 Documentation/gpu/rfc/index.rst             |   5 +
 Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
 Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
 3 files changed, 698 insertions(+)
 create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
 create mode 100644 Documentation/gpu/rfc/xe_sriov.rst

diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
index e4f7b005138d..fc5bc447f30d 100644
--- a/Documentation/gpu/rfc/index.rst
+++ b/Documentation/gpu/rfc/index.rst
@@ -35,3 +35,8 @@ host such documentation:
 .. toctree::
 
    xe.rst
+
+.. toctree::
+   :maxdepth: 1
+
+   xe_sriov.rst
diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
new file mode 100644
index 000000000000..77748204dd83
--- /dev/null
+++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
@@ -0,0 +1,501 @@
+.. Documentation/ABI/testing/sysfs-driver-xe-sriov
+..
+.. Intel Xe driver ABI (SR-IOV extensions)
+..
+    The Single Root I/O Virtualization (SR-IOV) extension to
+    the PCI Express (PCIe) specification suite is supported
+    starting from 12th generation of Intel Graphics processors.
+
+    This document describes Xe driver specific additions.
+
+    For description of generic SR-IOV sysfs attributes see
+    "Documentation/ABI/testing/sysfs-bus-pci" document.
+
+    /sys/bus/pci/drivers/xe/BDF/
+    ├── sriov_auto_provisioning
+    │   ├── admin_mode
+    │   ├── enabled
+    │   ├── reset_defaults
+    │   ├── resources
+    │   │   ├── default_contexts_quota
+    │   │   ├── default_doorbells_quota
+    │   │   ├── default_ggtt_quota
+    │   │   └── default_lmem_quota
+    │   ├── scheduling
+    │   │   ├── default_exec_quantum_ms
+    │   │   └── default_preempt_timeout_us
+    │   └── monitoring
+    │       ├── default_cat_error_count
+    │       ├── default_doorbell_time_us
+    │       ├── default_engine_reset_count
+    │       ├── default_h2g_time_us
+    │       ├── default_irq_time_us
+    │       └── default_page_fault_count
+
+    /sys/bus/pci/drivers/xe/BDF/
+    ├── sriov_extensions
+    │   ├── monitoring_period_ms
+    │   ├── strict_scheduling_enabled
+    │   ├── pf
+    │   │   ├── device -> ../../../BDF
+    │   │   ├── priority
+    │   │   ├── tile0
+    │   │   │   ├── gt0
+    │   │   │   │   ├── exec_quantum_ms
+    │   │   │   │   ├── preempt_timeout_us
+    │   │   │   │   └── thresholds
+    │   │   │   │       ├── cat_error_count
+    │   │   │   │       ├── doorbell_time_us
+    │   │   │   │       ├── engine_reset_count
+    │   │   │   │       ├── h2g_time_us
+    │   │   │   │       ├── irq_time_us
+    │   │   │   │       └── page_fault_count
+    │   │   │   └── gtX
+    │   │   └── tileT
+    │   ├── vf1
+    │   │   ├── device -> ../../../BDF+1
+    │   │   ├── stop
+    │   │   ├── tile0
+    │   │   │   ├── ggtt_quota
+    │   │   │   ├── lmem_quota
+    │   │   │   ├── gt0
+    │   │   │   │   ├── contexts_quota
+    │   │   │   │   ├── doorbells_quota
+    │   │   │   │   ├── exec_quantum_ms
+    │   │   │   │   ├── preempt_timeout_us
+    │   │   │   │   └── thresholds
+    │   │   │   │       ├── cat_error_count
+    │   │   │   │       ├── doorbell_time_us
+    │   │   │   │       ├── engine_reset_count
+    │   │   │   │       ├── h2g_time_us
+    │   │   │   │       ├── irq_time_us
+    │   │   │   │       └── page_fault_count
+    │   │   │   └── gtX
+    │   │   └── tileT
+    │   └── vfN
+..
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory appears on the device when:
+
+		 - device supports SR-IOV, and
+		 - device is a Physical Function (PF), and
+		 - xe driver supports SR-IOV PF on given device, and
+		 - xe driver supports automatic VFs provisioning.
+
+		This directory is used as a root for all attributes related to
+		automatic provisioning of SR-IOV Physical Function (PF) and/or
+		Virtual Functions (VFs).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool (0, 1)
+
+		This file represents configuration flag for the automatic VFs
+		(un)provisioning that could be performed by the PF.
+
+		The default value is 1 (true).
+
+		This flag can be set to false, unless manual provisioning is not
+		applicable for given platform or it is not supported by current
+		PF implementation. In such cases -EPERM will be returned.
+
+		This flag will be automatically set to false when there will be
+		other attempts to change any of VF's resource provisioning.
+		See "sriov_extensions" section for details.
+
+		This flag can be set back to true if and only if all VFs are
+		fully unprovisioned, otherwise -EEXIST error will be returned.
+
+		false = "disabled"
+			When disabled, then PF will not attempt to do automatic
+			VFs provisioning when VFs are being enabled and will not
+			perform automatic unprovisioning of the VFs when VFs will
+			be disabled.
+
+		true = "enabled"
+			When enabled, then on VFs enabling PF will do automatic
+			VFs provisioning based on the default settings described
+			below.
+
+			If automatic VFs provisioning fails due to some reasons,
+			then VFs will not be enabled.
+
+			If enabled, all resources allocated during VFs enabling
+			will be released during VFs disabling (automatic unprovisioning).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool (0, 1)
+
+		This file represents configuration flag for the automatic VFs
+		provisioning that could be performed by the PF.
+
+		The default value depends on the platform type.
+
+		This flag can be changed any time, but will have no effect if
+		VFs are already provisioned.
+
+		If enabled (default on discrete platforms) then the PF will
+		retain only minimum hardcoded resources for its own use when
+		doing VFs automatic provisioning and will not use any default
+		values described below for its own configuration.
+
+		If disabled (default on integrated platforms) then the PF will
+		treat itself like yet another additional VF in all fair resource
+		allocations and will also try to apply default provisioning
+		values described below for its own configuration.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(WO) bool (1)
+
+		Writing to this file will reset all default provisioning parameters
+		listed below to the default values.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent default provisioning that should be used
+		for VFs automatic provisioning.
+
+		These values can be changed any time, but will have no effect if
+		VFs are already provisioned.
+
+		default_contexts_quota: (RW) integer 0..U32_MAX
+			The number of GuC context IDs to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
+
+		default_doorbells_quota: (RW) integer 0..U32_MAX
+			The number of GuC doorbells to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
+
+		default_ggtt_quota: (RW) integer 0..U32_MAX
+			The size of the GGTT address space (in bytes) to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
+
+		default_lmem_quota: (RW) integer 0..U32_MAX
+			The size of the LMEM (in bytes) to provide to the VF.
+			The default value is 0 (use fair allocations).
+			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
+
+		default_exec_quantum_ms: (RW) integer 0..U32_MAX
+			The GT execution quantum (in millisecs) assigned to the function.
+			The default value is 0 (infinify).
+			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
+
+		default_preempt_timeout_us: (RW) integer 0..U32_MAX
+			The GT preemption timeout (in microsecs) assigned to the function.
+			The default value is 0 (infinity).
+			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
+
+		default_cat_error_count: (RW) integer 0..U32_MAX
+		default_doorbell_time_us: (RW) integer 0..U32_MAX
+		default_engine_reset_count: (RW) integer 0..U32_MAX
+		default_h2g_time_us: (RW) integer 0..U32_MAX
+		default_irq_time_us: (RW) integer 0..U32_MAX
+		default_page_fault_count: (RW) integer 0..U32_MAX
+			The monitoring threshold to be set for the function.
+			The default value is 0 (don't monitor).
+			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory appears on Xe device when:
+
+		 - device supports SR-IOV, and
+		 - device is a Physical Function (PF), and
+		 - driver is enabled to support SR-IOV PF on given device.
+
+		This directory is used as a root for all attributes required to
+		manage both Physical Function (PF) and Virtual Functions (VFs).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) bool
+
+		This file represents a flag used to determine if scheduling
+		parameters should be respected even if there is no active
+		workloads submitted by the PF or VFs.
+
+		This flag is disabled by default, unless strict scheduling is
+		not applicable on given platform. In such case this file will
+		be read-only.
+
+		The change to this file may have no effect if VFs are not yet enabled.
+		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) integer
+
+		This file represents the configuration knob used by adverse event
+		monitoring. A value here is the period in millisecs during which
+		events are counted and the total is checked against a threshold.
+		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
+
+		Default is 0 (monitoring is disabled).
+
+		If monitoring capability is not available, then attempt to enable
+		will fail with -EPERM error. If monitoring can't be enabled in
+		GuC then write will fail with -EIO.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all attributes related to the SR-IOV
+		Physical Function (PF).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all attributes related to the SR-IOV
+		Virtual Function (VF).
+
+		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
+		The Xe driver implementaton follows that naming schema.
+
+		There will be "vf1", "vf2" up to "vfN" directories, where N matches
+		value of the PCI "sriov_totalvfs" attribute.
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all SR-IOV attributes related to the device tile.
+		The tile numbers (T) start from 0.
+
+		There is at least one "tile0/" directory present.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		This directory holds all SR-IOV attributes related to the device GT.
+		The GT numbers (X) start from 0.
+
+		There is at least one "gt0/" directory present.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(symbolic link)
+
+		Backlink to the PCI device entry representing given function.
+		For PF this link is always present.
+		For VF this link is present only for currently enabled VFs.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(RW) string
+
+		This file represents a GuC Scheduler knob to override the default
+		round-robin or FIFO scheduler policies implemented by the GuC.
+
+		The default value is "peer".
+
+		This flag can be changed, unless such change is not applicable
+		for given platform or is not supported by current GuC firmware.
+		In such case this file could be read-only or will return -EPERM
+		on write attempt.
+
+		"immediate"
+			GuC will Schedule PF workloads immediately and PF
+			workloads only until the PF's work queues in GuC
+			are empty.
+
+		"lazy"
+			GuC will Schedule PF workloads at the next opportune
+			moment and PF workloads only until the PF work queues
+			in GuC are empty.
+
+		"peer"
+			GuC Scheduler will treat PF and VFs with equal priority.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		(WO) bool (1)
+
+		Write to this file will force GuC to stop handle any requests from
+		this VF, but without triggering a FLR.
+		To recover, the full FLR must be issued using generic "device/reset".
+
+		This file allows to implement custom policy mechanism when VF is
+		misbehaving and triggering adverse events above defined thresholds.
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent scheduling parameters of the functions.
+
+		These scheduling parameters can be changed even if VFs are enabled
+		and running, unless such change is not applicable on given platform
+		due to fixed hardware or firmware assignment.
+
+		exec_quantum_ms: (RW) integer 0..U32_MAX
+			The GT execution quantum in [ms] assigned to the function.
+			Requested quantum might be aligned per HW/FW requirements.
+
+			Default is 0 (unlimited).
+
+		preempt_timeout_us: (RW) integer 0..U32_MAX
+			The GT preemption timeout in [us] assigned to the function.
+			Requested timeout might be aligned per HW/FW requirements.
+
+			Default is 0 (unlimited).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent shared resource assigned to the functions.
+
+		These resource parameters can be changed, unless VF is already running,
+		or such change is not applicable on given platform due to fixed hardware
+		or firmware assignment.
+
+		Writes to these attributes may fail with:
+			-EPERM if change is not applicable on give HW/FW.
+			-E2BIG if value larger that HW/FW limit.
+			-EDQUOT if value is larger than maximum quota defined by the PF.
+			-ENOSPC if PF can't allocate required quota.
+			-EBUSY if the resource is currently in use by the VF.
+			-EIO if GuC refuses to change provisioning.
+
+		ggtt_quota: (RW) integer 0..U64_MAX
+			The size of the GGTT address space (in bytes) assigned to the VF.
+			The value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+		lmem_quota: (RW) integer 0..U64_MAX
+			The size of the Local Memory (in bytes) assigned to the VF.
+			The value might be aligned per HW/FW requirements.
+
+			This attribute is only available on discrete platforms.
+
+			Default is 0 (unprovisioned).
+
+		contexts_quota: (RW) 0..U16_MAX
+			The number of GuC submission contexts assigned to the VF.
+			This value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+		doorbells_quota: (RW) 0..U16_MAX
+			The number of GuC doorbells assigned to the VF.
+			This value might be aligned per HW/FW requirements.
+
+			Default is 0 (unprovisioned).
+
+
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
+What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
+Date:		2024
+KernelVersion:	TBD
+Contact:	intel-xe@lists.freedesktop.org
+Description:
+		These files represent threshold values used by the GuC to trigger
+		security events if adverse event monitoring is enabled.
+
+		These thresholds are checked every "monitoring_period_ms".
+		Refer to GuC ABI for details about each threshold category.
+
+		Default value for all thresholds is 0 (disabled).
+
+		cat_error_count: (RW) integer
+		doorbell_time_us: (RW) integer
+		engine_reset_count: (RW) integer
+		h2g_time_us: (RW) integer
+		irq_time_us: (RW) integer
+		page_fault_count: (RW) integer
diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
new file mode 100644
index 000000000000..574f6414eabb
--- /dev/null
+++ b/Documentation/gpu/rfc/xe_sriov.rst
@@ -0,0 +1,192 @@
+.. SPDX-License-Identifier: MIT
+
+========================
+Xe – SR-IOV Support Plan
+========================
+
+The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
+specification suite is supported starting from 12th generation of Intel Graphics
+processors.
+
+This document describes planned ABI of the new Xe driver (see xe.rst) that will
+provide flexible configuration and management options related to the SR-IOV.
+It will also highlight few most important changes to the Xe driver
+implementation to deal with Intel GPU SR-IOV specific requirements.
+
+
+SR-IOV Capability
+=================
+
+Due to SR-IOV complexity and required co-operation between hardware, firmware
+and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
+or fully functional.
+
+To control at the driver level which platform will provide support for SR-IOV,
+as we can't just rely on the PCI configuration data exposed by the hardware,
+we will introduce "has_sriov" flag to the struct xe_device_desc that describes
+a device capabilities that driver checks during the probe.
+
+Initially this flag will be set to disabled even on platforms that we plan to
+support. We will enable this flag only once we finish merging all required
+changes to the driver and related validated firmwares are also made available.
+
+
+SR-IOV Platforms
+================
+
+Initially we plan to add SR-IOV functionality to the following SDV platforms
+already supported by the Xe driver:
+
+ - TGL (up to 7 VFs)
+ - ADL (up to 7 VFs)
+ - MTL (up to 7 VFs)
+ - ATSM (up to 31 VFs)
+ - PVC (up to 63 VFs)
+
+Newer platforms will be supported later, but we hope that enabling will be
+much faster, as majority of the driver changes are either platform agnostic
+or are similar between earlier platforms (hence we start with SDVs).
+
+
+PF Mode
+=======
+
+Support in the driver for acting in Physical Function (PF) mode, i.e. mode
+that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
+enabled by default.
+
+However, due to potentially conflicting requirements for SR-IOV and other mega
+features, we might want to have an option to disable SR-IOV PF mode support at
+the driver load time.
+
+Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
+0 will force the driver to operate in the native (non-virtualized) mode.
+The same modparam could be used to limit number of supported Virtual Functions
+(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
+
+The name of this modparam corresponds to the existing PCI sysfs attribute, that
+by default exposes hardware capability.
+
+The default value of this param will allow to support all possible VFs as
+claimed by the hardware.
+
+This modparam will have no effect if driver is running on the VF device.
+
+
+VFs Enabling
+============
+
+To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
+the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
+provision VFs in our custom implementation of the "sriov_configure" hook from
+the struct pci_driver.
+
+If for some reason, including explicit request to disable SR-IOV PF mode using
+modparam, we will not be able to correctly support any VFs, driver will change
+number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
+to 0, thus preventing configuration of the VFs.
+
+
+VF Mode
+=======
+
+When driver is running on the VF device, then due to hardware enforcements,
+access to the privileged registers is not possible. To avoid relying on these
+registers, we plan to perform early detection if we are running on the VF
+device using dedicated VF_CAP(0x1901f8) register and then use global macro
+IS_SRIOV_VF(xe) to control the driver logic.
+
+To speed up merging of the required changes, we might first introduce dummy
+macro that is always set to false, to prepare driver to avoid some code paths
+before we finalize our VF mode detection and other VFs enabling changes.
+
+
+Resources
+=========
+
+Most of the hardware (or firmware) resources available on the Xe architecture,
+like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
+VFs and will require some provisioning steps to assign those resources for use
+by the VF.
+
+Until VFs are provisioned with resources, the PF driver will be able to use all
+resources, in the same way as it would be running in non-virtualized mode.
+
+If some resource (of part or region of it) is assigned to specific VF, then PF
+is not allowed to use that part or region of the resource, but can continue to
+use whatever is left available.
+
+Those resources are usually fully virtualized, so they will not require any
+special handling when used by the VF driver, except that VF driver must know
+the assigned quota.
+
+The most notable exception is the GGTT address space, as on some platforms,
+the VF driver must additionally know the real range that it can access.
+
+Once the resources were assigned to the VF use and the VF driver has started,
+then it is not allowed to change such provisioning, as that would break the
+VF driver. To make changes the VF driver, which was using these resources,
+must be unloaded (or the VM is terminated) and the VF device must be reset
+using the FLR.
+
+
+Scheduling
+==========
+
+The workloads from PF driver and VF drivers must be submitted to the hardware
+always by using the GuC submission mechanism. Unless VF has exclusive access
+to the GT then submissions from different VFs are time-sliced and controlled
+with additional "execution_quantum" and "preemption_timeout" parameters.
+
+In contrast to the resource provisioning, those scheduling parameters can be
+changed even if VF drivers are already running and are active.
+
+
+Automatic VFs Provisioning
+==========================
+
+To provide out-of-the box experience when user will be enabling VFs using
+generic "sriov_numvfs" attribute without requiring complex provisioning steps,
+the SR-IOV PF driver will implement automatic VFs resource provisioning.
+
+By default, all VFs will be allocated with the fair amount of the mandatory
+resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
+Such provisioning should be sufficient for most of the normal usages, when
+no strict SLA is required.
+
+The PF driver will also expose some additional sysfs files to allow adjusting
+this automatic VFs provisioning, like default values for most of the
+provisioning parameters that PF will then apply for each enabled VF.
+
+    Details about those extension can be found in
+    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
+
+
+Manual VFs Provisioning
+=======================
+
+If automatic VFs provisioning, which applies same configuration to every VF,
+is not sufficient or there is a need for advanced customization of some VF,
+the PF driver will also provide extended sysfs interface which will allow
+control every provisioning attribute to the lowest feasible level.
+
+It is expected that these low-level attributes will be mostly used by the
+advanced users or by the custom tools that will setup configurations that
+meet predefined and validated SLA as required by the customers.
+
+    Details about those extension can be found in
+    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
+
+
+VFs Monitoring
+==============
+
+In addition to the resource provisioning or changing scheduling parameters,
+the PF driver might also allow configure some monitoring parameters, like
+thresholds of adverse events or sample period, to track undesired behavior
+of the VFs that could impact the whole system.
+
+Once those thresholds are setup and sampling period is defined, the GuC will
+notify the PF driver about which VF is excessing the threshold and then PF is
+able to trigger the uevent to notify the administrator (or VMM) that could
+take some action against the VF.
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.Patch_applied: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
  (?)
@ 2023-11-10 18:25 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:25 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

=== Applying kernel patches on branch 'drm-xe-next' with base: ===
Base commit: b3890fb43 drm/xe: Drop EXECLIST_CONTROL from error state dump
=== git am output follows ===
Applying: drm/doc/rfc: SR-IOV support on the new Xe driver



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✗ CI.checkpatch: warning for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
  (?)
  (?)
@ 2023-11-10 18:25 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:25 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : warning

== Summary ==

+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt
Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master
63c2b6b160bca2df6efc7bc4cea6f442097d7854
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit c5d52ad5698f36efba079671726a71176862b722
Author: Michal Wajdeczko <michal.wajdeczko@intel.com>
Date:   Fri Nov 10 19:22:31 2023 +0100

    drm/doc/rfc: SR-IOV support on the new Xe driver
    
    The Single Root I/O Virtualization (SR-IOV) extension to the PCI
    Express (PCIe) specification suite is supported starting from 12th
    generation of Intel Graphics processors.
    
    This RFC aims to explain how do we want to add support for SR-IOV
    to the new Xe driver and to propose related additions to the sysfs.
    
    Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
    Cc: Oded Gabbay <ogabbay@kernel.org>
    Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
    Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
    Cc: Daniel Vetter <daniel@ffwll.ch>
+ /mt/dim checkpatch b3890fb4386dcef68a96888141c4cc773f6241ce drm-intel
c5d52ad56 drm/doc/rfc: SR-IOV support on the new Xe driver
Traceback (most recent call last):
  File "scripts/spdxcheck.py", line 6, in <module>
    from ply import lex, yacc
ModuleNotFoundError: No module named 'ply'
-:34: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#34: 
new file mode 100644

total: 0 errors, 1 warnings, 0 checks, 701 lines checked



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.KUnit: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
                   ` (2 preceding siblings ...)
  (?)
@ 2023-11-10 18:27 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:27 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig
stty: 'standard input': Inappropriate ioctl for device
[18:25:49] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
[18:25:54] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
[18:26:14] Starting KUnit Kernel (1/1)...
[18:26:14] ============================================================
[18:26:14] ========================== xe_bo  ==========================
[18:26:14] [SKIPPED] xe_ccs_migrate_kunit
[18:26:14] [SKIPPED] xe_bo_evict_kunit
[18:26:14] ===================== [SKIPPED] xe_bo ======================
[18:26:14] ======================= xe_dma_buf  ========================
[18:26:14] [SKIPPED] xe_dma_buf_kunit
[18:26:14] =================== [SKIPPED] xe_dma_buf ===================
[18:26:14] ======================= xe_migrate  ========================
[18:26:14] [SKIPPED] xe_migrate_sanity_kunit
[18:26:14] =================== [SKIPPED] xe_migrate ===================
[18:26:14] ========================= xe_pci  ==========================
[18:26:14] [PASSED] xe_gmdid_graphics_ip
[18:26:14] [PASSED] xe_gmdid_media_ip
[18:26:14] ===================== [PASSED] xe_pci ======================
[18:26:14] ========================= xe_rtp  ==========================
[18:26:14] ================== xe_rtp_process_tests  ===================
[18:26:14] [PASSED] coalesce-same-reg
[18:26:14] [PASSED] no-match-no-add
[18:26:14] [PASSED] no-match-no-add-multiple-rules
[18:26:14] [PASSED] two-regs-two-entries
[18:26:14] [PASSED] clr-one-set-other
[18:26:14] [PASSED] set-field
[18:26:14] [PASSED] conflict-duplicate
[18:26:14] [PASSED] conflict-not-disjoint
[18:26:14] [PASSED] conflict-reg-type
[18:26:14] ============== [PASSED] xe_rtp_process_tests ===============
[18:26:14] ===================== [PASSED] xe_rtp ======================
[18:26:14] ========================== xe_wa  ==========================
[18:26:14] ======================== xe_wa_gt  =========================
[18:26:14] [PASSED] TIGERLAKE (B0)
[18:26:14] [PASSED] DG1 (A0)
[18:26:14] [PASSED] DG1 (B0)
[18:26:14] [PASSED] ALDERLAKE_S (A0)
[18:26:14] [PASSED] ALDERLAKE_S (B0)
[18:26:14] [PASSED] ALDERLAKE_S (C0)
[18:26:14] [PASSED] ALDERLAKE_S (D0)
[18:26:14] [PASSED] ALDERLAKE_P (A0)
[18:26:14] [PASSED] ALDERLAKE_P (B0)
[18:26:14] [PASSED] ALDERLAKE_P (C0)
[18:26:14] [PASSED] ALDERLAKE_S_RPLS (D0)
[18:26:14] [PASSED] ALDERLAKE_P_RPLU (E0)
[18:26:14] [PASSED] DG2_G10 (A0)
[18:26:14] [PASSED] DG2_G10 (A1)
[18:26:14] [PASSED] DG2_G10 (B0)
[18:26:14] [PASSED] DG2_G10 (C0)
[18:26:14] [PASSED] DG2_G11 (A0)
[18:26:14] [PASSED] DG2_G11 (B0)
[18:26:14] [PASSED] DG2_G11 (B1)
[18:26:14] [PASSED] DG2_G12 (A0)
[18:26:14] [PASSED] DG2_G12 (A1)
[18:26:14] [PASSED] PVC (B0)
[18:26:14] [PASSED] PVC (B1)
[18:26:14] [PASSED] PVC (C0)
[18:26:14] ==================== [PASSED] xe_wa_gt =====================
[18:26:14] ====================== [PASSED] xe_wa ======================
[18:26:14] ============================================================
[18:26:14] Testing complete. Ran 39 tests: passed: 35, skipped: 4
[18:26:14] Elapsed time: 24.615s total, 4.210s configuring, 20.284s building, 0.101s running

+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/tests/.kunitconfig
[18:26:14] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
[18:26:16] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make ARCH=um O=.kunit --jobs=48
[18:26:35] Starting KUnit Kernel (1/1)...
[18:26:35] ============================================================
[18:26:35] ================== drm_test_pick_cmdline  ==================
[18:26:35] [PASSED] drm_test_pick_cmdline_res_1920_1080_60
[18:26:35] =============== drm_test_pick_cmdline_named  ===============
[18:26:35] [PASSED] NTSC
[18:26:35] [PASSED] NTSC-J
[18:26:35] [PASSED] PAL
[18:26:35] [PASSED] PAL-M
[18:26:35] =========== [PASSED] drm_test_pick_cmdline_named ===========
[18:26:35] ============== [PASSED] drm_test_pick_cmdline ==============
[18:26:35] ======================== drm_buddy  ========================
[18:26:35] [PASSED] drm_test_buddy_alloc_limit
[18:26:35] [PASSED] drm_test_buddy_alloc_range
[18:26:35] [PASSED] drm_test_buddy_alloc_optimistic
[18:26:35] [PASSED] drm_test_buddy_alloc_pessimistic
[18:26:35] [PASSED] drm_test_buddy_alloc_smoke
[18:26:35] [PASSED] drm_test_buddy_alloc_pathological
[18:26:35] ==================== [PASSED] drm_buddy ====================
[18:26:35] =================== drm_cmdline_parser  ====================
[18:26:35] [PASSED] drm_test_cmdline_force_d_only
[18:26:35] [PASSED] drm_test_cmdline_force_D_only_dvi
[18:26:35] [PASSED] drm_test_cmdline_force_D_only_hdmi
[18:26:35] [PASSED] drm_test_cmdline_force_D_only_not_digital
[18:26:35] [PASSED] drm_test_cmdline_force_e_only
[18:26:35] [PASSED] drm_test_cmdline_res
[18:26:35] [PASSED] drm_test_cmdline_res_vesa
[18:26:35] [PASSED] drm_test_cmdline_res_vesa_rblank
[18:26:35] [PASSED] drm_test_cmdline_res_rblank
[18:26:35] [PASSED] drm_test_cmdline_res_bpp
[18:26:35] [PASSED] drm_test_cmdline_res_refresh
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_margins
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_force_off
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_analog
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_digital
[18:26:35] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced_margins_force_on
[18:26:35] [PASSED] drm_test_cmdline_res_margins_force_on
[18:26:35] [PASSED] drm_test_cmdline_res_vesa_margins
[18:26:35] [PASSED] drm_test_cmdline_name
[18:26:35] [PASSED] drm_test_cmdline_name_bpp
[18:26:35] [PASSED] drm_test_cmdline_name_option
[18:26:35] [PASSED] drm_test_cmdline_name_bpp_option
[18:26:35] [PASSED] drm_test_cmdline_rotate_0
[18:26:35] [PASSED] drm_test_cmdline_rotate_90
[18:26:35] [PASSED] drm_test_cmdline_rotate_180
[18:26:35] [PASSED] drm_test_cmdline_rotate_270
[18:26:35] [PASSED] drm_test_cmdline_hmirror
[18:26:35] [PASSED] drm_test_cmdline_vmirror
[18:26:35] [PASSED] drm_test_cmdline_margin_options
[18:26:35] [PASSED] drm_test_cmdline_multiple_options
[18:26:35] [PASSED] drm_test_cmdline_bpp_extra_and_option
[18:26:35] [PASSED] drm_test_cmdline_extra_and_option
[18:26:35] [PASSED] drm_test_cmdline_freestanding_options
[18:26:35] [PASSED] drm_test_cmdline_freestanding_force_e_and_options
[18:26:35] [PASSED] drm_test_cmdline_panel_orientation
[18:26:35] ================ drm_test_cmdline_invalid  =================
[18:26:35] [PASSED] margin_only
[18:26:35] [PASSED] interlace_only
[18:26:35] [PASSED] res_missing_x
[18:26:35] [PASSED] res_missing_y
[18:26:35] [PASSED] res_bad_y
[18:26:35] [PASSED] res_missing_y_bpp
[18:26:35] [PASSED] res_bad_bpp
[18:26:35] [PASSED] res_bad_refresh
[18:26:35] [PASSED] res_bpp_refresh_force_on_off
[18:26:35] [PASSED] res_invalid_mode
[18:26:35] [PASSED] res_bpp_wrong_place_mode
[18:26:35] [PASSED] name_bpp_refresh
[18:26:35] [PASSED] name_refresh
[18:26:35] [PASSED] name_refresh_wrong_mode
[18:26:35] [PASSED] name_refresh_invalid_mode
[18:26:35] [PASSED] rotate_multiple
[18:26:35] [PASSED] rotate_invalid_val
[18:26:35] [PASSED] rotate_truncated
[18:26:35] [PASSED] invalid_option
[18:26:35] [PASSED] invalid_tv_option
[18:26:35] [PASSED] truncated_tv_option
[18:26:35] ============ [PASSED] drm_test_cmdline_invalid =============
[18:26:35] =============== drm_test_cmdline_tv_options  ===============
[18:26:35] [PASSED] NTSC
[18:26:35] [PASSED] NTSC_443
[18:26:35] [PASSED] NTSC_J
[18:26:35] [PASSED] PAL
[18:26:35] [PASSED] PAL_M
[18:26:35] [PASSED] PAL_N
[18:26:35] [PASSED] SECAM
[18:26:35] =========== [PASSED] drm_test_cmdline_tv_options ===========
[18:26:35] =============== [PASSED] drm_cmdline_parser ================
[18:26:35] ================ drm_get_tv_mode_from_name  ================
[18:26:35] ========== drm_test_get_tv_mode_from_name_valid  ===========
[18:26:35] [PASSED] NTSC
[18:26:35] [PASSED] NTSC-443
[18:26:35] [PASSED] NTSC-J
[18:26:35] [PASSED] PAL
[18:26:35] [PASSED] PAL-M
[18:26:35] [PASSED] PAL-N
[18:26:35] [PASSED] SECAM
[18:26:35] ====== [PASSED] drm_test_get_tv_mode_from_name_valid =======
[18:26:35] [PASSED] drm_test_get_tv_mode_from_name_truncated
[18:26:35] ============ [PASSED] drm_get_tv_mode_from_name ============
[18:26:35] ==================== drm_damage_helper  ====================
[18:26:35] [PASSED] drm_test_damage_iter_no_damage
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_fractional_src
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_src_moved
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_fractional_src_moved
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_not_visible
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_no_crtc
[18:26:35] [PASSED] drm_test_damage_iter_no_damage_no_fb
[18:26:35] [PASSED] drm_test_damage_iter_simple_damage
[18:26:35] [PASSED] drm_test_damage_iter_single_damage
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_intersect_src
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_outside_src
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_fractional_src
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_intersect_fractional_src
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_outside_fractional_src
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_src_moved
[18:26:35] [PASSED] drm_test_damage_iter_single_damage_fractional_src_moved
[18:26:35] [PASSED] drm_test_damage_iter_damage
[18:26:35] [PASSED] drm_test_damage_iter_damage_one_intersect
[18:26:35] [PASSED] drm_test_damage_iter_damage_one_outside
[18:26:35] [PASSED] drm_test_damage_iter_damage_src_moved
[18:26:35] [PASSED] drm_test_damage_iter_damage_not_visible
[18:26:35] ================ [PASSED] drm_damage_helper ================
[18:26:35] ==================== drm_dp_mst_helper  ====================
[18:26:35] ============== drm_test_dp_mst_calc_pbn_mode  ==============
[18:26:35] [PASSED] Clock 154000 BPP 30 DSC disabled
[18:26:35] [PASSED] Clock 234000 BPP 30 DSC disabled
[18:26:35] [PASSED] Clock 297000 BPP 24 DSC disabled
[18:26:35] [PASSED] Clock 332880 BPP 24 DSC enabled
[18:26:35] [PASSED] Clock 324540 BPP 24 DSC enabled
[18:26:35] ========== [PASSED] drm_test_dp_mst_calc_pbn_mode ==========
[18:26:35] ========= drm_test_dp_mst_sideband_msg_req_decode  =========
[18:26:35] [PASSED] DP_ENUM_PATH_RESOURCES with port number
[18:26:35] [PASSED] DP_POWER_UP_PHY with port number
[18:26:35] [PASSED] DP_POWER_DOWN_PHY with port number
[18:26:35] [PASSED] DP_ALLOCATE_PAYLOAD with SDP stream sinks
[18:26:35] [PASSED] DP_ALLOCATE_PAYLOAD with port number
[18:26:35] [PASSED] DP_ALLOCATE_PAYLOAD with VCPI
[18:26:35] [PASSED] DP_ALLOCATE_PAYLOAD with PBN
[18:26:35] [PASSED] DP_QUERY_PAYLOAD with port number
[18:26:35] [PASSED] DP_QUERY_PAYLOAD with VCPI
[18:26:35] [PASSED] DP_REMOTE_DPCD_READ with port number
[18:26:35] [PASSED] DP_REMOTE_DPCD_READ with DPCD address
[18:26:35] [PASSED] DP_REMOTE_DPCD_READ with max number of bytes
[18:26:35] [PASSED] DP_REMOTE_DPCD_WRITE with port number
[18:26:35] [PASSED] DP_REMOTE_DPCD_WRITE with DPCD address
[18:26:35] [PASSED] DP_REMOTE_DPCD_WRITE with data array
[18:26:35] [PASSED] DP_REMOTE_I2C_READ with port number
[18:26:35] [PASSED] DP_REMOTE_I2C_READ with I2C device ID
[18:26:35] [PASSED] DP_REMOTE_I2C_READ with transactions array
[18:26:35] [PASSED] DP_REMOTE_I2C_WRITE with port number
[18:26:35] [PASSED] DP_REMOTE_I2C_WRITE with I2C device ID
[18:26:35] [PASSED] DP_REMOTE_I2C_WRITE with data array
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream ID
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with client ID
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream event
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with valid stream event
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream behavior
[18:26:35] [PASSED] DP_QUERY_STREAM_ENC_STATUS with a valid stream behavior
[18:26:35] ===== [PASSED] drm_test_dp_mst_sideband_msg_req_decode =====
[18:26:35] ================ [PASSED] drm_dp_mst_helper ================
[18:26:35] ================= drm_format_helper_test  ==================
[18:26:35] ============== drm_test_fb_xrgb8888_to_gray8  ==============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ========== [PASSED] drm_test_fb_xrgb8888_to_gray8 ==========
[18:26:35] ============= drm_test_fb_xrgb8888_to_rgb332  ==============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb332 ==========
[18:26:35] ============= drm_test_fb_xrgb8888_to_rgb565  ==============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb565 ==========
[18:26:35] ============ drm_test_fb_xrgb8888_to_xrgb1555  =============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======== [PASSED] drm_test_fb_xrgb8888_to_xrgb1555 =========
[18:26:35] ============ drm_test_fb_xrgb8888_to_argb1555  =============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======== [PASSED] drm_test_fb_xrgb8888_to_argb1555 =========
[18:26:35] ============ drm_test_fb_xrgb8888_to_rgba5551  =============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======== [PASSED] drm_test_fb_xrgb8888_to_rgba5551 =========
[18:26:35] ============= drm_test_fb_xrgb8888_to_rgb888  ==============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb888 ==========
[18:26:35] ============ drm_test_fb_xrgb8888_to_argb8888  =============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======== [PASSED] drm_test_fb_xrgb8888_to_argb8888 =========
[18:26:35] =========== drm_test_fb_xrgb8888_to_xrgb2101010  ===========
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======= [PASSED] drm_test_fb_xrgb8888_to_xrgb2101010 =======
[18:26:35] =========== drm_test_fb_xrgb8888_to_argb2101010  ===========
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ======= [PASSED] drm_test_fb_xrgb8888_to_argb2101010 =======
[18:26:35] ============== drm_test_fb_xrgb8888_to_mono  ===============
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ========== [PASSED] drm_test_fb_xrgb8888_to_mono ===========
[18:26:35] ==================== drm_test_fb_swab  =====================
[18:26:35] [PASSED] single_pixel_source_buffer
[18:26:35] [PASSED] single_pixel_clip_rectangle
[18:26:35] [PASSED] well_known_colors
[18:26:35] [PASSED] destination_pitch
[18:26:35] ================ [PASSED] drm_test_fb_swab =================
[18:26:35] ================= drm_test_fb_clip_offset  =================
[18:26:35] [PASSED] pass through
[18:26:35] [PASSED] horizontal offset
[18:26:35] [PASSED] vertical offset
[18:26:35] [PASSED] horizontal and vertical offset
[18:26:35] [PASSED] horizontal offset (custom pitch)
[18:26:35] [PASSED] vertical offset (custom pitch)
[18:26:35] [PASSED] horizontal and vertical offset (custom pitch)
[18:26:35] ============= [PASSED] drm_test_fb_clip_offset =============
[18:26:35] ============== drm_test_fb_build_fourcc_list  ==============
[18:26:35] [PASSED] no native formats
[18:26:35] [PASSED] XRGB8888 as native format
[18:26:35] [PASSED] remove duplicates
[18:26:35] [PASSED] convert alpha formats
[18:26:35] [PASSED] random formats
[18:26:35] ========== [PASSED] drm_test_fb_build_fourcc_list ==========
[18:26:35] =================== drm_test_fb_memcpy  ====================
[18:26:35] [PASSED] single_pixel_source_buffer: XR24 little-endian (0x34325258)
[18:26:35] [PASSED] single_pixel_source_buffer: XRA8 little-endian (0x38415258)
[18:26:35] [PASSED] single_pixel_source_buffer: YU24 little-endian (0x34325559)
[18:26:35] [PASSED] single_pixel_clip_rectangle: XB24 little-endian (0x34324258)
[18:26:35] [PASSED] single_pixel_clip_rectangle: XRA8 little-endian (0x38415258)
[18:26:35] [PASSED] single_pixel_clip_rectangle: YU24 little-endian (0x34325559)
[18:26:35] [PASSED] well_known_colors: XB24 little-endian (0x34324258)
[18:26:35] [PASSED] well_known_colors: XRA8 little-endian (0x38415258)
[18:26:35] [PASSED] well_known_colors: YU24 little-endian (0x34325559)
[18:26:35] [PASSED] destination_pitch: XB24 little-endian (0x34324258)
[18:26:35] [PASSED] destination_pitch: XRA8 little-endian (0x38415258)
[18:26:35] [PASSED] destination_pitch: YU24 little-endian (0x34325559)
[18:26:35] =============== [PASSED] drm_test_fb_memcpy ================
[18:26:35] ============= [PASSED] drm_format_helper_test ==============
[18:26:35] ======================= drm_format  ========================
[18:26:35] [PASSED] drm_test_format_block_width_invalid
[18:26:35] [PASSED] drm_test_format_block_width_one_plane
[18:26:35] [PASSED] drm_test_format_block_width_two_plane
[18:26:35] [PASSED] drm_test_format_block_width_three_plane
[18:26:35] [PASSED] drm_test_format_block_width_tiled
[18:26:35] [PASSED] drm_test_format_block_height_invalid
[18:26:35] [PASSED] drm_test_format_block_height_one_plane
[18:26:35] [PASSED] drm_test_format_block_height_two_plane
[18:26:35] [PASSED] drm_test_format_block_height_three_plane
[18:26:35] [PASSED] drm_test_format_block_height_tiled
[18:26:35] [PASSED] drm_test_format_min_pitch_invalid
[18:26:35] [PASSED] drm_test_format_min_pitch_one_plane_8bpp
[18:26:35] [PASSED] drm_test_format_min_pitch_one_plane_16bpp
[18:26:35] [PASSED] drm_test_format_min_pitch_one_plane_24bpp
[18:26:35] [PASSED] drm_test_format_min_pitch_one_plane_32bpp
[18:26:35] [PASSED] drm_test_format_min_pitch_two_plane
[18:26:35] [PASSED] drm_test_format_min_pitch_three_plane_8bpp
[18:26:35] [PASSED] drm_test_format_min_pitch_tiled
[18:26:35] =================== [PASSED] drm_format ====================
[18:26:35] ===================== drm_framebuffer  =====================
[18:26:35] =============== drm_test_framebuffer_create  ===============
[18:26:35] [PASSED] ABGR8888 normal sizes
[18:26:35] [PASSED] ABGR8888 max sizes
[18:26:35] [PASSED] ABGR8888 pitch greater than min required
[18:26:35] [PASSED] ABGR8888 pitch less than min required
[18:26:35] [PASSED] ABGR8888 Invalid width
[18:26:35] [PASSED] ABGR8888 Invalid buffer handle
[18:26:35] [PASSED] No pixel format
[18:26:35] [PASSED] ABGR8888 Width 0
[18:26:35] [PASSED] ABGR8888 Height 0
[18:26:35] [PASSED] ABGR8888 Out of bound height * pitch combination
[18:26:35] [PASSED] ABGR8888 Large buffer offset
[18:26:35] [PASSED] ABGR8888 Set DRM_MODE_FB_MODIFIERS without modifiers
[18:26:35] [PASSED] ABGR8888 Valid buffer modifier
[18:26:35] [PASSED] ABGR8888 Invalid buffer modifier(DRM_FORMAT_MOD_SAMSUNG_64_32_TILE)
[18:26:35] [PASSED] ABGR8888 Extra pitches without DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] ABGR8888 Extra pitches with DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] NV12 Normal sizes
[18:26:35] [PASSED] NV12 Max sizes
[18:26:35] [PASSED] NV12 Invalid pitch
[18:26:35] [PASSED] NV12 Invalid modifier/missing DRM_MODE_FB_MODIFIERS flag
[18:26:35] [PASSED] NV12 different  modifier per-plane
[18:26:35] [PASSED] NV12 with DRM_FORMAT_MOD_SAMSUNG_64_32_TILE
[18:26:35] [PASSED] NV12 Valid modifiers without DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] NV12 Modifier for inexistent plane
[18:26:35] [PASSED] NV12 Handle for inexistent plane
[18:26:35] [PASSED] NV12 Handle for inexistent plane without DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] YVU420 DRM_MODE_FB_MODIFIERS set without modifier
[18:26:35] [PASSED] YVU420 Normal sizes
[18:26:35] [PASSED] YVU420 Max sizes
[18:26:35] [PASSED] YVU420 Invalid pitch
[18:26:35] [PASSED] YVU420 Different pitches
[18:26:35] [PASSED] YVU420 Different buffer offsets/pitches
[18:26:35] [PASSED] YVU420 Modifier set just for plane 0, without DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] YVU420 Modifier set just for planes 0, 1, without DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] YVU420 Modifier set just for plane 0, 1, with DRM_MODE_FB_MODIFIERS
[18:26:35] [PASSED] YVU420 Valid modifier
[18:26:35] [PASSED] YVU420 Different modifiers per plane
[18:26:35] [PASSED] YVU420 Modifier for inexistent plane
[18:26:35] [PASSED] X0L2 Normal sizes
[18:26:35] [PASSED] X0L2 Max sizes
[18:26:35] [PASSED] X0L2 Invalid pitch
[18:26:35] [PASSED] X0L2 Pitch greater than minimum required
[18:26:35] [PASSED] X0L2 Handle for inexistent plane
[18:26:35] [PASSED] X0L2 Offset for inexistent plane, without DRM_MODE_FB_MODIFIERS set
[18:26:35] [PASSED] X0L2 Modifier without DRM_MODE_FB_MODIFIERS set
[18:26:35] [PASSED] X0L2 Valid modifier
[18:26:35] [PASSED] X0L2 Modifier for inexistent plane
[18:26:35] =========== [PASSED] drm_test_framebuffer_create ===========
[18:26:35] ================= [PASSED] drm_framebuffer =================
[18:26:35] ==================== drm-test-managed  =====================
[18:26:35] [PASSED] drm_test_managed_run_action
[18:26:35] ================ [PASSED] drm-test-managed =================
[18:26:35] ========================= drm_mm  ==========================
[18:26:35] [PASSED] drm_test_mm_init
[18:26:35] [PASSED] drm_test_mm_debug
[18:26:46] [PASSED] drm_test_mm_reserve
[18:26:56] [PASSED] drm_test_mm_insert
[18:26:57] [PASSED] drm_test_mm_replace
[18:26:57] [PASSED] drm_test_mm_insert_range
[18:26:57] [PASSED] drm_test_mm_frag
[18:26:57] [PASSED] drm_test_mm_align
[18:26:57] [PASSED] drm_test_mm_align32
[18:26:58] [PASSED] drm_test_mm_align64
[18:26:58] [PASSED] drm_test_mm_evict
[18:26:58] [PASSED] drm_test_mm_evict_range
[18:26:58] [PASSED] drm_test_mm_topdown
[18:26:58] [PASSED] drm_test_mm_bottomup
[18:26:58] [PASSED] drm_test_mm_lowest
[18:26:58] [PASSED] drm_test_mm_highest
[18:26:59] [PASSED] drm_test_mm_color
[18:26:59] [PASSED] drm_test_mm_color_evict
[18:26:59] [PASSED] drm_test_mm_color_evict_range
[18:26:59] ===================== [PASSED] drm_mm ======================
[18:26:59] =================== drm_modes_analog_tv  ===================
[18:26:59] [PASSED] drm_test_modes_analog_tv_ntsc_480i
[18:26:59] [PASSED] drm_test_modes_analog_tv_ntsc_480i_inlined
[18:26:59] [PASSED] drm_test_modes_analog_tv_pal_576i
[18:26:59] [PASSED] drm_test_modes_analog_tv_pal_576i_inlined
[18:26:59] =============== [PASSED] drm_modes_analog_tv ===============
[18:26:59] ==================== drm_plane_helper  =====================
[18:26:59] =============== drm_test_check_plane_state  ================
[18:26:59] [PASSED] clipping_simple
[18:26:59] [PASSED] clipping_rotate_reflect
[18:26:59] [PASSED] positioning_simple
[18:26:59] [PASSED] upscaling
[18:26:59] [PASSED] downscaling
[18:26:59] [PASSED] rounding1
[18:26:59] [PASSED] rounding2
[18:26:59] [PASSED] rounding3
[18:26:59] [PASSED] rounding4
[18:26:59] =========== [PASSED] drm_test_check_plane_state ============
[18:26:59] =========== drm_test_check_invalid_plane_state  ============
[18:26:59] [PASSED] positioning_invalid
[18:26:59] [PASSED] upscaling_invalid
[18:26:59] [PASSED] downscaling_invalid
[18:26:59] ======= [PASSED] drm_test_check_invalid_plane_state ========
[18:26:59] ================ [PASSED] drm_plane_helper =================
[18:26:59] ============ drm_connector_helper_tv_get_modes  ============
[18:26:59] ====== drm_test_connector_helper_tv_get_modes_check  =======
[18:26:59] [PASSED] None
[18:26:59] [PASSED] PAL
[18:26:59] [PASSED] NTSC
[18:26:59] [PASSED] Both, NTSC Default
[18:26:59] [PASSED] Both, PAL Default
[18:26:59] [PASSED] Both, NTSC Default, with PAL on command-line
[18:26:59] [PASSED] Both, PAL Default, with NTSC on command-line
[18:26:59] == [PASSED] drm_test_connector_helper_tv_get_modes_check ===
[18:26:59] ======== [PASSED] drm_connector_helper_tv_get_modes ========
[18:26:59] ======================== drm_rect  =========================
[18:26:59] [PASSED] drm_test_rect_clip_scaled_div_by_zero
[18:26:59] [PASSED] drm_test_rect_clip_scaled_not_clipped
[18:26:59] [PASSED] drm_test_rect_clip_scaled_clipped
[18:26:59] [PASSED] drm_test_rect_clip_scaled_signed_vs_unsigned
[18:26:59] ================= drm_test_rect_intersect  =================
[18:26:59] [PASSED] top-left x bottom-right: 2x2+1+1 x 2x2+0+0
[18:26:59] [PASSED] top-right x bottom-left: 2x2+0+0 x 2x2+1-1
[18:26:59] [PASSED] bottom-left x top-right: 2x2+1-1 x 2x2+0+0
[18:26:59] [PASSED] bottom-right x top-left: 2x2+0+0 x 2x2+1+1
[18:26:59] [PASSED] right x left: 2x1+0+0 x 3x1+1+0
[18:26:59] [PASSED] left x right: 3x1+1+0 x 2x1+0+0
[18:26:59] [PASSED] up x bottom: 1x2+0+0 x 1x3+0-1
[18:26:59] [PASSED] bottom x up: 1x3+0-1 x 1x2+0+0
[18:26:59] [PASSED] touching corner: 1x1+0+0 x 2x2+1+1
[18:26:59] [PASSED] touching side: 1x1+0+0 x 1x1+1+0
[18:26:59] [PASSED] equal rects: 2x2+0+0 x 2x2+0+0
[18:26:59] [PASSED] inside another: 2x2+0+0 x 1x1+1+1
[18:26:59] [PASSED] far away: 1x1+0+0 x 1x1+3+6
[18:26:59] [PASSED] points intersecting: 0x0+5+10 x 0x0+5+10
[18:26:59] [PASSED] points not intersecting: 0x0+0+0 x 0x0+5+10
[18:26:59] ============= [PASSED] drm_test_rect_intersect =============
[18:26:59] ================ drm_test_rect_calc_hscale  ================
[18:26:59] [PASSED] normal use
[18:26:59] [PASSED] out of max range
[18:26:59] [PASSED] out of min range
[18:26:59] [PASSED] zero dst
[18:26:59] [PASSED] negative src
[18:26:59] [PASSED] negative dst
[18:26:59] ============ [PASSED] drm_test_rect_calc_hscale ============
[18:26:59] ================ drm_test_rect_calc_vscale  ================
[18:26:59] [PASSED] normal use
[18:26:59] [PASSED] out of max range
[18:26:59] [PASSED] out of min range
[18:26:59] [PASSED] zero dst
[18:26:59] [PASSED] negative src
[18:26:59] [PASSED] negative dst
[18:26:59] ============ [PASSED] drm_test_rect_calc_vscale ============
[18:26:59] ================== drm_test_rect_rotate  ===================
[18:26:59] [PASSED] reflect-x
[18:26:59] [PASSED] reflect-y
[18:26:59] [PASSED] rotate-0
[18:26:59] [PASSED] rotate-90
[18:26:59] [PASSED] rotate-180
[18:26:59] [PASSED] rotate-270
[18:26:59] ============== [PASSED] drm_test_rect_rotate ===============
[18:26:59] ================ drm_test_rect_rotate_inv  =================
[18:26:59] [PASSED] reflect-x
[18:26:59] [PASSED] reflect-y
[18:26:59] [PASSED] rotate-0
[18:26:59] [PASSED] rotate-90
[18:26:59] [PASSED] rotate-180
[18:26:59] [PASSED] rotate-270
[18:26:59] ============ [PASSED] drm_test_rect_rotate_inv =============
stty: 'standard input': Inappropriate ioctl for device
[18:26:59] ==================== [PASSED] drm_rect =====================
[18:26:59] ======================== drm_exec  =========================
[18:26:59] [PASSED] sanitycheck
[18:26:59] [PASSED] test_lock
[18:26:59] [PASSED] test_lock_unlock
[18:26:59] [PASSED] test_duplicates
[18:26:59] [PASSED] test_prepare
[18:26:59] [PASSED] test_prepare_array
[18:26:59] [PASSED] test_multiple_loops
[18:26:59] ==================== [PASSED] drm_exec =====================
[18:26:59] ============================================================
[18:26:59] Testing complete. Ran 368 tests: passed: 368
[18:26:59] Elapsed time: 45.230s total, 1.682s configuring, 19.506s building, 24.038s running

+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.Build: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
                   ` (3 preceding siblings ...)
  (?)
@ 2023-11-10 18:34 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:34 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

+ trap cleanup EXIT
+ cd /kernel
+ git clone https://gitlab.freedesktop.org/drm/xe/ci.git .ci
Cloning into '.ci'...
++ date +%s
+ echo -e '\e[0Ksection_start:1699640830:build_x86_64[collapsed=true]\r\e[0KBuild x86-64'
+ mkdir -p build64-default
^[[0Ksection_start:1699640830:build_x86_64[collapsed=true]
^[[0KBuild x86-64
+ cp .ci/kernel/kconfig build64-default/.config
+ make O=build64-default olddefconfig
make[1]: Entering directory '/kernel/build64-default'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
make[1]: Leaving directory '/kernel/build64-default'
++ nproc
+ make O=build64-default -j48
make[1]: Entering directory '/kernel/build64-default'
  GEN     Makefile
  WRAP    arch/x86/include/generated/uapi/asm/bpf_perf_event.h
  WRAP    arch/x86/include/generated/uapi/asm/errno.h
  WRAP    arch/x86/include/generated/uapi/asm/fcntl.h
  WRAP    arch/x86/include/generated/uapi/asm/ioctl.h
  GEN     arch/x86/include/generated/asm/orc_hash.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
  WRAP    arch/x86/include/generated/uapi/asm/ioctls.h
  WRAP    arch/x86/include/generated/uapi/asm/ipcbuf.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
  WRAP    arch/x86/include/generated/uapi/asm/param.h
  WRAP    arch/x86/include/generated/uapi/asm/poll.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
  WRAP    arch/x86/include/generated/uapi/asm/resource.h
  SYSHDR  arch/x86/include/generated/asm/unistd_32_ia32.h
  SYSHDR  arch/x86/include/generated/asm/unistd_64_x32.h
  WRAP    arch/x86/include/generated/uapi/asm/socket.h
  WRAP    arch/x86/include/generated/uapi/asm/sockios.h
  WRAP    arch/x86/include/generated/uapi/asm/termbits.h
  WRAP    arch/x86/include/generated/uapi/asm/termios.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_64.h
  WRAP    arch/x86/include/generated/uapi/asm/types.h
  HOSTCC  arch/x86/tools/relocs_32.o
  HOSTCC  arch/x86/tools/relocs_64.o
  HOSTCC  arch/x86/tools/relocs_common.o
  WRAP    arch/x86/include/generated/asm/early_ioremap.h
  WRAP    arch/x86/include/generated/asm/mcs_spinlock.h
  WRAP    arch/x86/include/generated/asm/export.h
  WRAP    arch/x86/include/generated/asm/irq_regs.h
  WRAP    arch/x86/include/generated/asm/kmap_size.h
  WRAP    arch/x86/include/generated/asm/local64.h
  WRAP    arch/x86/include/generated/asm/mmiowb.h
  WRAP    arch/x86/include/generated/asm/module.lds.h
  WRAP    arch/x86/include/generated/asm/rwonce.h
  WRAP    arch/x86/include/generated/asm/unaligned.h
  UPD     include/generated/uapi/linux/version.h
  UPD     include/config/kernel.release
  UPD     include/generated/compile.h
  HOSTCC  scripts/kallsyms
  HOSTCC  scripts/sorttable
  HOSTCC  scripts/unifdef
  HOSTCC  scripts/asn1_compiler
  UPD     include/generated/utsrelease.h
  DESCEND objtool
  HOSTCC  /kernel/build64-default/tools/objtool/fixdep.o
  HOSTLD  /kernel/build64-default/tools/objtool/fixdep-in.o
  LINK    /kernel/build64-default/tools/objtool/fixdep
  HOSTLD  arch/x86/tools/relocs
  INSTALL /kernel/build64-default/tools/objtool/libsubcmd/include/subcmd/exec-cmd.h
  INSTALL /kernel/build64-default/tools/objtool/libsubcmd/include/subcmd/help.h
  INSTALL /kernel/build64-default/tools/objtool/libsubcmd/include/subcmd/pager.h
  INSTALL /kernel/build64-default/tools/objtool/libsubcmd/include/subcmd/parse-options.h
  INSTALL /kernel/build64-default/tools/objtool/libsubcmd/include/subcmd/run-command.h
  CC      /kernel/build64-default/tools/objtool/libsubcmd/exec-cmd.o
  CC      /kernel/build64-default/tools/objtool/libsubcmd/help.o
  CC      /kernel/build64-default/tools/objtool/libsubcmd/pager.o
  CC      /kernel/build64-default/tools/objtool/libsubcmd/parse-options.o
  CC      /kernel/build64-default/tools/objtool/libsubcmd/run-command.o
  INSTALL libsubcmd_headers
  CC      /kernel/build64-default/tools/objtool/libsubcmd/sigchain.o
  CC      /kernel/build64-default/tools/objtool/libsubcmd/subcmd-config.o
  CC      scripts/mod/empty.o
  HOSTCC  scripts/mod/mk_elfconfig
  CC      scripts/mod/devicetable-offsets.s
  HDRINST usr/include/video/edid.h
  HDRINST usr/include/video/sisfb.h
  HDRINST usr/include/video/uvesafb.h
  HDRINST usr/include/drm/amdgpu_drm.h
  HDRINST usr/include/drm/qaic_accel.h
  HDRINST usr/include/drm/i915_drm.h
  HDRINST usr/include/drm/virtgpu_drm.h
  HDRINST usr/include/drm/vgem_drm.h
  HDRINST usr/include/drm/xe_drm.h
  HDRINST usr/include/drm/omap_drm.h
  HDRINST usr/include/drm/radeon_drm.h
  HDRINST usr/include/drm/drm_mode.h
  HDRINST usr/include/drm/tegra_drm.h
  HDRINST usr/include/drm/exynos_drm.h
  HDRINST usr/include/drm/ivpu_accel.h
  HDRINST usr/include/drm/v3d_drm.h
  HDRINST usr/include/drm/drm_sarea.h
  HDRINST usr/include/drm/qxl_drm.h
  HDRINST usr/include/drm/drm_fourcc.h
  HDRINST usr/include/drm/nouveau_drm.h
  UPD     scripts/mod/devicetable-offsets.h
  HDRINST usr/include/drm/habanalabs_accel.h
  HDRINST usr/include/drm/vmwgfx_drm.h
  HDRINST usr/include/drm/msm_drm.h
  HDRINST usr/include/drm/etnaviv_drm.h
  HDRINST usr/include/drm/vc4_drm.h
  HDRINST usr/include/drm/panfrost_drm.h
  HDRINST usr/include/drm/lima_drm.h
  HDRINST usr/include/drm/drm.h
  HDRINST usr/include/drm/armada_drm.h
  HDRINST usr/include/mtd/nftl-user.h
  HDRINST usr/include/mtd/inftl-user.h
  HDRINST usr/include/mtd/mtd-user.h
  HDRINST usr/include/mtd/ubi-user.h
  HDRINST usr/include/mtd/mtd-abi.h
  HDRINST usr/include/xen/gntdev.h
  HDRINST usr/include/xen/gntalloc.h
  HDRINST usr/include/xen/evtchn.h
  HDRINST usr/include/xen/privcmd.h
  HDRINST usr/include/asm-generic/auxvec.h
  HDRINST usr/include/asm-generic/bitsperlong.h
  HDRINST usr/include/asm-generic/posix_types.h
  HDRINST usr/include/asm-generic/mman.h
  HDRINST usr/include/asm-generic/ioctls.h
  HDRINST usr/include/asm-generic/shmbuf.h
  HDRINST usr/include/asm-generic/bpf_perf_event.h
  HDRINST usr/include/asm-generic/types.h
  HDRINST usr/include/asm-generic/poll.h
  HDRINST usr/include/asm-generic/msgbuf.h
  HDRINST usr/include/asm-generic/swab.h
  HDRINST usr/include/asm-generic/statfs.h
  HDRINST usr/include/asm-generic/unistd.h
  HDRINST usr/include/asm-generic/hugetlb_encode.h
  HDRINST usr/include/asm-generic/resource.h
  HDRINST usr/include/asm-generic/param.h
  MKELF   scripts/mod/elfconfig.h
  HDRINST usr/include/asm-generic/termbits-common.h
  HDRINST usr/include/asm-generic/sockios.h
  HDRINST usr/include/asm-generic/kvm_para.h
  HDRINST usr/include/asm-generic/errno.h
  HDRINST usr/include/asm-generic/termios.h
  HOSTCC  scripts/mod/modpost.o
  HOSTCC  scripts/mod/file2alias.o
  HDRINST usr/include/asm-generic/mman-common.h
  HOSTCC  scripts/mod/sumversion.o
  HDRINST usr/include/asm-generic/ioctl.h
  HDRINST usr/include/asm-generic/socket.h
  HDRINST usr/include/asm-generic/signal-defs.h
  HDRINST usr/include/asm-generic/termbits.h
  HDRINST usr/include/asm-generic/int-ll64.h
  HDRINST usr/include/asm-generic/signal.h
  HDRINST usr/include/asm-generic/siginfo.h
  HDRINST usr/include/asm-generic/stat.h
  HDRINST usr/include/asm-generic/int-l64.h
  HDRINST usr/include/asm-generic/errno-base.h
  HDRINST usr/include/asm-generic/fcntl.h
  HDRINST usr/include/asm-generic/setup.h
  HDRINST usr/include/asm-generic/ipcbuf.h
  HDRINST usr/include/asm-generic/sembuf.h
  HDRINST usr/include/asm-generic/ucontext.h
  HDRINST usr/include/rdma/mlx5_user_ioctl_cmds.h
  HDRINST usr/include/rdma/irdma-abi.h
  HDRINST usr/include/rdma/mana-abi.h
  HDRINST usr/include/rdma/hfi/hfi1_user.h
  HDRINST usr/include/rdma/hfi/hfi1_ioctl.h
  HDRINST usr/include/rdma/rdma_user_rxe.h
  HDRINST usr/include/rdma/rdma_user_ioctl.h
  HDRINST usr/include/rdma/mlx5_user_ioctl_verbs.h
  HDRINST usr/include/rdma/bnxt_re-abi.h
  HDRINST usr/include/rdma/hns-abi.h
  HDRINST usr/include/rdma/qedr-abi.h
  HDRINST usr/include/rdma/ib_user_ioctl_cmds.h
  HDRINST usr/include/rdma/vmw_pvrdma-abi.h
  HDRINST usr/include/rdma/ib_user_sa.h
  HDRINST usr/include/rdma/ib_user_ioctl_verbs.h
  HDRINST usr/include/rdma/rvt-abi.h
  HDRINST usr/include/rdma/mlx5-abi.h
  HDRINST usr/include/rdma/rdma_netlink.h
  HDRINST usr/include/rdma/erdma-abi.h
  HDRINST usr/include/rdma/rdma_user_ioctl_cmds.h
  HDRINST usr/include/rdma/rdma_user_cm.h
  HDRINST usr/include/rdma/ib_user_verbs.h
  HDRINST usr/include/rdma/efa-abi.h
  HDRINST usr/include/rdma/siw-abi.h
  HDRINST usr/include/rdma/mlx4-abi.h
  HDRINST usr/include/rdma/mthca-abi.h
  HDRINST usr/include/rdma/ib_user_mad.h
  HDRINST usr/include/rdma/ocrdma-abi.h
  HDRINST usr/include/rdma/cxgb4-abi.h
  HDRINST usr/include/misc/xilinx_sdfec.h
  HDRINST usr/include/misc/uacce/hisi_qm.h
  HDRINST usr/include/misc/uacce/uacce.h
  HDRINST usr/include/misc/cxl.h
  HDRINST usr/include/misc/ocxl.h
  HDRINST usr/include/misc/fastrpc.h
  HDRINST usr/include/misc/pvpanic.h
  HDRINST usr/include/linux/i8k.h
  HDRINST usr/include/linux/acct.h
  HDRINST usr/include/linux/atmmpc.h
  HDRINST usr/include/linux/fs.h
  HDRINST usr/include/linux/cifs/cifs_mount.h
  HDRINST usr/include/linux/cifs/cifs_netlink.h
  HDRINST usr/include/linux/route.h
  HDRINST usr/include/linux/if_packet.h
  HDRINST usr/include/linux/patchkey.h
  HDRINST usr/include/linux/tc_ematch/tc_em_cmp.h
  HDRINST usr/include/linux/tc_ematch/tc_em_ipt.h
  HDRINST usr/include/linux/tc_ematch/tc_em_meta.h
  HDRINST usr/include/linux/tc_ematch/tc_em_nbyte.h
  HDRINST usr/include/linux/tc_ematch/tc_em_text.h
  HDRINST usr/include/linux/virtio_pmem.h
  HDRINST usr/include/linux/rkisp1-config.h
  HDRINST usr/include/linux/vhost.h
  HDRINST usr/include/linux/cec-funcs.h
  HDRINST usr/include/linux/ppdev.h
  HDRINST usr/include/linux/isdn/capicmd.h
  HDRINST usr/include/linux/virtio_fs.h
  HDRINST usr/include/linux/netfilter_ipv6.h
  HDRINST usr/include/linux/lirc.h
  HDRINST usr/include/linux/mroute6.h
  HDRINST usr/include/linux/nl80211-vnd-intel.h
  HDRINST usr/include/linux/ivtvfb.h
  HDRINST usr/include/linux/auxvec.h
  HDRINST usr/include/linux/dm-log-userspace.h
  HDRINST usr/include/linux/dccp.h
  HDRINST usr/include/linux/virtio_scmi.h
  HDRINST usr/include/linux/atmarp.h
  HDRINST usr/include/linux/arcfb.h
  HDRINST usr/include/linux/nbd-netlink.h
  HDRINST usr/include/linux/sched/types.h
  HDRINST usr/include/linux/tcp.h
  HDRINST usr/include/linux/neighbour.h
  HDRINST usr/include/linux/dlm_device.h
  HDRINST usr/include/linux/wmi.h
  HDRINST usr/include/linux/btrfs_tree.h
  HDRINST usr/include/linux/virtio_crypto.h
  HDRINST usr/include/linux/vbox_err.h
  HDRINST usr/include/linux/edd.h
  HDRINST usr/include/linux/loop.h
  HDRINST usr/include/linux/nvme_ioctl.h
  HDRINST usr/include/linux/mmtimer.h
  HDRINST usr/include/linux/if_pppol2tp.h
  HDRINST usr/include/linux/mtio.h
  HDRINST usr/include/linux/if_arcnet.h
  HDRINST usr/include/linux/romfs_fs.h
  HDRINST usr/include/linux/posix_types.h
  HDRINST usr/include/linux/rtc.h
  HDRINST usr/include/linux/landlock.h
  HDRINST usr/include/linux/gpio.h
  HDRINST usr/include/linux/selinux_netlink.h
  HDRINST usr/include/linux/pps.h
  HDRINST usr/include/linux/ndctl.h
  HDRINST usr/include/linux/virtio_gpu.h
  HDRINST usr/include/linux/android/binderfs.h
  HDRINST usr/include/linux/android/binder.h
  HDRINST usr/include/linux/virtio_vsock.h
  HDRINST usr/include/linux/sound.h
  HDRINST usr/include/linux/vtpm_proxy.h
  HDRINST usr/include/linux/nfs_fs.h
  HDRINST usr/include/linux/elf-fdpic.h
  HDRINST usr/include/linux/adfs_fs.h
  HDRINST usr/include/linux/target_core_user.h
  HDRINST usr/include/linux/netlink_diag.h
  HDRINST usr/include/linux/const.h
  HDRINST usr/include/linux/firewire-cdev.h
  HDRINST usr/include/linux/vdpa.h
  HDRINST usr/include/linux/if_infiniband.h
  HDRINST usr/include/linux/serial.h
  HDRINST usr/include/linux/iio/types.h
  HDRINST usr/include/linux/iio/buffer.h
  HDRINST usr/include/linux/iio/events.h
  HDRINST usr/include/linux/baycom.h
  HDRINST usr/include/linux/major.h
  HDRINST usr/include/linux/atmppp.h
  HDRINST usr/include/linux/ipv6_route.h
  HDRINST usr/include/linux/spi/spidev.h
  HDRINST usr/include/linux/spi/spi.h
  HDRINST usr/include/linux/virtio_ring.h
  HDRINST usr/include/linux/hdlc/ioctl.h
  HDRINST usr/include/linux/remoteproc_cdev.h
  HDRINST usr/include/linux/hyperv.h
  HDRINST usr/include/linux/rpl_iptunnel.h
  HDRINST usr/include/linux/sync_file.h
  HDRINST usr/include/linux/v4l2-dv-timings.h
  HDRINST usr/include/linux/igmp.h
  HDRINST usr/include/linux/virtio_i2c.h
  HDRINST usr/include/linux/xfrm.h
  HDRINST usr/include/linux/capability.h
  HDRINST usr/include/linux/gtp.h
  HDRINST usr/include/linux/xdp_diag.h
  HDRINST usr/include/linux/pkt_cls.h
  HDRINST usr/include/linux/suspend_ioctls.h
  HDRINST usr/include/linux/vt.h
  HDRINST usr/include/linux/loadpin.h
  HDRINST usr/include/linux/dlm_plock.h
  HDRINST usr/include/linux/fb.h
  HDRINST usr/include/linux/max2175.h
  HDRINST usr/include/linux/sunrpc/debug.h
  HDRINST usr/include/linux/gsmmux.h
  HDRINST usr/include/linux/watchdog.h
  HDRINST usr/include/linux/vhost_types.h
  HDRINST usr/include/linux/vduse.h
  HDRINST usr/include/linux/ila.h
  HDRINST usr/include/linux/tdx-guest.h
  HDRINST usr/include/linux/close_range.h
  HDRINST usr/include/linux/ivtv.h
  HDRINST usr/include/linux/cryptouser.h
  HDRINST usr/include/linux/netfilter/xt_string.h
  HDRINST usr/include/linux/netfilter/nfnetlink_compat.h
  HDRINST usr/include/linux/netfilter/nf_nat.h
  HDRINST usr/include/linux/netfilter/xt_recent.h
  HDRINST usr/include/linux/netfilter/xt_addrtype.h
  HDRINST usr/include/linux/netfilter/nf_conntrack_tcp.h
  HDRINST usr/include/linux/netfilter/xt_MARK.h
  HDRINST usr/include/linux/netfilter/xt_SYNPROXY.h
  HDRINST usr/include/linux/netfilter/xt_multiport.h
  HDRINST usr/include/linux/netfilter/nfnetlink.h
  HDRINST usr/include/linux/netfilter/xt_cgroup.h
  HDRINST usr/include/linux/netfilter/nf_synproxy.h
  HDRINST usr/include/linux/netfilter/xt_TCPOPTSTRIP.h
  HDRINST usr/include/linux/netfilter/nfnetlink_log.h
  HDRINST usr/include/linux/netfilter/xt_TPROXY.h
  HDRINST usr/include/linux/netfilter/xt_u32.h
  HDRINST usr/include/linux/netfilter/nfnetlink_osf.h
  HDRINST usr/include/linux/netfilter/xt_ecn.h
  HDRINST usr/include/linux/netfilter/xt_esp.h
  HDRINST usr/include/linux/netfilter/nfnetlink_hook.h
  HDRINST usr/include/linux/netfilter/xt_mac.h
  HDRINST usr/include/linux/netfilter/xt_comment.h
  HDRINST usr/include/linux/netfilter/xt_NFQUEUE.h
  HDRINST usr/include/linux/netfilter/xt_osf.h
  HDRINST usr/include/linux/netfilter/xt_hashlimit.h
  HDRINST usr/include/linux/netfilter/nf_conntrack_sctp.h
  HDRINST usr/include/linux/netfilter/xt_socket.h
  HDRINST usr/include/linux/netfilter/xt_connmark.h
  HDRINST usr/include/linux/netfilter/xt_sctp.h
  HDRINST usr/include/linux/netfilter/xt_tcpudp.h
  HDRINST usr/include/linux/netfilter/xt_DSCP.h
  HDRINST usr/include/linux/netfilter/xt_time.h
  HDRINST usr/include/linux/netfilter/xt_IDLETIMER.h
  HDRINST usr/include/linux/netfilter/xt_policy.h
  HDRINST usr/include/linux/netfilter/xt_rpfilter.h
  HDRINST usr/include/linux/netfilter/xt_nfacct.h
  HDRINST usr/include/linux/netfilter/xt_SECMARK.h
  HDRINST usr/include/linux/netfilter/xt_length.h
  HDRINST usr/include/linux/netfilter/nfnetlink_cthelper.h
  HDRINST usr/include/linux/netfilter/xt_quota.h
  HDRINST usr/include/linux/netfilter/xt_CLASSIFY.h
  HDRINST usr/include/linux/netfilter/xt_ipcomp.h
  HDRINST usr/include/linux/netfilter/xt_iprange.h
  HDRINST usr/include/linux/netfilter/xt_bpf.h
  HDRINST usr/include/linux/netfilter/xt_LOG.h
  HDRINST usr/include/linux/netfilter/xt_rateest.h
  HDRINST usr/include/linux/netfilter/xt_CONNSECMARK.h
  HDRINST usr/include/linux/netfilter/xt_HMARK.h
  HDRINST usr/include/linux/netfilter/xt_CONNMARK.h
  HDRINST usr/include/linux/netfilter/xt_pkttype.h
  HDRINST usr/include/linux/netfilter/xt_ipvs.h
  HDRINST usr/include/linux/netfilter/xt_devgroup.h
  HDRINST usr/include/linux/netfilter/xt_AUDIT.h
  HDRINST usr/include/linux/netfilter/xt_realm.h
  HDRINST usr/include/linux/netfilter/nf_conntrack_common.h
  HDRINST usr/include/linux/netfilter/xt_set.h
  HDRINST usr/include/linux/netfilter/xt_LED.h
  HDRINST usr/include/linux/netfilter/xt_connlabel.h
  HDRINST usr/include/linux/netfilter/xt_owner.h
  HDRINST usr/include/linux/netfilter/xt_dccp.h
  HDRINST usr/include/linux/netfilter/xt_limit.h
  HDRINST usr/include/linux/netfilter/xt_conntrack.h
  HDRINST usr/include/linux/netfilter/xt_TEE.h
  HDRINST usr/include/linux/netfilter/xt_RATEEST.h
  HDRINST usr/include/linux/netfilter/xt_connlimit.h
  HDRINST usr/include/linux/netfilter/ipset/ip_set.h
  HDRINST usr/include/linux/netfilter/ipset/ip_set_list.h
  HDRINST usr/include/linux/netfilter/ipset/ip_set_hash.h
  HDRINST usr/include/linux/netfilter/ipset/ip_set_bitmap.h
  HDRINST usr/include/linux/netfilter/x_tables.h
  HDRINST usr/include/linux/netfilter/xt_dscp.h
  HDRINST usr/include/linux/netfilter/nf_conntrack_ftp.h
  HDRINST usr/include/linux/netfilter/xt_cluster.h
  HDRINST usr/include/linux/netfilter/nf_conntrack_tuple_common.h
  HDRINST usr/include/linux/netfilter/nf_log.h
  HDRINST usr/include/linux/netfilter/xt_tcpmss.h
  HDRINST usr/include/linux/netfilter/xt_NFLOG.h
  HDRINST usr/include/linux/netfilter/xt_l2tp.h
  HDRINST usr/include/linux/netfilter/xt_helper.h
  HDRINST usr/include/linux/netfilter/xt_statistic.h
  HDRINST usr/include/linux/netfilter/nfnetlink_queue.h
  HDRINST usr/include/linux/netfilter/nfnetlink_cttimeout.h
  HDRINST usr/include/linux/netfilter/xt_CT.h
  HDRINST usr/include/linux/netfilter/xt_CHECKSUM.h
  HDRINST usr/include/linux/netfilter/xt_connbytes.h
  HDRINST usr/include/linux/netfilter/xt_state.h
  HDRINST usr/include/linux/netfilter/nf_tables.h
  HDRINST usr/include/linux/netfilter/xt_mark.h
  HDRINST usr/include/linux/netfilter/xt_cpu.h
  HDRINST usr/include/linux/netfilter/nf_tables_compat.h
  HDRINST usr/include/linux/netfilter/xt_physdev.h
  HDRINST usr/include/linux/netfilter/nfnetlink_conntrack.h
  HDRINST usr/include/linux/netfilter/nfnetlink_acct.h
  HDRINST usr/include/linux/netfilter/xt_TCPMSS.h
  HDRINST usr/include/linux/tty_flags.h
  HDRINST usr/include/linux/if_phonet.h
  HDRINST usr/include/linux/elf-em.h
  HDRINST usr/include/linux/vm_sockets.h
  HDRINST usr/include/linux/dlmconstants.h
  HDRINST usr/include/linux/bsg.h
  HDRINST usr/include/linux/matroxfb.h
  HDRINST usr/include/linux/sysctl.h
  HDRINST usr/include/linux/unix_diag.h
  HDRINST usr/include/linux/pcitest.h
  HDRINST usr/include/linux/mman.h
  HDRINST usr/include/linux/if_plip.h
  HDRINST usr/include/linux/virtio_balloon.h
  HDRINST usr/include/linux/pidfd.h
  HDRINST usr/include/linux/f2fs.h
  HDRINST usr/include/linux/x25.h
  HDRINST usr/include/linux/if_cablemodem.h
  HDRINST usr/include/linux/utsname.h
  HDRINST usr/include/linux/counter.h
  HDRINST usr/include/linux/atm_tcp.h
  HDRINST usr/include/linux/atalk.h
  HDRINST usr/include/linux/virtio_rng.h
  HDRINST usr/include/linux/vboxguest.h
  HDRINST usr/include/linux/bpf_perf_event.h
  HDRINST usr/include/linux/ipmi_ssif_bmc.h
  HDRINST usr/include/linux/nfs_mount.h
  HDRINST usr/include/linux/sonet.h
  HDRINST usr/include/linux/netfilter.h
  HDRINST usr/include/linux/keyctl.h
  HDRINST usr/include/linux/nl80211.h
  HDRINST usr/include/linux/misc/bcm_vk.h
  HDRINST usr/include/linux/audit.h
  HDRINST usr/include/linux/tipc_config.h
  HDRINST usr/include/linux/tipc_sockets_diag.h
  HDRINST usr/include/linux/futex.h
  HDRINST usr/include/linux/sev-guest.h
  HDRINST usr/include/linux/ublk_cmd.h
  HDRINST usr/include/linux/types.h
  HDRINST usr/include/linux/virtio_input.h
  HDRINST usr/include/linux/if_slip.h
  HDRINST usr/include/linux/personality.h
  HDRINST usr/include/linux/openat2.h
  HDRINST usr/include/linux/poll.h
  HDRINST usr/include/linux/posix_acl.h
  HDRINST usr/include/linux/smc_diag.h
  HDRINST usr/include/linux/snmp.h
  HDRINST usr/include/linux/errqueue.h
  HDRINST usr/include/linux/if_tunnel.h
  HDRINST usr/include/linux/fanotify.h
  HDRINST usr/include/linux/kernel.h
  LD      /kernel/build64-default/tools/objtool/libsubcmd/libsubcmd-in.o
  HDRINST usr/include/linux/rtnetlink.h
  HDRINST usr/include/linux/rpl.h
  HDRINST usr/include/linux/memfd.h
  HDRINST usr/include/linux/serial_core.h
  HDRINST usr/include/linux/dns_resolver.h
  HDRINST usr/include/linux/pr.h
  HDRINST usr/include/linux/atm_eni.h
  HDRINST usr/include/linux/lp.h
  HDRINST usr/include/linux/virtio_mem.h
  HDRINST usr/include/linux/ultrasound.h
  HDRINST usr/include/linux/sctp.h
  HDRINST usr/include/linux/uio.h
  HDRINST usr/include/linux/tcp_metrics.h
  HDRINST usr/include/linux/wwan.h
  HDRINST usr/include/linux/atmbr2684.h
  HDRINST usr/include/linux/in_route.h
  HDRINST usr/include/linux/qemu_fw_cfg.h
  HDRINST usr/include/linux/if_macsec.h
  HDRINST usr/include/linux/usb/charger.h
  HDRINST usr/include/linux/usb/g_uvc.h
  HDRINST usr/include/linux/usb/gadgetfs.h
  HDRINST usr/include/linux/usb/raw_gadget.h
  HDRINST usr/include/linux/usb/cdc-wdm.h
  HDRINST usr/include/linux/usb/g_printer.h
  HDRINST usr/include/linux/usb/midi.h
  HDRINST usr/include/linux/usb/tmc.h
  HDRINST usr/include/linux/usb/video.h
  HDRINST usr/include/linux/usb/functionfs.h
  AR      /kernel/build64-default/tools/objtool/libsubcmd/libsubcmd.a
  HDRINST usr/include/linux/usb/ch11.h
  HDRINST usr/include/linux/usb/audio.h
  HDRINST usr/include/linux/usb/ch9.h
  HDRINST usr/include/linux/usb/cdc.h
  HDRINST usr/include/linux/jffs2.h
  HDRINST usr/include/linux/ax25.h
  HDRINST usr/include/linux/auto_fs.h
  HDRINST usr/include/linux/tiocl.h
  HDRINST usr/include/linux/scc.h
  HDRINST usr/include/linux/psci.h
  HDRINST usr/include/linux/swab.h
  HDRINST usr/include/linux/cec.h
  HDRINST usr/include/linux/kfd_ioctl.h
  HDRINST usr/include/linux/smc.h
  HDRINST usr/include/linux/qrtr.h
  HDRINST usr/include/linux/screen_info.h
  HDRINST usr/include/linux/nfsacl.h
  HDRINST usr/include/linux/seg6_hmac.h
  HDRINST usr/include/linux/gameport.h
  HDRINST usr/include/linux/wireless.h
  HDRINST usr/include/linux/fdreg.h
  HDRINST usr/include/linux/cciss_defs.h
  HDRINST usr/include/linux/serial_reg.h
  HDRINST usr/include/linux/perf_event.h
  HDRINST usr/include/linux/in6.h
  HDRINST usr/include/linux/hid.h
  HDRINST usr/include/linux/netlink.h
  HDRINST usr/include/linux/fuse.h
  HDRINST usr/include/linux/magic.h
  HDRINST usr/include/linux/ioam6_iptunnel.h
  HDRINST usr/include/linux/stm.h
  HDRINST usr/include/linux/vsockmon.h
  HDRINST usr/include/linux/seg6.h
  HDRINST usr/include/linux/idxd.h
  HDRINST usr/include/linux/nitro_enclaves.h
  HDRINST usr/include/linux/ptrace.h
  HDRINST usr/include/linux/ioam6_genl.h
  HDRINST usr/include/linux/fsl_mc.h
  HDRINST usr/include/linux/qnx4_fs.h
  HDRINST usr/include/linux/net_tstamp.h
  HDRINST usr/include/linux/msg.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_TTL.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_ttl.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_ah.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_ECN.h
  HDRINST usr/include/linux/netfilter_ipv4/ip_tables.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_ecn.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_CLUSTERIP.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_REJECT.h
  HDRINST usr/include/linux/netfilter_ipv4/ipt_LOG.h
  HDRINST usr/include/linux/sem.h
  HDRINST usr/include/linux/net_namespace.h
  HDRINST usr/include/linux/radeonfb.h
  HDRINST usr/include/linux/tee.h
  HDRINST usr/include/linux/udp.h
  HDRINST usr/include/linux/virtio_bt.h
  HDRINST usr/include/linux/v4l2-subdev.h
  HDRINST usr/include/linux/posix_acl_xattr.h
  HDRINST usr/include/linux/v4l2-mediabus.h
  HDRINST usr/include/linux/atmapi.h
  CC      /kernel/build64-default/tools/objtool/weak.o
  HDRINST usr/include/linux/raid/md_p.h
  CC      /kernel/build64-default/tools/objtool/check.o
  HDRINST usr/include/linux/raid/md_u.h
  CC      /kernel/build64-default/tools/objtool/special.o
  HDRINST usr/include/linux/zorro_ids.h
  HDRINST usr/include/linux/nbd.h
  CC      /kernel/build64-default/tools/objtool/builtin-check.o
  MKDIR   /kernel/build64-default/tools/objtool/arch/x86/
  HDRINST usr/include/linux/isst_if.h
  HDRINST usr/include/linux/rxrpc.h
  CC      /kernel/build64-default/tools/objtool/elf.o
  HDRINST usr/include/linux/unistd.h
  HDRINST usr/include/linux/if_arp.h
  CC      /kernel/build64-default/tools/objtool/objtool.o
  MKDIR   /kernel/build64-default/tools/objtool/arch/x86/lib/
  CC      /kernel/build64-default/tools/objtool/orc_gen.o
  HDRINST usr/include/linux/atm_zatm.h
  CC      /kernel/build64-default/tools/objtool/orc_dump.o
  CC      /kernel/build64-default/tools/objtool/arch/x86/special.o
  CC      /kernel/build64-default/tools/objtool/libstring.o
  GEN     /kernel/build64-default/tools/objtool/arch/x86/lib/inat-tables.c
  HDRINST usr/include/linux/io_uring.h
  CC      /kernel/build64-default/tools/objtool/libctype.o
  HDRINST usr/include/linux/if_fddi.h
  HDRINST usr/include/linux/bpqether.h
  HDRINST usr/include/linux/sysinfo.h
  CC      /kernel/build64-default/tools/objtool/str_error_r.o
  HDRINST usr/include/linux/auto_dev-ioctl.h
  HDRINST usr/include/linux/nfs4_mount.h
  HDRINST usr/include/linux/keyboard.h
  CC      /kernel/build64-default/tools/objtool/librbtree.o
  HDRINST usr/include/linux/virtio_mmio.h
  HDRINST usr/include/linux/input.h
  HDRINST usr/include/linux/qnxtypes.h
  HDRINST usr/include/linux/mdio.h
  HDRINST usr/include/linux/lwtunnel.h
  HDRINST usr/include/linux/gfs2_ondisk.h
  HDRINST usr/include/linux/eventfd.h
  HDRINST usr/include/linux/nfs4.h
  HDRINST usr/include/linux/ptp_clock.h
  HDRINST usr/include/linux/nubus.h
  HDRINST usr/include/linux/if_bonding.h
  HDRINST usr/include/linux/kcov.h
  HDRINST usr/include/linux/fadvise.h
  HDRINST usr/include/linux/taskstats.h
  HDRINST usr/include/linux/veth.h
  HDRINST usr/include/linux/atm.h
  HDRINST usr/include/linux/ipmi.h
  HDRINST usr/include/linux/kdev_t.h
  HDRINST usr/include/linux/mount.h
  HDRINST usr/include/linux/shm.h
  HDRINST usr/include/linux/resource.h
  HDRINST usr/include/linux/prctl.h
  HDRINST usr/include/linux/watch_queue.h
  HDRINST usr/include/linux/sched.h
  HDRINST usr/include/linux/phonet.h
  HDRINST usr/include/linux/random.h
  HDRINST usr/include/linux/tty.h
  HDRINST usr/include/linux/apm_bios.h
  HDRINST usr/include/linux/fd.h
  HDRINST usr/include/linux/um_timetravel.h
  HDRINST usr/include/linux/tls.h
  HDRINST usr/include/linux/rpmsg_types.h
  HDRINST usr/include/linux/pfrut.h
  HDRINST usr/include/linux/mei.h
  HDRINST usr/include/linux/fsi.h
  HDRINST usr/include/linux/if_x25.h
  HDRINST usr/include/linux/rds.h
  HDRINST usr/include/linux/param.h
  HDRINST usr/include/linux/netdevice.h
  HDRINST usr/include/linux/binfmts.h
  HDRINST usr/include/linux/if_pppox.h
  HDRINST usr/include/linux/sockios.h
  HDRINST usr/include/linux/kcm.h
  HDRINST usr/include/linux/virtio_9p.h
  HDRINST usr/include/linux/genwqe/genwqe_card.h
  HDRINST usr/include/linux/if_tun.h
  HDRINST usr/include/linux/ext4.h
  HDRINST usr/include/linux/if_ether.h
  HDRINST usr/include/linux/kvm_para.h
  HDRINST usr/include/linux/kernel-page-flags.h
  CC      /kernel/build64-default/tools/objtool/arch/x86/decode.o
  HDRINST usr/include/linux/cdrom.h
  HDRINST usr/include/linux/un.h
  HDRINST usr/include/linux/module.h
  HDRINST usr/include/linux/mqueue.h
  HDRINST usr/include/linux/a.out.h
  HDRINST usr/include/linux/input-event-codes.h
  HDRINST usr/include/linux/coda.h
  HDRINST usr/include/linux/rio_mport_cdev.h
  HDRINST usr/include/linux/ipsec.h
  HDRINST usr/include/linux/blkpg.h
  HDRINST usr/include/linux/blkzoned.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_arpreply.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_redirect.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_nflog.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_802_3.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_nat.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_mark_m.h
  HDRINST usr/include/linux/netfilter_bridge/ebtables.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_vlan.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_limit.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_log.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_stp.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_pkttype.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_ip.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_ip6.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_arp.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_mark_t.h
  HDRINST usr/include/linux/netfilter_bridge/ebt_among.h
  HDRINST usr/include/linux/reiserfs_fs.h
  HDRINST usr/include/linux/cciss_ioctl.h
  HDRINST usr/include/linux/fsmap.h
  HDRINST usr/include/linux/smiapp.h
  HDRINST usr/include/linux/switchtec_ioctl.h
  HDRINST usr/include/linux/atmdev.h
  HDRINST usr/include/linux/hpet.h
  HDRINST usr/include/linux/virtio_config.h
  HDRINST usr/include/linux/string.h
  HDRINST usr/include/linux/kfd_sysfs.h
  HDRINST usr/include/linux/inet_diag.h
  HDRINST usr/include/linux/netdev.h
  HDRINST usr/include/linux/xattr.h
  HDRINST usr/include/linux/iommufd.h
  HDRINST usr/include/linux/user_events.h
  HDRINST usr/include/linux/errno.h
  HDRINST usr/include/linux/icmp.h
  HDRINST usr/include/linux/i2o-dev.h
  HDRINST usr/include/linux/pg.h
  HDRINST usr/include/linux/if_bridge.h
  HDRINST usr/include/linux/thermal.h
  HDRINST usr/include/linux/uinput.h
  HDRINST usr/include/linux/handshake.h
  HDRINST usr/include/linux/dqblk_xfs.h
  HDRINST usr/include/linux/v4l2-common.h
  HDRINST usr/include/linux/nvram.h
  HDRINST usr/include/linux/if_vlan.h
  HDRINST usr/include/linux/uhid.h
  HDRINST usr/include/linux/omap3isp.h
  HDRINST usr/include/linux/rose.h
  HDRINST usr/include/linux/phantom.h
  HDRINST usr/include/linux/ipmi_msgdefs.h
  HDRINST usr/include/linux/bcm933xx_hcs.h
  HDRINST usr/include/linux/bpf.h
  HDRINST usr/include/linux/mempolicy.h
  HDRINST usr/include/linux/efs_fs_sb.h
  HDRINST usr/include/linux/nexthop.h
  HDRINST usr/include/linux/net_dropmon.h
  HDRINST usr/include/linux/surface_aggregator/cdev.h
  HDRINST usr/include/linux/surface_aggregator/dtx.h
  HDRINST usr/include/linux/net.h
  HDRINST usr/include/linux/mii.h
  HDRINST usr/include/linux/virtio_pcidev.h
  HDRINST usr/include/linux/termios.h
  HDRINST usr/include/linux/cgroupstats.h
  HDRINST usr/include/linux/mpls.h
  HDRINST usr/include/linux/iommu.h
  HDRINST usr/include/linux/toshiba.h
  HDRINST usr/include/linux/virtio_scsi.h
  HDRINST usr/include/linux/zorro.h
  HDRINST usr/include/linux/chio.h
  HDRINST usr/include/linux/pkt_sched.h
  HDRINST usr/include/linux/cramfs_fs.h
  HDRINST usr/include/linux/nfs3.h
  HDRINST usr/include/linux/vfio_ccw.h
  HDRINST usr/include/linux/atm_nicstar.h
  HDRINST usr/include/linux/ncsi.h
  HDRINST usr/include/linux/virtio_net.h
  HDRINST usr/include/linux/ioctl.h
  HDRINST usr/include/linux/stddef.h
  HDRINST usr/include/linux/limits.h
  HDRINST usr/include/linux/ipmi_bmc.h
  HDRINST usr/include/linux/netfilter_arp.h
  HDRINST usr/include/linux/if_addr.h
  HDRINST usr/include/linux/rpmsg.h
  HDRINST usr/include/linux/media-bus-format.h
  HDRINST usr/include/linux/kernelcapi.h
  HDRINST usr/include/linux/ppp_defs.h
  HDRINST usr/include/linux/ethtool.h
  HDRINST usr/include/linux/aspeed-video.h
  HDRINST usr/include/linux/hdlc.h
  HDRINST usr/include/linux/fscrypt.h
  HDRINST usr/include/linux/batadv_packet.h
  HDRINST usr/include/linux/uuid.h
  HDRINST usr/include/linux/capi.h
  HDRINST usr/include/linux/mptcp.h
  HDRINST usr/include/linux/hidraw.h
  HDRINST usr/include/linux/virtio_console.h
  HDRINST usr/include/linux/irqnr.h
  HDRINST usr/include/linux/coresight-stm.h
  HDRINST usr/include/linux/cxl_mem.h
  HDRINST usr/include/linux/iso_fs.h
  HDRINST usr/include/linux/virtio_blk.h
  HDRINST usr/include/linux/udf_fs_i.h
  HDRINST usr/include/linux/coff.h
  HDRINST usr/include/linux/dma-buf.h
  HDRINST usr/include/linux/ife.h
  HDRINST usr/include/linux/agpgart.h
  HDRINST usr/include/linux/socket.h
  HDRINST usr/include/linux/nilfs2_ondisk.h
  HDRINST usr/include/linux/connector.h
  HDRINST usr/include/linux/auto_fs4.h
  HDRINST usr/include/linux/bt-bmc.h
  HDRINST usr/include/linux/map_to_7segment.h
  HDRINST usr/include/linux/tc_act/tc_skbedit.h
  HDRINST usr/include/linux/tc_act/tc_ctinfo.h
  HDRINST usr/include/linux/tc_act/tc_defact.h
  HDRINST usr/include/linux/tc_act/tc_gact.h
  HDRINST usr/include/linux/tc_act/tc_vlan.h
  HDRINST usr/include/linux/tc_act/tc_skbmod.h
  HDRINST usr/include/linux/tc_act/tc_sample.h
  HDRINST usr/include/linux/tc_act/tc_tunnel_key.h
  HDRINST usr/include/linux/tc_act/tc_gate.h
  HDRINST usr/include/linux/tc_act/tc_mirred.h
  HDRINST usr/include/linux/tc_act/tc_nat.h
  HDRINST usr/include/linux/tc_act/tc_csum.h
  HDRINST usr/include/linux/tc_act/tc_connmark.h
  HDRINST usr/include/linux/tc_act/tc_ife.h
  HDRINST usr/include/linux/tc_act/tc_mpls.h
  HDRINST usr/include/linux/tc_act/tc_ct.h
  HDRINST usr/include/linux/tc_act/tc_pedit.h
  HDRINST usr/include/linux/tc_act/tc_bpf.h
  HDRINST usr/include/linux/tc_act/tc_ipt.h
  HDRINST usr/include/linux/netrom.h
  HDRINST usr/include/linux/joystick.h
  HDRINST usr/include/linux/falloc.h
  HDRINST usr/include/linux/cycx_cfm.h
  HDRINST usr/include/linux/omapfb.h
  HDRINST usr/include/linux/msdos_fs.h
  HDRINST usr/include/linux/virtio_types.h
  HDRINST usr/include/linux/mroute.h
  HDRINST usr/include/linux/psample.h
  HDRINST usr/include/linux/ipv6.h
  HDRINST usr/include/linux/dw100.h
  HDRINST usr/include/linux/psp-sev.h
  HDRINST usr/include/linux/vfio.h
  HDRINST usr/include/linux/if_ppp.h
  HDRINST usr/include/linux/byteorder/big_endian.h
  HDRINST usr/include/linux/byteorder/little_endian.h
  HDRINST usr/include/linux/comedi.h
  HDRINST usr/include/linux/scif_ioctl.h
  HDRINST usr/include/linux/timerfd.h
  HDRINST usr/include/linux/time_types.h
  HDRINST usr/include/linux/firewire-constants.h
  HDRINST usr/include/linux/virtio_snd.h
  HDRINST usr/include/linux/ppp-ioctl.h
  HDRINST usr/include/linux/fib_rules.h
  HDRINST usr/include/linux/gen_stats.h
  HDRINST usr/include/linux/virtio_iommu.h
  HDRINST usr/include/linux/genetlink.h
  HDRINST usr/include/linux/uvcvideo.h
  HDRINST usr/include/linux/pfkeyv2.h
  HDRINST usr/include/linux/soundcard.h
  HDRINST usr/include/linux/times.h
  HDRINST usr/include/linux/nfc.h
  HDRINST usr/include/linux/affs_hardblocks.h
  HDRINST usr/include/linux/nilfs2_api.h
  HDRINST usr/include/linux/rseq.h
  HDRINST usr/include/linux/caif/caif_socket.h
  HDRINST usr/include/linux/caif/if_caif.h
  HDRINST usr/include/linux/i2c-dev.h
  HDRINST usr/include/linux/cuda.h
  HDRINST usr/include/linux/mei_uuid.h
  HDRINST usr/include/linux/cn_proc.h
  HDRINST usr/include/linux/parport.h
  HDRINST usr/include/linux/v4l2-controls.h
  HDRINST usr/include/linux/hsi/cs-protocol.h
  HDRINST usr/include/linux/hsi/hsi_char.h
  HDRINST usr/include/linux/seg6_genl.h
  HDRINST usr/include/linux/am437x-vpfe.h
  HDRINST usr/include/linux/amt.h
  HDRINST usr/include/linux/netconf.h
  HDRINST usr/include/linux/erspan.h
  HDRINST usr/include/linux/nsfs.h
  HDRINST usr/include/linux/xilinx-v4l2-controls.h
  HDRINST usr/include/linux/aspeed-p2a-ctrl.h
  HDRINST usr/include/linux/vfio_zdev.h
  HDRINST usr/include/linux/serio.h
  HDRINST usr/include/linux/acrn.h
  HDRINST usr/include/linux/nfs2.h
  HDRINST usr/include/linux/virtio_pci.h
  HDRINST usr/include/linux/ipc.h
  HDRINST usr/include/linux/kd.h
  HDRINST usr/include/linux/ethtool_netlink.h
  HDRINST usr/include/linux/elf.h
  HDRINST usr/include/linux/videodev2.h
  HDRINST usr/include/linux/if_alg.h
  HDRINST usr/include/linux/sonypi.h
  HDRINST usr/include/linux/fsverity.h
  HDRINST usr/include/linux/if.h
  HDRINST usr/include/linux/btrfs.h
  HDRINST usr/include/linux/vm_sockets_diag.h
  HDRINST usr/include/linux/netfilter_bridge.h
  HDRINST usr/include/linux/packet_diag.h
  HDRINST usr/include/linux/netfilter_ipv4.h
  HDRINST usr/include/linux/kvm.h
  HDRINST usr/include/linux/pci.h
  HDRINST usr/include/linux/if_addrlabel.h
  HDRINST usr/include/linux/hdlcdrv.h
  HDRINST usr/include/linux/cfm_bridge.h
  HDRINST usr/include/linux/fiemap.h
  HDRINST usr/include/linux/dm-ioctl.h
  HDRINST usr/include/linux/aspeed-lpc-ctrl.h
  HDRINST usr/include/linux/atmioc.h
  HDRINST usr/include/linux/dlm.h
  HDRINST usr/include/linux/pci_regs.h
  HDRINST usr/include/linux/cachefiles.h
  HDRINST usr/include/linux/membarrier.h
  HDRINST usr/include/linux/nfs_idmap.h
  HDRINST usr/include/linux/ip.h
  HDRINST usr/include/linux/atm_he.h
  HDRINST usr/include/linux/nfsd/export.h
  HDRINST usr/include/linux/nfsd/stats.h
  HDRINST usr/include/linux/nfsd/debug.h
  HDRINST usr/include/linux/nfsd/cld.h
  HDRINST usr/include/linux/ip_vs.h
  HDRINST usr/include/linux/vmcore.h
  HDRINST usr/include/linux/vbox_vmmdev_types.h
  HDRINST usr/include/linux/dvb/osd.h
  HDRINST usr/include/linux/dvb/dmx.h
  HDRINST usr/include/linux/dvb/net.h
  HDRINST usr/include/linux/dvb/frontend.h
  HDRINST usr/include/linux/dvb/ca.h
  HDRINST usr/include/linux/dvb/version.h
  HDRINST usr/include/linux/dvb/video.h
  HDRINST usr/include/linux/dvb/audio.h
  HDRINST usr/include/linux/nfs.h
  HDRINST usr/include/linux/if_link.h
  HDRINST usr/include/linux/wait.h
  HDRINST usr/include/linux/icmpv6.h
  HDRINST usr/include/linux/media.h
  HDRINST usr/include/linux/seg6_local.h
  HDRINST usr/include/linux/tps6594_pfsm.h
  HDRINST usr/include/linux/openvswitch.h
  HDRINST usr/include/linux/atmsap.h
  HDRINST usr/include/linux/bpfilter.h
  HDRINST usr/include/linux/fpga-dfl.h
  HDRINST usr/include/linux/userio.h
  HDRINST usr/include/linux/signal.h
  HDRINST usr/include/linux/map_to_14segment.h
  HDRINST usr/include/linux/hdreg.h
  HDRINST usr/include/linux/utime.h
  HDRINST usr/include/linux/usbdevice_fs.h
  HDRINST usr/include/linux/timex.h
  HDRINST usr/include/linux/if_fc.h
  HDRINST usr/include/linux/reiserfs_xattr.h
  HDRINST usr/include/linux/hw_breakpoint.h
  HDRINST usr/include/linux/quota.h
  HDRINST usr/include/linux/ioprio.h
  HDRINST usr/include/linux/eventpoll.h
  HDRINST usr/include/linux/atmclip.h
  HDRINST usr/include/linux/can.h
  HDRINST usr/include/linux/if_team.h
  HDRINST usr/include/linux/usbip.h
  HDRINST usr/include/linux/stat.h
  HDRINST usr/include/linux/fou.h
  HDRINST usr/include/linux/hash_info.h
  HDRINST usr/include/linux/ppp-comp.h
  HDRINST usr/include/linux/ip6_tunnel.h
  HDRINST usr/include/linux/tipc_netlink.h
  HDRINST usr/include/linux/in.h
  HDRINST usr/include/linux/wireguard.h
  HDRINST usr/include/linux/btf.h
  HDRINST usr/include/linux/batman_adv.h
  HDRINST usr/include/linux/fcntl.h
  HDRINST usr/include/linux/if_ltalk.h
  HDRINST usr/include/linux/i2c.h
  HDRINST usr/include/linux/atm_idt77105.h
  HDRINST usr/include/linux/kexec.h
  HDRINST usr/include/linux/arm_sdei.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6_tables.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_ah.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_NPT.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_rt.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_REJECT.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_opts.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_srh.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_LOG.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_mh.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_HL.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_hl.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_frag.h
  HDRINST usr/include/linux/netfilter_ipv6/ip6t_ipv6header.h
  HDRINST usr/include/linux/minix_fs.h
  HDRINST usr/include/linux/aio_abi.h
  HDRINST usr/include/linux/pktcdvd.h
  HDRINST usr/include/linux/libc-compat.h
  HDRINST usr/include/linux/atmlec.h
  HDRINST usr/include/linux/signalfd.h
  HDRINST usr/include/linux/bpf_common.h
  HDRINST usr/include/linux/seg6_iptunnel.h
  HDRINST usr/include/linux/synclink.h
  HDRINST usr/include/linux/mpls_iptunnel.h
  HDRINST usr/include/linux/mctp.h
  HDRINST usr/include/linux/if_xdp.h
  HDRINST usr/include/linux/llc.h
  HDRINST usr/include/linux/atmsvc.h
  HDRINST usr/include/linux/sed-opal.h
  HDRINST usr/include/linux/sock_diag.h
  HDRINST usr/include/linux/time.h
  HDRINST usr/include/linux/securebits.h
  HDRINST usr/include/linux/fsl_hypervisor.h
  HDRINST usr/include/linux/if_hippi.h
  HDRINST usr/include/linux/seccomp.h
  HDRINST usr/include/linux/oom.h
  HDRINST usr/include/linux/filter.h
  HDRINST usr/include/linux/inotify.h
  HDRINST usr/include/linux/rfkill.h
  HDRINST usr/include/linux/reboot.h
  HDRINST usr/include/linux/can/vxcan.h
  HDRINST usr/include/linux/can/j1939.h
  HDRINST usr/include/linux/can/netlink.h
  HDRINST usr/include/linux/can/bcm.h
  HDRINST usr/include/linux/can/raw.h
  HDRINST usr/include/linux/can/gw.h
  HDRINST usr/include/linux/can/error.h
  HDRINST usr/include/linux/can/isotp.h
  HDRINST usr/include/linux/if_eql.h
  HDRINST usr/include/linux/psp-dbc.h
  HDRINST usr/include/linux/hiddev.h
  HDRINST usr/include/linux/blktrace_api.h
  HDRINST usr/include/linux/ccs.h
  HDRINST usr/include/linux/ioam6.h
  HDRINST usr/include/linux/hsr_netlink.h
  HDRINST usr/include/linux/mmc/ioctl.h
  HDRINST usr/include/linux/bfs_fs.h
  HDRINST usr/include/linux/rio_cm_cdev.h
  HDRINST usr/include/linux/uleds.h
  HDRINST usr/include/linux/mrp_bridge.h
  HDRINST usr/include/linux/adb.h
  HDRINST usr/include/linux/pmu.h
  HDRINST usr/include/linux/udmabuf.h
  HDRINST usr/include/linux/kcmp.h
  HDRINST usr/include/linux/dma-heap.h
  HDRINST usr/include/linux/userfaultfd.h
  HDRINST usr/include/linux/netfilter_arp/arpt_mangle.h
  HDRINST usr/include/linux/netfilter_arp/arp_tables.h
  HDRINST usr/include/linux/tipc.h
  HDRINST usr/include/linux/virtio_ids.h
  HDRINST usr/include/linux/l2tp.h
  HDRINST usr/include/linux/devlink.h
  HDRINST usr/include/linux/virtio_gpio.h
  HDRINST usr/include/linux/dcbnl.h
  HDRINST usr/include/linux/cyclades.h
  HDRINST usr/include/sound/intel/avs/tokens.h
  HDRINST usr/include/sound/sof/fw.h
  HDRINST usr/include/sound/sof/abi.h
  HDRINST usr/include/sound/sof/tokens.h
  HDRINST usr/include/sound/sof/header.h
  HDRINST usr/include/sound/usb_stream.h
  HDRINST usr/include/sound/sfnt_info.h
  HDRINST usr/include/sound/asequencer.h
  HDRINST usr/include/sound/tlv.h
  HDRINST usr/include/sound/asound.h
  HDRINST usr/include/sound/asoc.h
  HDRINST usr/include/sound/sb16_csp.h
  HDRINST usr/include/sound/compress_offload.h
  HDRINST usr/include/sound/hdsp.h
  HDRINST usr/include/sound/emu10k1.h
  HDRINST usr/include/sound/snd_ar_tokens.h
  HDRINST usr/include/sound/snd_sst_tokens.h
  HDRINST usr/include/sound/asound_fm.h
  HDRINST usr/include/sound/hdspm.h
  HDRINST usr/include/sound/compress_params.h
  HDRINST usr/include/sound/firewire.h
  HDRINST usr/include/sound/skl-tplg-interface.h
  HDRINST usr/include/scsi/scsi_bsg_ufs.h
  HDRINST usr/include/scsi/scsi_netlink_fc.h
  HDRINST usr/include/scsi/scsi_bsg_mpi3mr.h
  HDRINST usr/include/scsi/fc/fc_ns.h
  HDRINST usr/include/scsi/fc/fc_fs.h
  HDRINST usr/include/scsi/fc/fc_els.h
  HDRINST usr/include/scsi/fc/fc_gs.h
  HDRINST usr/include/scsi/scsi_bsg_fc.h
  HDRINST usr/include/scsi/cxlflash_ioctl.h
  HDRINST usr/include/scsi/scsi_netlink.h
  HDRINST usr/include/linux/version.h
  HDRINST usr/include/asm/processor-flags.h
  HDRINST usr/include/asm/auxvec.h
  HDRINST usr/include/asm/svm.h
  HDRINST usr/include/asm/bitsperlong.h
  HDRINST usr/include/asm/kvm_perf.h
  HDRINST usr/include/asm/mce.h
  HDRINST usr/include/asm/posix_types.h
  HDRINST usr/include/asm/msr.h
  HDRINST usr/include/asm/sigcontext32.h
  HDRINST usr/include/asm/mman.h
  HDRINST usr/include/asm/shmbuf.h
  HDRINST usr/include/asm/e820.h
  HDRINST usr/include/asm/posix_types_64.h
  HDRINST usr/include/asm/msgbuf.h
  HDRINST usr/include/asm/vsyscall.h
  HDRINST usr/include/asm/swab.h
  HDRINST usr/include/asm/statfs.h
  HDRINST usr/include/asm/posix_types_x32.h
  HDRINST usr/include/asm/ptrace.h
  HDRINST usr/include/asm/unistd.h
  HDRINST usr/include/asm/ist.h
  HDRINST usr/include/asm/prctl.h
  HDRINST usr/include/asm/boot.h
  HDRINST usr/include/asm/sigcontext.h
  HDRINST usr/include/asm/kvm_para.h
  HDRINST usr/include/asm/posix_types_32.h
  HDRINST usr/include/asm/a.out.h
  HDRINST usr/include/asm/mtrr.h
  HDRINST usr/include/asm/hwcap2.h
  HDRINST usr/include/asm/amd_hsmp.h
  HDRINST usr/include/asm/ptrace-abi.h
  HDRINST usr/include/asm/vm86.h
  HDRINST usr/include/asm/vmx.h
  HDRINST usr/include/asm/ldt.h
  HDRINST usr/include/asm/perf_regs.h
  HDRINST usr/include/asm/kvm.h
  HDRINST usr/include/asm/debugreg.h
  HDRINST usr/include/asm/signal.h
  HDRINST usr/include/asm/bootparam.h
  HDRINST usr/include/asm/siginfo.h
  HDRINST usr/include/asm/hw_breakpoint.h
  HDRINST usr/include/asm/stat.h
  HDRINST usr/include/asm/setup.h
  HDRINST usr/include/asm/sembuf.h
  HDRINST usr/include/asm/sgx.h
  HDRINST usr/include/asm/ucontext.h
  HDRINST usr/include/asm/byteorder.h
  HDRINST usr/include/asm/unistd_64.h
  HDRINST usr/include/asm/ioctls.h
  HDRINST usr/include/asm/bpf_perf_event.h
  HDRINST usr/include/asm/types.h
  HDRINST usr/include/asm/poll.h
  HDRINST usr/include/asm/resource.h
  HDRINST usr/include/asm/param.h
  HDRINST usr/include/asm/sockios.h
  HDRINST usr/include/asm/errno.h
  HDRINST usr/include/asm/unistd_x32.h
  HDRINST usr/include/asm/termios.h
  HDRINST usr/include/asm/ioctl.h
  HDRINST usr/include/asm/socket.h
  HDRINST usr/include/asm/unistd_32.h
  HDRINST usr/include/asm/termbits.h
  HDRINST usr/include/asm/fcntl.h
  HDRINST usr/include/asm/ipcbuf.h
  HOSTLD  scripts/mod/modpost
  CC      kernel/bounds.s
  CHKSHA1 ../include/linux/atomic/atomic-arch-fallback.h
  CHKSHA1 ../include/linux/atomic/atomic-instrumented.h
  CHKSHA1 ../include/linux/atomic/atomic-long.h
  UPD     include/generated/timeconst.h
  UPD     include/generated/bounds.h
  CC      arch/x86/kernel/asm-offsets.s
  LD      /kernel/build64-default/tools/objtool/arch/x86/objtool-in.o
  UPD     include/generated/asm-offsets.h
  CALL    ../scripts/checksyscalls.sh
  LD      /kernel/build64-default/tools/objtool/objtool-in.o
  LINK    /kernel/build64-default/tools/objtool/objtool
  LDS     scripts/module.lds
  CC      security/commoncap.o
  CC      security/min_addr.o
  HOSTCC  usr/gen_init_cpio
  CC      security/inode.o
  CC      security/device_cgroup.o
  AR      certs/built-in.a
  CC      init/main.o
  AR      arch/x86/video/built-in.a
  CC      arch/x86/power/cpu.o
  CC      io_uring/io_uring.o
  AS      arch/x86/lib/clear_page_64.o
  CC      ipc/compat.o
  CC [M]  arch/x86/video/fbdev.o
  CC      block/bdev.o
  CC      io_uring/xattr.o
  CC      arch/x86/power/hibernate_64.o
  CC      init/do_mounts.o
  AS      arch/x86/power/hibernate_asm_64.o
  UPD     init/utsversion-tmp.h
  CC      arch/x86/lib/cmdline.o
  AR      virt/lib/built-in.a
  CC      arch/x86/realmode/init.o
  AR      drivers/cache/built-in.a
  AS      arch/x86/lib/cmpxchg16b_emu.o
  CC      security/keys/gc.o
  CC      arch/x86/pci/i386.o
  CC      block/partitions/core.o
  CC      mm/filemap.o
  AR      drivers/irqchip/built-in.a
  CC      net/llc/llc_core.o
  CC      net/core/sock.o
  CC      fs/iomap/trace.o
  AS      arch/x86/crypto/aesni-intel_asm.o
  CC      net/llc/llc_input.o
  CC      arch/x86/events/amd/core.o
  CC      arch/x86/events/intel/core.o
  CC      fs/nfs_common/grace.o
  AR      sound/i2c/other/built-in.a
  AR      drivers/bus/mhi/built-in.a
  AR      sound/isa/ad1816a/built-in.a
  CC      sound/core/seq/seq.o
  CC      fs/notify/dnotify/dnotify.o
  CC      arch/x86/mm/pat/set_memory.o
  CC      fs/notify/inotify/inotify_fsnotify.o
  AR      sound/drivers/opl3/built-in.a
  CC [M]  virt/lib/irqbypass.o
  CC      block/partitions/ldm.o
  AR      sound/i2c/built-in.a
  AR      drivers/bus/built-in.a
  CC      arch/x86/kernel/fpu/init.o
  AR      sound/pci/ac97/built-in.a
  AR      sound/isa/ad1848/built-in.a
  CC      arch/x86/lib/copy_mc.o
  CC      lib/kunit/hooks.o
  AR      sound/drivers/opl4/built-in.a
  CC      arch/x86/events/zhaoxin/core.o
  CC      arch/x86/entry/vdso/vma.o
  AR      sound/drivers/mpu401/built-in.a
  AR      sound/pci/ali5451/built-in.a
  AR      sound/drivers/vx/built-in.a
  AR      sound/isa/cs423x/built-in.a
  AR      drivers/phy/allwinner/built-in.a
  CC      block/fops.o
  CC      kernel/sched/core.o
  AR      sound/pci/asihpi/built-in.a
  AR      sound/isa/es1688/built-in.a
  CC      mm/kasan/common.o
  CC      arch/x86/kernel/cpu/mce/core.o
  CC      arch/x86/crypto/aesni-intel_glue.o
  AR      sound/drivers/pcsp/built-in.a
  AR      drivers/phy/amlogic/built-in.a
  AR      sound/drivers/built-in.a
  AR      sound/pci/au88x0/built-in.a
  AR      sound/isa/galaxy/built-in.a
  CC      crypto/api.o
  AR      drivers/phy/broadcom/built-in.a
  AR      sound/pci/aw2/built-in.a
  AS      arch/x86/crypto/aesni-intel_avx-x86_64.o
  AR      sound/isa/gus/built-in.a
  AR      drivers/phy/cadence/built-in.a
  AR      sound/pci/ctxfi/built-in.a
  AR      drivers/phy/freescale/built-in.a
  AR      sound/isa/msnd/built-in.a
  AR      sound/pci/ca0106/built-in.a
  CC      arch/x86/kernel/acpi/boot.o
  AR      drivers/phy/hisilicon/built-in.a
  AR      sound/isa/opti9xx/built-in.a
  AR      sound/pci/cs46xx/built-in.a
  AR      sound/isa/sb/built-in.a
  AR      drivers/phy/ingenic/built-in.a
  AR      sound/pci/cs5535audio/built-in.a
  AR      sound/isa/wavefront/built-in.a
  AR      drivers/phy/intel/built-in.a
  AR      sound/pci/lola/built-in.a
  AR      sound/isa/wss/built-in.a
  AR      drivers/phy/lantiq/built-in.a
  AR      sound/isa/built-in.a
  AR      sound/pci/lx6464es/built-in.a
  AR      drivers/phy/marvell/built-in.a
  AR      drivers/phy/mediatek/built-in.a
  AR      sound/pci/echoaudio/built-in.a
  CC      ipc/util.o
  AR      drivers/phy/microchip/built-in.a
  AR      sound/pci/emu10k1/built-in.a
  AR      drivers/phy/motorola/built-in.a
  CC [M]  lib/kunit/test.o
  AR      sound/pci/hda/built-in.a
  AR      drivers/phy/mscc/built-in.a
  CC [M]  sound/pci/hda/hda_bind.o
  AR      drivers/phy/qualcomm/built-in.a
  GEN     usr/initramfs_data.cpio
  AR      drivers/phy/ralink/built-in.a
  COPY    usr/initramfs_inc_data
  AS      usr/initramfs_data.o
  AR      drivers/phy/realtek/built-in.a
  AR      usr/built-in.a
  AR      drivers/phy/renesas/built-in.a
  AS      arch/x86/realmode/rm/header.o
  AR      drivers/phy/rockchip/built-in.a
  AR      drivers/phy/samsung/built-in.a
  AS      arch/x86/realmode/rm/trampoline_64.o
  AS      arch/x86/lib/copy_mc_64.o
  AR      drivers/phy/socionext/built-in.a
  AS      arch/x86/lib/copy_page_64.o
  AR      drivers/phy/st/built-in.a
  CC      mm/kasan/report.o
  AS      arch/x86/realmode/rm/stack.o
  AS      arch/x86/lib/copy_user_64.o
  CC      arch/x86/kernel/fpu/bugs.o
  AR      drivers/phy/starfive/built-in.a
  AS      arch/x86/realmode/rm/reboot.o
  AR      drivers/phy/sunplus/built-in.a
  AS      arch/x86/lib/copy_user_uncached_64.o
  AR      drivers/phy/tegra/built-in.a
  AS      arch/x86/realmode/rm/wakeup_asm.o
  CC      arch/x86/lib/cpu.o
  AR      drivers/phy/ti/built-in.a
  AR      drivers/phy/xilinx/built-in.a
  CC      arch/x86/realmode/rm/wakemain.o
  AR      virt/built-in.a
  CC      drivers/phy/phy-core.o
  CC      arch/x86/events/core.o
  CC      net/llc/llc_output.o
  CC      crypto/cipher.o
  CC      fs/notify/inotify/inotify_user.o
  CC      sound/core/seq/seq_lock.o
  CC      arch/x86/realmode/rm/video-mode.o
  CC      arch/x86/kernel/fpu/core.o
  CC      block/partitions/msdos.o
  CC      arch/x86/kernel/cpu/mtrr/mtrr.o
  CC      security/keys/key.o
  AS      arch/x86/realmode/rm/copy.o
  AR      fs/notify/dnotify/built-in.a
  CC      arch/x86/events/intel/bts.o
  AS      arch/x86/realmode/rm/bioscall.o
  CC      arch/x86/kernel/cpu/mtrr/if.o
  CC      arch/x86/realmode/rm/regs.o
  CC      arch/x86/pci/init.o
  AR      fs/nfs_common/built-in.a
  CC      arch/x86/kernel/cpu/mtrr/generic.o
  AR      sound/pci/ice1712/built-in.a
  CC      arch/x86/realmode/rm/video-vga.o
  CC      arch/x86/realmode/rm/video-vesa.o
  CC      lib/math/div64.o
  CC      fs/notify/fanotify/fanotify.o
  CC      lib/math/gcd.o
  CC      arch/x86/lib/delay.o
  AR      arch/x86/events/zhaoxin/built-in.a
  CC      arch/x86/power/hibernate.o
  CC      fs/notify/fanotify/fanotify_user.o
  AS      arch/x86/lib/getuser.o
  CC      io_uring/nop.o
  CC      arch/x86/entry/vdso/extable.o
  CC      arch/x86/realmode/rm/video-bios.o
  CC      lib/math/lcm.o
  CC      arch/x86/events/intel/ds.o
  CC      arch/x86/events/intel/knc.o
  CC      arch/x86/kernel/cpu/microcode/core.o
  CC      lib/math/int_log.o
  CC      sound/core/seq/seq_clientmgr.o
  CC      arch/x86/mm/pat/memtype.o
  PASYMS  arch/x86/realmode/rm/pasyms.h
  LDS     arch/x86/realmode/rm/realmode.lds
  CC [M]  lib/kunit/resource.o
  LD      arch/x86/realmode/rm/realmode.elf
  RELOCS  arch/x86/realmode/rm/realmode.relocs
  OBJCOPY arch/x86/realmode/rm/realmode.bin
  AS      arch/x86/realmode/rmpiggy.o
  CC [M]  sound/pci/hda/hda_codec.o
  CC      lib/math/int_pow.o
  AR      arch/x86/realmode/built-in.a
  GEN     arch/x86/lib/inat-tables.c
  CC      fs/iomap/iter.o
  AS      arch/x86/crypto/aes_ctrby8_avx-x86_64.o
  CC      lib/math/int_sqrt.o
  CC      block/bio.o
  CC      arch/x86/events/amd/lbr.o
  CC      kernel/sched/fair.o
  CC      lib/math/reciprocal_div.o
  CC      net/core/request_sock.o
  CC      arch/x86/lib/insn-eval.o
  CC      crypto/compress.o
  CC      arch/x86/events/probe.o
  AS [M]  arch/x86/crypto/ghash-clmulni-intel_asm.o
  AR      arch/x86/ia32/built-in.a
  CC [M]  arch/x86/crypto/ghash-clmulni-intel_glue.o
  CC      arch/x86/kernel/cpu/mtrr/cleanup.o
  CC      ipc/msgutil.o
  CC      lib/math/rational.o
  AR      net/llc/built-in.a
  CC      mm/kasan/init.o
  AR      arch/x86/platform/atom/built-in.a
  CC      mm/kasan/generic.o
  CC      arch/x86/kernel/acpi/sleep.o
  AR      arch/x86/platform/ce4100/built-in.a
  CC      arch/x86/platform/efi/memmap.o
  CC      arch/x86/pci/mmconfig_64.o
  AR      arch/x86/platform/geode/built-in.a
  AR      arch/x86/platform/iris/built-in.a
  CC      crypto/algapi.o
  CC      block/partitions/efi.o
  CC      net/ethernet/eth.o
  CC      io_uring/fs.o
  AR      drivers/phy/built-in.a
  CC      init/do_mounts_initrd.o
  AR      drivers/pinctrl/actions/built-in.a
  AR      drivers/pinctrl/bcm/built-in.a
  CC [M]  lib/kunit/static_stub.o
  CC      arch/x86/entry/vdso/vdso32-setup.o
  LDS     arch/x86/entry/vdso/vdso.lds
  AR      drivers/pinctrl/cirrus/built-in.a
  CC      kernel/sched/build_policy.o
  AR      drivers/pinctrl/freescale/built-in.a
  AR      fs/notify/inotify/built-in.a
  CC      crypto/scatterwalk.o
  CC [M]  lib/math/prime_numbers.o
  CC      fs/iomap/buffered-io.o
  CC      drivers/pinctrl/intel/pinctrl-baytrail.o
  CC      fs/iomap/direct-io.o
  CC      security/keys/keyring.o
  AR      arch/x86/power/built-in.a
  CC      drivers/pinctrl/intel/pinctrl-intel.o
  CC      security/keys/keyctl.o
  CC      arch/x86/kernel/cpu/microcode/intel.o
  CC [M]  drivers/pinctrl/intel/pinctrl-cherryview.o
  CC      arch/x86/kernel/cpu/mce/severity.o
  CC      fs/iomap/fiemap.o
  CC      arch/x86/kernel/cpu/mce/genpool.o
  AS [M]  arch/x86/crypto/crc32-pclmul_asm.o
  CC [M]  arch/x86/crypto/crc32-pclmul_glue.o
  AS      arch/x86/entry/vdso/vdso-note.o
  CC      crypto/proc.o
  CC      arch/x86/entry/vdso/vclock_gettime.o
  CC      ipc/msg.o
  CC      arch/x86/mm/pat/memtype_interval.o
  CC      crypto/aead.o
  CC      arch/x86/kernel/fpu/regset.o
  CC      arch/x86/events/amd/ibs.o
  CC      arch/x86/lib/insn.o
  AS      arch/x86/kernel/acpi/wakeup_64.o
  CC      arch/x86/pci/direct.o
  CC [M]  lib/kunit/string-stream.o
  CC      arch/x86/events/amd/uncore.o
  CC      arch/x86/kernel/acpi/apei.o
  CC      arch/x86/platform/efi/quirks.o
  AR      lib/math/built-in.a
  AR      arch/x86/kernel/cpu/mtrr/built-in.a
  CC      lib/crypto/memneq.o
  CC      arch/x86/kernel/cpu/cacheinfo.o
  CC      crypto/geniv.o
  CC      init/initramfs.o
  CC      init/calibrate.o
  CC [M]  drivers/pinctrl/intel/pinctrl-broxton.o
  AS [M]  arch/x86/crypto/crct10dif-pcl-asm_64.o
  CC      mm/kasan/report_generic.o
  CC [M]  arch/x86/crypto/crct10dif-pclmul_glue.o
  AR      fs/notify/fanotify/built-in.a
  CC      fs/notify/fsnotify.o
  CC      lib/zlib_inflate/inffast.o
  CC      arch/x86/entry/vdso/vgetcpu.o
  AR      block/partitions/built-in.a
  CC      arch/x86/pci/mmconfig-shared.o
  CC      arch/x86/pci/fixup.o
  AS      arch/x86/lib/memcpy_64.o
  CC      arch/x86/entry/vsyscall/vsyscall_64.o
  CC      sound/core/seq/seq_memory.o
  HOSTCC  arch/x86/entry/vdso/vdso2c
  AS      arch/x86/lib/memmove_64.o
  CC      arch/x86/kernel/cpu/mce/intel.o
  AS      arch/x86/lib/memset_64.o
  AR      arch/x86/mm/pat/built-in.a
  CC      fs/iomap/seek.o
  CC      arch/x86/mm/init.o
  CC      arch/x86/lib/misc.o
  CC      lib/zlib_inflate/inflate.o
  CC      arch/x86/lib/pc-conf-reg.o
  CC      arch/x86/kernel/cpu/microcode/amd.o
  AS      arch/x86/lib/putuser.o
  CC [M]  lib/kunit/assert.o
  CC      fs/notify/notification.o
  CC [M]  lib/kunit/try-catch.o
  CC [M]  lib/kunit/executor.o
  CC      arch/x86/kernel/fpu/signal.o
  CC      lib/zlib_inflate/infutil.o
  CC      init/init_task.o
  CC      lib/crypto/utils.o
  CC      kernel/sched/build_utility.o
  CC      arch/x86/kernel/acpi/cppc.o
  AR      net/ethernet/built-in.a
  AR      sound/pci/korg1212/built-in.a
  CC      net/802/p8022.o
  AS      arch/x86/lib/retpoline.o
  LDS     arch/x86/entry/vdso/vdso32/vdso32.lds
  CC      arch/x86/mm/init_64.o
  CC      arch/x86/lib/usercopy.o
  CC      arch/x86/events/intel/lbr.o
  AS      arch/x86/entry/vdso/vdso32/note.o
  CC      net/802/psnap.o
  AS      arch/x86/entry/vdso/vdso32/system_call.o
  CC      ipc/sem.o
  LD [M]  arch/x86/crypto/ghash-clmulni-intel.o
  CC      block/elevator.o
  LD [M]  arch/x86/crypto/crc32-pclmul.o
  AS      arch/x86/entry/vdso/vdso32/sigreturn.o
  LD [M]  arch/x86/crypto/crct10dif-pclmul.o
  CC      net/802/stp.o
  AR      arch/x86/crypto/built-in.a
  CC      arch/x86/entry/vdso/vdso32/vclock_gettime.o
  CC      security/keys/permission.o
  CC      arch/x86/platform/intel/iosf_mbi.o
  CC [M]  drivers/pinctrl/intel/pinctrl-geminilake.o
  CC      crypto/skcipher.o
  CC      arch/x86/kernel/cpu/scattered.o
  CC      arch/x86/platform/efi/efi.o
  CC      lib/crypto/chacha.o
  CC      fs/iomap/swapfile.o
  CC      arch/x86/platform/efi/efi_64.o
  AR      arch/x86/events/amd/built-in.a
  CC      mm/kasan/shadow.o
  CC      arch/x86/kernel/cpu/mce/threshold.o
  CC      arch/x86/entry/vdso/vdso32/vgetcpu.o
  CC      arch/x86/kernel/cpu/mce/apei.o
  CC      arch/x86/lib/usercopy_64.o
  AS      arch/x86/entry/vsyscall/vsyscall_emu_64.o
  CC      kernel/locking/mutex.o
  CC      kernel/power/qos.o
  CC      block/blk-core.o
  CC      kernel/printk/printk.o
  CC [M]  lib/kunit/attributes.o
  CC      lib/crypto/aes.o
  CC      kernel/power/main.o
  CC [M]  sound/pci/hda/hda_jack.o
  CC      arch/x86/kernel/acpi/cstate.o
  CC      fs/notify/group.o
  CC      lib/zlib_inflate/inftrees.o
  CC      ipc/shm.o
  CC      sound/core/seq/seq_queue.o
  AR      arch/x86/entry/vsyscall/built-in.a
  CC      kernel/power/console.o
  AS      arch/x86/platform/efi/efi_stub_64.o
  CC      init/version.o
  CC      arch/x86/pci/acpi.o
  CC      arch/x86/kernel/cpu/topology.o
  AR      arch/x86/kernel/cpu/microcode/built-in.a
  CC      arch/x86/kernel/cpu/common.o
  CC      kernel/locking/semaphore.o
  CC [M]  drivers/pinctrl/intel/pinctrl-sunrisepoint.o
  CC      lib/zlib_inflate/inflate_syms.o
  AS      arch/x86/entry/entry.o
  CC      security/keys/process_keys.o
  CC      sound/core/sound.o
  CC      arch/x86/mm/fault.o
  CC      arch/x86/kernel/fpu/xstate.o
  VDSO    arch/x86/entry/vdso/vdso64.so.dbg
  CC      net/core/skbuff.o
  VDSO    arch/x86/entry/vdso/vdso32.so.dbg
  OBJCOPY arch/x86/entry/vdso/vdso64.so
  OBJCOPY arch/x86/entry/vdso/vdso32.so
  VDSO2C  arch/x86/entry/vdso/vdso-image-64.c
  VDSO2C  arch/x86/entry/vdso/vdso-image-32.c
  AR      init/built-in.a
  CC      arch/x86/entry/vdso/vdso-image-64.o
  CC [M]  sound/pci/hda/hda_auto_parser.o
  CC      net/core/datagram.o
  AR      arch/x86/platform/intel/built-in.a
  CC [M]  sound/pci/hda/hda_sysfs.o
  CC      sound/core/init.o
  AS      arch/x86/entry/entry_64.o
  AR      net/802/built-in.a
  CC      arch/x86/entry/vdso/vdso-image-32.o
  CC      arch/x86/entry/syscall_64.o
  CC      arch/x86/lib/msr-smp.o
  CC      arch/x86/entry/common.o
  CC      mm/kasan/quarantine.o
  AR      lib/kunit/built-in.a
  LD [M]  lib/kunit/kunit.o
  CC      net/core/stream.o
  CC      arch/x86/lib/cache-smp.o
  AR      lib/zlib_inflate/built-in.a
  AR      fs/iomap/built-in.a
  CC      kernel/locking/rwsem.o
  CC      security/keys/request_key.o
  CC      lib/zlib_deflate/deflate.o
  AR      arch/x86/kernel/acpi/built-in.a
  CC      arch/x86/kernel/apic/apic.o
  CC      lib/crypto/gf128mul.o
  CC      block/blk-sysfs.o
  AR      arch/x86/kernel/cpu/mce/built-in.a
  CC      arch/x86/kernel/apic/apic_common.o
  CC      lib/zlib_deflate/deftree.o
  AR      arch/x86/entry/vdso/built-in.a
  AR      drivers/pinctrl/intel/built-in.a
  CC      arch/x86/kernel/apic/apic_noop.o
  CC      arch/x86/kernel/apic/ipi.o
  AR      drivers/pinctrl/mediatek/built-in.a
  CC      fs/notify/mark.o
  AR      drivers/pinctrl/mvebu/built-in.a
  AR      arch/x86/platform/efi/built-in.a
  AR      arch/x86/platform/intel-mid/built-in.a
  AR      drivers/pinctrl/nomadik/built-in.a
  AR      arch/x86/platform/intel-quark/built-in.a
  AR      drivers/pinctrl/nuvoton/built-in.a
  CC      arch/x86/events/intel/p4.o
  CC      arch/x86/lib/msr.o
  AR      drivers/pinctrl/nxp/built-in.a
  AR      arch/x86/platform/olpc/built-in.a
  CC      lib/zlib_deflate/deflate_syms.o
  AR      drivers/pinctrl/qcom/built-in.a
  AR      arch/x86/platform/scx200/built-in.a
  AR      drivers/pinctrl/sprd/built-in.a
  AR      arch/x86/platform/ts5500/built-in.a
  CC      crypto/seqiv.o
  AR      drivers/pinctrl/sunplus/built-in.a
  AR      arch/x86/platform/uv/built-in.a
  CC      kernel/locking/percpu-rwsem.o
  AR      arch/x86/platform/built-in.a
  CC      sound/core/seq/seq_fifo.o
  AR      drivers/pinctrl/ti/built-in.a
  CC      arch/x86/pci/legacy.o
  CC      sound/core/seq/seq_prioq.o
  CC      kernel/power/process.o
  CC      drivers/pinctrl/core.o
  CC      fs/notify/fdinfo.o
  CC [M]  sound/pci/hda/hda_controller.o
  CC      sound/core/memory.o
  CC      lib/crypto/blake2s.o
  CC      net/sched/sch_generic.o
  CC      arch/x86/kernel/apic/vector.o
  CC      net/sched/sch_mq.o
  CC      kernel/printk/printk_safe.o
  CC      security/keys/request_key_auth.o
  AS      arch/x86/entry/thunk_64.o
  CC      security/keys/user_defined.o
  CC      lib/crypto/blake2s-generic.o
  AS      arch/x86/entry/entry_64_compat.o
  AR      mm/kasan/built-in.a
  CC      arch/x86/entry/syscall_32.o
  CC      lib/crypto/blake2s-selftest.o
  CC      mm/mempool.o
  CC      kernel/locking/irqflag-debug.o
  CC      net/sched/sch_frag.o
  AR      sound/ppc/built-in.a
  AR      sound/arm/built-in.a
  AR      sound/sh/built-in.a
  AR      sound/synth/emux/built-in.a
  AR      sound/synth/built-in.a
  CC      io_uring/splice.o
  CC      arch/x86/pci/irq.o
  CC      crypto/echainiv.o
  AR      sound/usb/misc/built-in.a
  AR      sound/usb/usx2y/built-in.a
  CC      io_uring/sync.o
  CC      crypto/ahash.o
  AR      sound/usb/caiaq/built-in.a
  AR      sound/usb/6fire/built-in.a
  CC [M]  sound/pci/hda/hda_proc.o
  AR      lib/zlib_deflate/built-in.a
  CC      ipc/syscall.o
  AR      sound/usb/hiface/built-in.a
  AR      sound/usb/bcd2000/built-in.a
  CC      sound/core/control.o
  CC      sound/core/seq/seq_timer.o
  AR      sound/firewire/built-in.a
  CC      sound/core/seq/seq_system.o
  AR      sound/usb/built-in.a
  AR      fs/quota/built-in.a
  AR      arch/x86/kernel/fpu/built-in.a
  CC      mm/oom_kill.o
  CC      fs/kernfs/mount.o
  CC      drivers/pinctrl/pinctrl-utils.o
  AR      sound/sparc/built-in.a
  CC      fs/kernfs/inode.o
  AS      arch/x86/lib/msr-reg.o
  CC      arch/x86/kernel/cpu/rdrand.o
  CC      arch/x86/kernel/kprobes/core.o
  CC      arch/x86/lib/msr-reg-export.o
  CC      arch/x86/mm/ioremap.o
  CC      fs/kernfs/dir.o
  CC      fs/proc/task_mmu.o
  CC      fs/kernfs/file.o
  CC      kernel/locking/mutex-debug.o
  CC      lib/crypto/des.o
  CC      arch/x86/events/intel/p6.o
  AS      arch/x86/lib/hweight.o
  AR      fs/notify/built-in.a
  CC      arch/x86/lib/iomem.o
  CC      arch/x86/kernel/kprobes/opt.o
  CC      arch/x86/kernel/cpu/match.o
  CC      arch/x86/pci/common.o
  CC      net/core/scm.o
  CC      net/core/gen_stats.o
  CC      kernel/locking/lockdep.o
  CC      kernel/power/suspend.o
  CC      security/keys/compat.o
  CC      kernel/printk/printk_ringbuffer.o
  AR      arch/x86/entry/built-in.a
  AR      sound/pci/mixart/built-in.a
  AR      arch/x86/net/built-in.a
  CC      lib/crypto/sha1.o
  AR      sound/pci/nm256/built-in.a
  CC      arch/x86/kernel/cpu/bugs.o
  CC      arch/x86/mm/extable.o
  CC      sound/core/seq/seq_ports.o
  CC      drivers/gpio/gpiolib.o
  AS      arch/x86/lib/iomap_copy_64.o
  AR      drivers/pwm/built-in.a
  CC      block/blk-flush.o
  CC      net/sched/sch_api.o
  CC      arch/x86/lib/inat.o
  CC      mm/fadvise.o
  CC      drivers/pinctrl/pinmux.o
  CC      ipc/ipc_sysctl.o
  CC      drivers/gpio/gpiolib-devres.o
  CC      sound/core/seq/seq_info.o
  CC      io_uring/advise.o
  CC      fs/sysfs/file.o
  CC      io_uring/filetable.o
  AR      arch/x86/lib/built-in.a
  CC      arch/x86/kernel/apic/init.o
  AR      arch/x86/lib/lib.a
  CC      io_uring/openclose.o
  CC      fs/sysfs/dir.o
  CC      fs/kernfs/symlink.o
  CC      crypto/shash.o
  CC      arch/x86/events/intel/pt.o
  AR      sound/pci/oxygen/built-in.a
  CC      lib/crypto/sha256.o
  CC      net/sched/sch_blackhole.o
  CC      drivers/pci/msi/pcidev_msi.o
  CC      fs/configfs/inode.o
  CC [M]  sound/pci/hda/hda_hwdep.o
  CC      fs/devpts/inode.o
  CC      kernel/printk/sysctl.o
  CC      security/keys/proc.o
  CC      arch/x86/mm/mmap.o
  CC      arch/x86/kernel/kprobes/ftrace.o
  CC      arch/x86/pci/early.o
  CC      arch/x86/mm/pgtable.o
  CC      security/keys/sysctl.o
  CC [M]  lib/crypto/arc4.o
  CC      net/core/gen_estimator.o
  CC      arch/x86/kernel/apic/hw_nmi.o
  CC      ipc/mqueue.o
  CC      drivers/gpio/gpiolib-legacy.o
  CC      arch/x86/mm/physaddr.o
  AR      kernel/printk/built-in.a
  CC      arch/x86/pci/bus_numa.o
  CC      net/sched/sch_fifo.o
  CC      arch/x86/mm/tlb.o
  CC      kernel/irq/irqdesc.o
  AR      sound/core/seq/built-in.a
  CC      kernel/irq/handle.o
  CC      sound/core/misc.o
  AR      fs/kernfs/built-in.a
  CC      arch/x86/pci/amd_bus.o
  CC      arch/x86/mm/cpu_entry_area.o
  CC      block/blk-settings.o
  CC      net/core/net_namespace.o
  CC      fs/sysfs/symlink.o
  CC      drivers/pinctrl/pinconf.o
  CC      kernel/power/hibernate.o
  CC      arch/x86/events/utils.o
  CC      drivers/gpio/gpiolib-cdev.o
  CC      drivers/pci/msi/api.o
  AR      lib/crypto/built-in.a
  LD [M]  lib/crypto/libarc4.o
  CC      lib/lzo/lzo1x_compress.o
  CC      drivers/pci/msi/msi.o
  CC      kernel/rcu/update.o
  CC      drivers/pci/msi/irqdomain.o
  CC      drivers/gpio/gpiolib-sysfs.o
  CC      drivers/gpio/gpiolib-acpi.o
  CC [M]  sound/pci/hda/hda_generic.o
  CC      fs/configfs/file.o
  AR      arch/x86/kernel/kprobes/built-in.a
  AR      security/keys/built-in.a
  CC      sound/core/device.o
  AR      security/built-in.a
  CC      drivers/pci/pcie/portdrv.o
  CC      drivers/pci/hotplug/pci_hotplug_core.o
  CC      arch/x86/events/rapl.o
  CC      crypto/akcipher.o
  CC      drivers/pci/pcie/rcec.o
  AR      fs/devpts/built-in.a
  CC      drivers/pci/hotplug/acpi_pcihp.o
  CC      sound/core/info.o
  CC      arch/x86/mm/maccess.o
  CC      arch/x86/kernel/apic/io_apic.o
  CC      io_uring/uring_cmd.o
  CC      mm/maccess.o
  CC      lib/lzo/lzo1x_decompress_safe.o
  CC      mm/page-writeback.o
  CC      kernel/irq/manage.o
  CC      arch/x86/kernel/cpu/aperfmperf.o
  CC      fs/proc/inode.o
  CC      fs/proc/root.o
  CC      fs/sysfs/mount.o
  CC      drivers/pinctrl/pinconf-generic.o
  CC      arch/x86/events/intel/uncore.o
  CC      lib/lz4/lz4_compress.o
  CC      drivers/video/console/dummycon.o
  CC      lib/lz4/lz4hc_compress.o
  CC      arch/x86/mm/pgprot.o
  CC      drivers/video/console/vgacon.o
  AR      arch/x86/pci/built-in.a
  CC      lib/lz4/lz4_decompress.o
  CC      net/core/secure_seq.o
  CC      drivers/video/logo/logo.o
  CC      net/core/flow_dissector.o
  CC      drivers/video/backlight/backlight.o
  CC      sound/core/isadma.o
  CC      block/blk-ioc.o
  AR      drivers/pci/controller/dwc/built-in.a
  CC      fs/configfs/dir.o
  AR      lib/lzo/built-in.a
  AR      drivers/pci/controller/mobiveil/built-in.a
  AR      drivers/pci/switch/built-in.a
  CC      drivers/pci/controller/vmd.o
  CC      drivers/pci/access.o
  CC      drivers/gpio/gpiolib-swnode.o
  CC      lib/zstd/zstd_compress_module.o
  CC      lib/xz/xz_dec_syms.o
  CC      drivers/pci/bus.o
  CC      drivers/pci/pcie/aspm.o
  CC      crypto/sig.o
  CC      drivers/pci/hotplug/pciehp_core.o
  CC      arch/x86/kernel/cpu/cpuid-deps.o
  AR      drivers/pci/msi/built-in.a
  CC      lib/xz/xz_dec_stream.o
  HOSTCC  drivers/video/logo/pnmtologo
  CC      arch/x86/mm/hugetlbpage.o
  CC      arch/x86/events/msr.o
  CC      lib/xz/xz_dec_lzma2.o
  CC      arch/x86/kernel/cpu/umwait.o
  AR      drivers/pinctrl/built-in.a
  CC      arch/x86/kernel/cpu/proc.o
  CC      kernel/power/snapshot.o
  CC      sound/core/vmaster.o
  CC      sound/core/ctljack.o
  AR      net/sched/built-in.a
  CC      sound/core/jack.o
  CC      sound/core/timer.o
  CC      block/blk-map.o
  CC      fs/sysfs/group.o
  CC      lib/xz/xz_dec_bcj.o
  LOGO    drivers/video/logo/logo_linux_clut224.c
  CC      drivers/video/logo/logo_linux_clut224.o
  CC      io_uring/epoll.o
  CC      lib/zstd/compress/fse_compress.o
  CC      fs/proc/base.o
  AR      drivers/video/logo/built-in.a
  MKCAP   arch/x86/kernel/cpu/capflags.c
  CC      drivers/pci/probe.o
  CC      kernel/rcu/sync.o
  CC      ipc/namespace.o
  CC      lib/zstd/compress/hist.o
  CC      lib/zstd/compress/huf_compress.o
  CC      io_uring/statx.o
  CC      drivers/video/fbdev/core/fb_notify.o
  AR      drivers/video/backlight/built-in.a
  CC      kernel/rcu/srcutree.o
  AR      drivers/video/fbdev/omap/built-in.a
  CC      kernel/rcu/tree.o
  AR      drivers/video/console/built-in.a
  CC      lib/zstd/compress/zstd_compress.o
  AR      kernel/livepatch/built-in.a
  AR      drivers/gpio/built-in.a
  CC      kernel/rcu/rcu_segcblist.o
  CC      block/blk-merge.o
  CC      kernel/dma/mapping.o
  CC      crypto/kpp.o
  CC      crypto/acompress.o
  CC      kernel/entry/common.o
  AR      kernel/sched/built-in.a
  CC      crypto/scompress.o
  CC      block/blk-timeout.o
  CC      fs/ext4/balloc.o
  CC      fs/jbd2/transaction.o
  CC      kernel/dma/direct.o
  CC      drivers/pci/hotplug/pciehp_ctrl.o
  CC      fs/ramfs/inode.o
  CC      kernel/dma/ops_helpers.o
  CC      kernel/irq/spurious.o
  CC      kernel/irq/resend.o
  CC      fs/jbd2/commit.o
  CC      fs/hugetlbfs/inode.o
  CC      fs/jbd2/recovery.o
  CC      arch/x86/mm/kasan_init_64.o
  AR      lib/xz/built-in.a
  CC      fs/jbd2/checkpoint.o
  AR      drivers/pci/controller/built-in.a
  AR      fs/sysfs/built-in.a
  CC      sound/core/hrtimer.o
  CC      drivers/pci/host-bridge.o
  CC      lib/zstd/compress/zstd_compress_literals.o
  CC      arch/x86/kernel/apic/msi.o
  CC      ipc/mq_sysctl.o
  CC      crypto/algboss.o
  CC      kernel/locking/lockdep_proc.o
  AR      lib/lz4/built-in.a
  CC      kernel/locking/spinlock.o
  CC      fs/configfs/symlink.o
  CC      drivers/pci/pcie/aer.o
  CC      drivers/pci/pcie/err.o
  CC      arch/x86/events/intel/uncore_nhmex.o
  CC      arch/x86/events/intel/uncore_snb.o
  CC      lib/raid6/algos.o
  CC      io_uring/net.o
  CC      lib/fonts/fonts.o
  CC [M]  drivers/video/fbdev/core/fb_info.o
  CC [M]  drivers/video/fbdev/core/fbmem.o
  CC      kernel/irq/chip.o
  CC      lib/argv_split.o
  CC      arch/x86/kernel/apic/x2apic_phys.o
  CC      net/netlink/af_netlink.o
  AR      net/bpf/built-in.a
  CC      crypto/testmgr.o
  CC      fs/ramfs/file-mmu.o
  AR      ipc/built-in.a
  CC      net/ethtool/ioctl.o
  CC      arch/x86/kernel/cpu/powerflags.o
  CC [M]  net/netfilter/ipvs/ip_vs_conn.o
  CC      net/ipv4/route.o
  AR      net/ipv4/netfilter/built-in.a
  CC [M]  net/ipv4/netfilter/nf_defrag_ipv4.o
  CC      drivers/pci/hotplug/pciehp_pci.o
  CC      mm/folio-compat.o
  CC      kernel/power/swap.o
  CC      net/ipv4/inetpeer.o
  CC      arch/x86/mm/numa.o
  CC      kernel/entry/syscall_user_dispatch.o
  CC      kernel/dma/dummy.o
  CC      kernel/entry/kvm.o
  CC      fs/configfs/mount.o
  CC      lib/fonts/font_8x8.o
  CC      net/core/sysctl_net_core.o
  CC      lib/fonts/font_8x16.o
  CC      sound/core/seq_device.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/kvm_main.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/eventfd.o
  CC      mm/readahead.o
  CC      lib/bug.o
  CC      fs/fat/cache.o
  CC      fs/nfs/client.o
  CC      arch/x86/kernel/apic/x2apic_cluster.o
  CC      fs/fat/dir.o
  CC      lib/raid6/recov.o
  CC      kernel/locking/osq_lock.o
  CC      arch/x86/mm/numa_64.o
  AR      fs/ramfs/built-in.a
  CC      kernel/locking/qspinlock.o
  CC      arch/x86/kernel/apic/apic_flat_64.o
  CC      io_uring/msg_ring.o
  LDS     arch/x86/kernel/vmlinux.lds
  CC      block/blk-lib.o
  CC      net/ipv4/protocol.o
  AR      lib/fonts/built-in.a
  CC      lib/buildid.o
  AR      fs/hugetlbfs/built-in.a
  HOSTCC  lib/raid6/mktables
  CC      drivers/pci/hotplug/pciehp_hpc.o
  CC      arch/x86/events/intel/uncore_snbep.o
  CC      kernel/power/user.o
  CC      kernel/module/main.o
  CC      kernel/dma/contiguous.o
  CC      kernel/dma/swiotlb.o
  CC      drivers/pci/pcie/aer_inject.o
  CC      fs/configfs/item.o
  CC [M]  drivers/video/fbdev/core/fbcmap.o
  CC      fs/ext4/bitmap.o
  CC      kernel/dma/remap.o
  CC      fs/jbd2/revoke.o
  CC [M]  sound/core/control_led.o
  CC      kernel/irq/dummychip.o
  CC      kernel/irq/devres.o
  CC      kernel/locking/rtmutex_api.o
  AR      kernel/entry/built-in.a
  CC      fs/nfs/dir.o
  CC      kernel/time/time.o
  UNROLL  lib/raid6/int1.c
  UNROLL  lib/raid6/int2.c
  UNROLL  lib/raid6/int4.c
  CC      kernel/time/timer.o
  UNROLL  lib/raid6/int8.c
  UNROLL  lib/raid6/int16.c
  UNROLL  lib/raid6/int32.c
  CC      lib/raid6/recov_ssse3.o
  CC [M]  net/ipv4/netfilter/nf_reject_ipv4.o
  CC      arch/x86/mm/amdtopology.o
  CC      lib/zstd/compress/zstd_compress_sequences.o
  CC      arch/x86/kernel/apic/probe_64.o
  CC      fs/proc/generic.o
  CC      block/blk-mq.o
  CC      lib/raid6/recov_avx2.o
  CC [M]  sound/pci/hda/patch_realtek.o
  CC      io_uring/timeout.o
  CC      mm/swap.o
  AR      fs/configfs/built-in.a
  CC      net/xfrm/xfrm_policy.o
  CC      fs/exportfs/expfs.o
  CC      net/xfrm/xfrm_state.o
  CC      arch/x86/mm/srat.o
  CC      kernel/irq/autoprobe.o
  CC      net/unix/af_unix.o
  CC      net/core/dev.o
  CC      net/unix/garbage.o
  CC      net/core/dev_addr_lists.o
  CC [M]  net/ipv4/netfilter/ip_tables.o
  AR      arch/x86/kernel/apic/built-in.a
  CC      fs/proc/array.o
  CC      kernel/power/poweroff.o
  AS      arch/x86/kernel/head_64.o
  CC      fs/ext4/block_validity.o
  CC      drivers/pci/pcie/pme.o
  CC [M]  sound/core/hwdep.o
  CC      drivers/pci/pcie/dpc.o
  CC      arch/x86/mm/pkeys.o
  CC [M]  drivers/video/fbdev/core/modedb.o
  CC      block/blk-mq-tag.o
  CC      block/blk-stat.o
  CC      fs/jbd2/journal.o
  AR      kernel/power/built-in.a
  CC      lib/zstd/compress/zstd_compress_superblock.o
  CC      fs/fat/fatent.o
  CC [M]  net/netfilter/ipvs/ip_vs_core.o
  CC      drivers/pci/hotplug/acpiphp_core.o
  CC      fs/fat/file.o
  CC      lib/raid6/mmx.o
  CC      lib/raid6/sse1.o
  CC      lib/raid6/sse2.o
  CC      kernel/irq/irqdomain.o
  AR      kernel/dma/built-in.a
  CC      arch/x86/mm/pti.o
  CC      kernel/futex/core.o
  AR      fs/exportfs/built-in.a
  CC      kernel/irq/proc.o
  CC      lib/zstd/compress/zstd_double_fast.o
  CC      kernel/futex/syscalls.o
  CC      lib/raid6/avx2.o
  CC      fs/lockd/clntlock.o
  CC [M]  net/netfilter/ipvs/ip_vs_ctl.o
  CC      crypto/cmac.o
  CC      fs/lockd/clntproc.o
  CC      kernel/locking/spinlock_debug.o
  CC      kernel/locking/qrwlock.o
  CC      net/unix/sysctl_net_unix.o
  CC      io_uring/sqpoll.o
  CC [M]  sound/core/pcm.o
  CC      net/ethtool/common.o
  AR      drivers/pci/pcie/built-in.a
  CC      fs/lockd/clntxdr.o
  CC      fs/ext4/dir.o
  CC      fs/lockd/host.o
  CC      drivers/pci/hotplug/acpiphp_glue.o
  CC [M]  drivers/video/fbdev/core/fbcvt.o
  CC      net/unix/diag.o
  CC      block/blk-mq-sysfs.o
  CC      net/xfrm/xfrm_hash.o
  CC      fs/proc/fd.o
  CC      arch/x86/kernel/head64.o
  CC      net/netlink/genetlink.o
  CC      mm/truncate.o
  AR      arch/x86/mm/built-in.a
  CC      lib/raid6/avx512.o
  CC [M]  drivers/video/fbdev/core/fb_cmdline.o
  CC      fs/proc/proc_tty.o
  CC      crypto/hmac.o
  CC      fs/proc/cmdline.o
  AR      kernel/locking/built-in.a
  CC      lib/raid6/recov_avx512.o
  CC      arch/x86/events/intel/uncore_discovery.o
  CC      fs/fat/inode.o
  CC [M]  net/ipv4/netfilter/iptable_filter.o
  CC      fs/fat/misc.o
  CC      kernel/futex/pi.o
  CC      kernel/module/strict_rwx.o
  CC      kernel/irq/migration.o
  CC      kernel/time/hrtimer.o
  CC      fs/fat/nfs.o
  CC [M]  drivers/video/fbdev/core/fb_io_fops.o
  CC [M]  sound/core/pcm_native.o
  CC      kernel/irq/cpuhotplug.o
  CC      arch/x86/kernel/cpu/feat_ctl.o
  CC      crypto/vmac.o
  CC      net/ethtool/netlink.o
  CC      crypto/xcbc.o
  CC      io_uring/fdinfo.o
  CC      drivers/idle/intel_idle.o
  CC [M]  net/ipv4/netfilter/iptable_mangle.o
  CC      fs/ext4/ext4_jbd2.o
  CC      crypto/crypto_null.o
  CC [M]  net/ipv4/netfilter/iptable_nat.o
  CC [M]  net/ipv4/netfilter/ipt_REJECT.o
  CC      io_uring/tctx.o
  TABLE   lib/raid6/tables.c
  CC      fs/proc/consoles.o
  CC      lib/raid6/int1.o
  CC      fs/proc/cpuinfo.o
  CC      lib/raid6/int2.o
  AR      drivers/pci/hotplug/built-in.a
  CC      drivers/pci/remove.o
  AR      kernel/rcu/built-in.a
  CC      arch/x86/kernel/cpu/intel.o
  CC      kernel/irq/pm.o
  CC      kernel/module/kmod.o
  CC      net/xfrm/xfrm_input.o
  CC      kernel/futex/requeue.o
  CC      kernel/futex/waitwake.o
  CC      fs/nls/nls_base.o
  CC      arch/x86/kernel/cpu/intel_pconfig.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/binary_stats.o
  CC      mm/vmscan.o
  CC      fs/lockd/svc.o
  CC      arch/x86/events/intel/cstate.o
  CC [M]  sound/pci/hda/patch_analog.o
  AR      drivers/char/ipmi/built-in.a
  CC      fs/ext4/extents.o
  CC      mm/shmem.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/vfio.o
  CC      fs/fat/namei_vfat.o
  CC      kernel/irq/msi.o
  CC      fs/nfs/file.o
  CC      kernel/module/tree_lookup.o
  CC      net/unix/scm.o
  CC [M]  drivers/video/fbdev/core/fb_backlight.o
  CC      crypto/md5.o
  CC      fs/ext4/extents_status.o
  CC      fs/proc/devices.o
  CC [M]  net/netfilter/ipvs/ip_vs_sched.o
  AR      fs/unicode/built-in.a
  CC [M]  arch/x86/kvm/../../../virt/kvm/coalesced_mmio.o
  CC      fs/nls/nls_cp437.o
  CC      lib/raid6/int4.o
  CC      io_uring/poll.o
  CC      drivers/pci/pci.o
  AR      kernel/futex/built-in.a
  CC      net/netlink/policy.o
  CC      fs/fat/namei_msdos.o
  CC      fs/nfs/getroot.o
  CC      io_uring/cancel.o
  CC      drivers/pci/pci-driver.o
  CC      net/netlink/diag.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/async_pf.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/irqchip.o
  AR      fs/jbd2/built-in.a
  CC      arch/x86/kernel/ebda.o
  CC      net/netfilter/core.o
  CC      lib/cmdline.o
  CC      kernel/module/debug_kmemleak.o
  CC      net/ethtool/bitset.o
  CC      net/ipv4/ip_input.o
  AR      drivers/idle/built-in.a
  CC [M]  drivers/video/fbdev/core/fbmon.o
  CC      fs/nls/nls_ascii.o
  CC      crypto/sha1_generic.o
  CC      kernel/time/timekeeping.o
  CC      net/ipv4/ip_fragment.o
  AR      arch/x86/events/intel/built-in.a
  AR      arch/x86/events/built-in.a
  CC      crypto/sha256_generic.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/dirty_ring.o
  CC      arch/x86/kernel/cpu/tsx.o
  CC      drivers/acpi/acpica/dsargs.o
  CC      drivers/acpi/acpica/dscontrol.o
  CC [M]  sound/pci/hda/patch_hdmi.o
  CC      lib/cpumask.o
  CC      fs/proc/interrupts.o
  CC [M]  net/netfilter/ipvs/ip_vs_xmit.o
  CC      lib/raid6/int8.o
  AR      net/unix/built-in.a
  CC      net/xfrm/xfrm_output.o
  AR      net/ipv6/netfilter/built-in.a
  CC [M]  net/ipv6/netfilter/nf_defrag_ipv6_hooks.o
  CC      block/blk-mq-cpumap.o
  CC [M]  net/ipv6/netfilter/nf_conntrack_reasm.o
  CC      fs/nls/nls_iso8859-1.o
  CC      drivers/acpi/numa/srat.o
  CC      net/ipv6/af_inet6.o
  CC      kernel/module/kallsyms.o
  CC      kernel/irq/affinity.o
  CC      fs/lockd/svclock.o
  CC      arch/x86/kernel/cpu/intel_epb.o
  CC      drivers/video/aperture.o
  CC      drivers/video/cmdline.o
  CC      drivers/acpi/acpica/dsdebug.o
  CC      drivers/video/nomodeset.o
  CC      lib/zstd/compress/zstd_fast.o
  CC      crypto/sha512_generic.o
  AR      fs/fat/built-in.a
  CC      drivers/pnp/pnpacpi/core.o
  CC [M]  drivers/video/fbdev/core/fb_defio.o
  AR      sound/spi/built-in.a
  CC      fs/proc/loadavg.o
  AR      sound/pci/pcxhr/built-in.a
  CC [M]  drivers/video/fbdev/core/fb_chrdev.o
  CC      fs/ntfs/aops.o
  CC [M]  drivers/video/fbdev/core/fb_procfs.o
  CC      fs/proc/meminfo.o
  CC      fs/nls/nls_utf8.o
  CC [M]  drivers/video/fbdev/core/fbsysfs.o
  AR      net/netlink/built-in.a
  AR      drivers/amba/built-in.a
  CC      fs/ntfs/attrib.o
  CC      net/packet/af_packet.o
  CC      io_uring/kbuf.o
  CC      net/packet/diag.o
  CC      arch/x86/kernel/cpu/amd.o
  CC      drivers/acpi/acpica/dsfield.o
  CC [M]  arch/x86/kvm/../../../virt/kvm/pfncache.o
  CC      kernel/irq/matrix.o
  CC      fs/nfs/inode.o
  CC      block/blk-mq-sched.o
  CC      lib/raid6/int16.o
  CC      lib/zstd/compress/zstd_lazy.o
  CC      net/ethtool/strset.o
  CC [M]  fs/nls/nls_ucs2_utils.o
  CC [M]  drivers/video/fbdev/core/fbcon.o
  AR      drivers/acpi/numa/built-in.a
  CC      drivers/acpi/apei/apei-base.o
  CC      drivers/acpi/apei/hest.o
  CC      fs/proc/stat.o
  CC      drivers/pnp/pnpacpi/rsparser.o
  CC      kernel/module/procfs.o
  CC      net/netfilter/nf_log.o
  CC      drivers/video/hdmi.o
  CC      fs/ext4/file.o
  CC      lib/zstd/compress/zstd_ldm.o
  CC      crypto/sha3_generic.o
  CC      fs/proc/uptime.o
  CC      kernel/time/ntp.o
  CC      drivers/pnp/core.o
  CC      net/ipv4/ip_forward.o
  CC [M]  sound/core/pcm_lib.o
  CC      fs/ntfs/collate.o
  AR      sound/parisc/built-in.a
  CC      drivers/acpi/acpica/dsinit.o
  CC      drivers/acpi/acpica/dsmethod.o
  CC      drivers/acpi/acpica/dsmthdat.o
  CC      net/xfrm/xfrm_sysctl.o
  CC      lib/ctype.o
  CC      lib/raid6/int32.o
  CC      net/xfrm/xfrm_replay.o
  AR      fs/nls/built-in.a
  CC      net/netfilter/nf_queue.o
  CC      drivers/acpi/apei/erst.o
  LD [M]  net/ipv6/netfilter/nf_defrag_ipv6.o
  CC      fs/lockd/svcshare.o
  CC      net/ipv6/anycast.o
  CC [M]  net/netfilter/ipvs/ip_vs_app.o
  CC      kernel/module/sysfs.o
  CC [M]  arch/x86/kvm/x86.o
  CC      net/ipv6/ip6_output.o
  CC      fs/proc/util.o
  CC      io_uring/rsrc.o
  CC      net/xfrm/xfrm_device.o
  CC      fs/lockd/svcproc.o
  CC      drivers/acpi/apei/bert.o
  CC      arch/x86/kernel/cpu/hygon.o
  CC      drivers/acpi/acpica/dsobject.o
  CC      crypto/blake2b_generic.o
  CC      block/ioctl.o
  CC      net/key/af_key.o
  CC      net/netfilter/nf_sockopt.o
  CC      net/netfilter/utils.o
  AR      net/bridge/netfilter/built-in.a
  CC      net/bridge/br.o
  CC [M]  net/netfilter/nfnetlink.o
  CC [M]  arch/x86/kvm/emulate.o
  CC      fs/ntfs/compress.o
  CC      net/bridge/br_device.o
  CC [M]  sound/pci/hda/hda_eld.o
  CC      kernel/time/clocksource.o
  CC      drivers/pnp/card.o
  CC      lib/dec_and_lock.o
  AR      drivers/pnp/pnpacpi/built-in.a
  CC      net/ethtool/linkinfo.o
  CC      fs/lockd/svcsubs.o
  AR      kernel/irq/built-in.a
  CC      net/ipv6/ip6_input.o
  CC      lib/raid6/tables.o
  CC      fs/ext4/fsmap.o
  CC      arch/x86/kernel/cpu/centaur.o
  CC      fs/proc/version.o
  CC      drivers/pnp/driver.o
  CC      drivers/pci/search.o
  CC      drivers/acpi/acpica/dsopcode.o
  CC      net/xfrm/xfrm_algo.o
  CC      net/ipv4/ip_options.o
  CC      arch/x86/kernel/cpu/zhaoxin.o
  AR      kernel/module/built-in.a
  CC      net/ethtool/linkmodes.o
  CC      fs/lockd/mon.o
  CC      drivers/acpi/apei/ghes.o
  AR      net/dsa/built-in.a
  CC [M]  net/sunrpc/auth_gss/auth_gss.o
  CC      crypto/ecb.o
  CC      fs/proc/softirqs.o
  CC [M]  net/sunrpc/auth_gss/gss_generic_token.o
  CC      arch/x86/kernel/cpu/perfctr-watchdog.o
  CC      block/genhd.o
  AR      lib/raid6/built-in.a
  CC [M]  net/netfilter/ipvs/ip_vs_sync.o
  CC      kernel/cgroup/cgroup.o
  CC [M]  net/netfilter/ipvs/ip_vs_est.o
  CC      fs/ext4/fsync.o
  CC      fs/proc/namespaces.o
  CC      drivers/pnp/resource.o
  CC      arch/x86/kernel/cpu/vmware.o
  CC      drivers/pnp/manager.o
  CC [M]  net/sunrpc/auth_gss/gss_mech_switch.o
  CC      drivers/acpi/acpica/dspkginit.o
  CC [M]  drivers/video/fbdev/core/bitblit.o
  CC      kernel/time/jiffies.o
  CC [M]  sound/pci/hda/hda_intel.o
  CC      lib/decompress.o
  CC [M]  sound/core/pcm_misc.o
  CC      net/ipv6/addrconf.o
  CC [M]  sound/core/pcm_memory.o
  CC      drivers/pci/pci-sysfs.o
  CC      io_uring/rw.o
  CC      net/ethtool/rss.o
  CC      fs/ntfs/debug.o
  CC      net/bridge/br_fdb.o
  CC      mm/util.o
  CC      crypto/cbc.o
  CC      drivers/pnp/support.o
  CC [M]  net/netfilter/ipvs/ip_vs_proto.o
  AR      drivers/video/fbdev/omap2/omapfb/dss/built-in.a
  AR      drivers/video/fbdev/omap2/omapfb/displays/built-in.a
  CC      fs/nfs/super.o
  AR      drivers/video/fbdev/omap2/omapfb/built-in.a
  AR      drivers/video/fbdev/omap2/built-in.a
  CC      kernel/time/timer_list.o
  CC [M]  drivers/video/fbdev/uvesafb.o
  CC      net/bridge/br_forward.o
  CC      drivers/acpi/acpica/dsutils.o
  CC      fs/ntfs/dir.o
  CC      net/xfrm/xfrm_user.o
  CC      drivers/pnp/interface.o
  CC      kernel/time/timeconv.o
  CC      arch/x86/kernel/cpu/hypervisor.o
  CC      kernel/cgroup/rstat.o
  CC [M]  net/sunrpc/auth_gss/svcauth_gss.o
  CC      fs/ntfs/file.o
  CC      fs/proc/self.o
  CC      fs/lockd/trace.o
  CC      net/ipv4/ip_output.o
  CC      fs/nfs/io.o
  CC      net/ethtool/linkstate.o
  CC      crypto/pcbc.o
  AR      drivers/acpi/apei/built-in.a
  CC      drivers/pci/rom.o
  CC      drivers/pnp/quirks.o
  CC      drivers/pci/setup-res.o
  CC      arch/x86/kernel/cpu/mshyperv.o
  CC [M]  sound/core/memalloc.o
  CC      net/core/dst.o
  CC      kernel/time/timecounter.o
  CC [M]  drivers/video/fbdev/core/softcursor.o
  CC [M]  sound/core/pcm_timer.o
  CC      net/ipv6/addrlabel.o
  CC      kernel/time/alarmtimer.o
  CC      drivers/acpi/acpica/dswexec.o
  CC      net/ethtool/debug.o
  CC      fs/ext4/hash.o
  AR      net/packet/built-in.a
  CC      mm/mmzone.o
  CC      net/8021q/vlan_core.o
  CC      block/ioprio.o
  CC      block/badblocks.o
  CC [M]  net/sunrpc/auth_gss/gss_rpc_upcall.o
  LD [M]  sound/pci/hda/snd-hda-codec.o
  CC      fs/proc/thread_self.o
  CC      drivers/pci/irq.o
  CC      crypto/cts.o
  AR      fs/hostfs/built-in.a
  CC      fs/debugfs/inode.o
  CC      drivers/pci/vpd.o
  CC      drivers/dma/dw/core.o
  AR      drivers/clk/actions/built-in.a
  AR      drivers/soc/apple/built-in.a
  AR      drivers/clk/analogbits/built-in.a
  AR      drivers/soc/aspeed/built-in.a
  AR      net/key/built-in.a
  AR      drivers/clk/bcm/built-in.a
  CC      arch/x86/kernel/platform-quirks.o
  AR      drivers/soc/bcm/built-in.a
  AR      drivers/clk/imgtec/built-in.a
  AR      drivers/soc/fsl/built-in.a
  AR      drivers/clk/imx/built-in.a
  AR      drivers/soc/fujitsu/built-in.a
  AR      drivers/clk/ingenic/built-in.a
  AR      drivers/soc/hisilicon/built-in.a
  AR      drivers/clk/mediatek/built-in.a
  AR      drivers/clk/microchip/built-in.a
  CC      drivers/acpi/acpica/dswload.o
  AR      drivers/soc/imx/built-in.a
  CC      drivers/dma/dw/dw.o
  AR      drivers/clk/mstar/built-in.a
  AR      drivers/soc/ixp4xx/built-in.a
  CC      fs/ntfs/index.o
  CC      drivers/dma/dw/idma32.o
  AR      drivers/clk/mvebu/built-in.a
  CC      drivers/dma/dw/acpi.o
  AR      drivers/soc/loongson/built-in.a
  AR      drivers/soc/mediatek/built-in.a
  CC      drivers/pnp/system.o
  AR      drivers/clk/ralink/built-in.a
  AR      drivers/clk/renesas/built-in.a
  AR      drivers/soc/microchip/built-in.a
  LD [M]  sound/pci/hda/snd-hda-codec-generic.o
  AR      drivers/clk/socfpga/built-in.a
  AR      drivers/soc/nuvoton/built-in.a
  CC [M]  net/netfilter/ipvs/ip_vs_pe.o
  LD [M]  sound/pci/hda/snd-hda-codec-realtek.o
  AR      drivers/clk/sprd/built-in.a
  AR      drivers/soc/pxa/built-in.a
  AR      drivers/soc/amlogic/built-in.a
  AR      drivers/clk/starfive/built-in.a
  LD [M]  sound/pci/hda/snd-hda-codec-analog.o
  CC      io_uring/opdef.o
  AR      drivers/soc/qcom/built-in.a
  AR      drivers/clk/sunxi-ng/built-in.a
  LD [M]  sound/pci/hda/snd-hda-codec-hdmi.o
  AR      drivers/soc/renesas/built-in.a
  AR      drivers/clk/ti/built-in.a
  CC      arch/x86/kernel/cpu/capflags.o
  CC      net/bridge/br_if.o
  CC [M]  drivers/video/fbdev/core/tileblit.o
  AR      drivers/soc/rockchip/built-in.a
  AR      drivers/clk/versatile/built-in.a
  LD [M]  sound/pci/hda/snd-hda-intel.o
  CC      mm/vmstat.o
  AR      drivers/soc/sifive/built-in.a
  CC [M]  drivers/video/fbdev/core/fb_logo.o
  CC      drivers/clk/x86/clk-lpss-atom.o
  CC      fs/lockd/xdr.o
  AR      drivers/soc/sunxi/built-in.a
  AR      arch/x86/kernel/cpu/built-in.a
  AR      sound/pci/riptide/built-in.a
  AR      sound/pci/rme9652/built-in.a
  AR      drivers/soc/ti/built-in.a
  CC      arch/x86/kernel/process_64.o
  AR      sound/pci/trident/built-in.a
  AR      drivers/soc/xilinx/built-in.a
  AR      sound/pci/ymfpci/built-in.a
  CC      fs/nfs/direct.o
  CC      net/ethtool/wol.o
  AR      drivers/soc/built-in.a
  CC      fs/proc/proc_sysctl.o
  LD [M]  sound/core/snd-ctl-led.o
  AR      sound/pci/vx222/built-in.a
  CC      net/ethtool/features.o
  AR      sound/pci/built-in.a
  LD [M]  sound/core/snd-hwdep.o
  LD [M]  sound/core/snd-pcm.o
  CC      fs/ext4/ialloc.o
  CC      arch/x86/kernel/signal.o
  AR      sound/core/built-in.a
  CC      net/ipv6/route.o
  AR      drivers/clk/xilinx/built-in.a
  CC      drivers/dma/dw/pci.o
  AR      sound/pcmcia/vx/built-in.a
  AR      sound/pcmcia/pdaudiocf/built-in.a
  CC      lib/zstd/compress/zstd_opt.o
  AR      sound/pcmcia/built-in.a
  CC      net/ipv6/ip6_fib.o
  CC      drivers/clk/clk-devres.o
  CC [M]  net/netfilter/ipvs/ip_vs_proto_tcp.o
  AR      sound/mips/built-in.a
  CC      block/blk-rq-qos.o
  AR      sound/soc/built-in.a
  AR      sound/atmel/built-in.a
  AR      sound/hda/built-in.a
  CC      crypto/lrw.o
  CC      drivers/acpi/acpica/dswload2.o
  CC [M]  sound/hda/hda_bus_type.o
  CC      net/ipv6/ipv6_sockglue.o
  CC      net/core/netevent.o
  AR      drivers/pnp/built-in.a
  AR      sound/x86/built-in.a
  CC      drivers/clk/x86/clk-pmc-atom.o
  CC      kernel/time/posix-timers.o
  AR      drivers/pmdomain/actions/built-in.a
  CC      drivers/virtio/virtio.o
  AR      drivers/pmdomain/amlogic/built-in.a
  AR      drivers/pmdomain/apple/built-in.a
  AR      drivers/pmdomain/bcm/built-in.a
  CC      drivers/tty/vt/vt_ioctl.o
  CC      drivers/pci/setup-bus.o
  AR      drivers/pmdomain/imx/built-in.a
  CC      fs/debugfs/file.o
  CC      drivers/virtio/virtio_ring.o
  AR      drivers/pmdomain/mediatek/built-in.a
  CC      drivers/tty/hvc/hvc_console.o
  CC [M]  net/sunrpc/auth_gss/gss_rpc_xdr.o
  AR      drivers/pmdomain/qcom/built-in.a
  AR      drivers/pmdomain/renesas/built-in.a
  AR      drivers/pmdomain/rockchip/built-in.a
  AR      drivers/pmdomain/samsung/built-in.a
  CC      fs/ntfs/inode.o
  AR      drivers/pmdomain/st/built-in.a
  AR      drivers/pmdomain/starfive/built-in.a
  AR      drivers/pmdomain/sunxi/built-in.a
  AR      drivers/pmdomain/tegra/built-in.a
  CC      drivers/tty/vt/vc_screen.o
  AR      drivers/pmdomain/ti/built-in.a
  AR      drivers/pmdomain/xilinx/built-in.a
  AR      drivers/pmdomain/built-in.a
  CC      kernel/time/posix-cpu-timers.o
  CC [M]  net/8021q/vlan.o
  CC      drivers/char/hw_random/core.o
  CC      kernel/cgroup/namespace.o
  CC      drivers/acpi/acpica/dswscope.o
  CC      drivers/acpi/acpica/dswstate.o
  CC [M]  sound/hda/hdac_bus.o
  CC      mm/backing-dev.o
  CC      io_uring/notif.o
  AR      drivers/dma/dw/built-in.a
  CC [M]  drivers/video/fbdev/core/cfbfillrect.o
  CC      drivers/tty/vt/selection.o
  CC      drivers/dma/hsu/hsu.o
  CC      drivers/tty/vt/keyboard.o
  CC [M]  net/netfilter/ipvs/ip_vs_proto_udp.o
  CC      fs/lockd/clnt4xdr.o
  CC      crypto/xts.o
  CC      block/disk-events.o
  AR      drivers/clk/x86/built-in.a
  CC      drivers/clk/clk-bulk.o
  CC      fs/lockd/xdr4.o
  CC      net/core/neighbour.o
  CC      net/ethtool/privflags.o
  CC      arch/x86/kernel/signal_64.o
  CC      fs/proc/proc_net.o
  CC      drivers/acpi/acpica/evevent.o
  AR      net/xfrm/built-in.a
  CC      net/dcb/dcbnl.o
  CC      net/bridge/br_input.o
  CC      kernel/cgroup/cgroup-v1.o
  CC      kernel/cgroup/freezer.o
  AR      drivers/tty/hvc/built-in.a
  CC      net/ethtool/rings.o
  CC      drivers/char/agp/backend.o
  AR      fs/debugfs/built-in.a
  CC      drivers/tty/vt/consolemap.o
  CC      drivers/char/hw_random/intel-rng.o
  CC      drivers/char/agp/generic.o
  CC      fs/ntfs/mft.o
  CC      net/dcb/dcbevent.o
  CC      drivers/clk/clkdev.o
  CC      kernel/cgroup/legacy_freezer.o
  CC      net/ipv4/ip_sockglue.o
  CC [M]  net/sunrpc/auth_gss/trace.o
  CC [M]  sound/hda/hdac_device.o
  HOSTCC  drivers/tty/vt/conmakehash
  CC      drivers/acpi/acpica/evgpe.o
  CC      crypto/ctr.o
  CC      block/blk-ia-ranges.o
  CC [M]  drivers/video/fbdev/core/cfbcopyarea.o
  AR      drivers/dma/hsu/built-in.a
  CC [M]  net/8021q/vlan_dev.o
  AR      drivers/dma/idxd/built-in.a
  AR      drivers/dma/mediatek/built-in.a
  CC      kernel/time/posix-clock.o
  AR      drivers/dma/qcom/built-in.a
  CC      drivers/tty/vt/vt.o
  CC      fs/nfs/pagelist.o
  AR      drivers/dma/ti/built-in.a
  CC      fs/nfs/read.o
  CC      io_uring/io-wq.o
  AR      drivers/dma/xilinx/built-in.a
  CC [M]  drivers/dma/ioat/init.o
  CC      arch/x86/kernel/traps.o
  CC      crypto/gcm.o
  CC      fs/proc/kcore.o
  CC      drivers/clk/clk.o
  CC      drivers/pci/vc.o
  CC      drivers/char/tpm/tpm-chip.o
  CC      drivers/virtio/virtio_anchor.o
  CC      drivers/char/tpm/tpm-dev-common.o
  CC      net/bridge/br_ioctl.o
  AR      drivers/char/hw_random/built-in.a
  CC      arch/x86/kernel/idt.o
  CC [M]  net/netfilter/ipvs/ip_vs_nfct.o
  CC      fs/ext4/indirect.o
  CC      drivers/char/mem.o
  CC      fs/lockd/svc4proc.o
  CC      drivers/acpi/acpica/evgpeblk.o
  CC      drivers/acpi/acpica/evgpeinit.o
  CC      net/ipv4/inet_hashtables.o
  CC      net/ipv4/inet_timewait_sock.o
  CC      fs/ntfs/mst.o
  CC      mm/mm_init.o
  CC      drivers/virtio/virtio_pci_modern_dev.o
  CC      net/ethtool/channels.o
  CC      drivers/char/random.o
  CC      net/ipv6/ndisc.o
  CC      block/early-lookup.o
  CC [M]  arch/x86/kvm/i8259.o
  CC [M]  sound/hda/hdac_sysfs.o
  CC      kernel/time/itimer.o
  CC      drivers/char/agp/isoch.o
  CC [M]  drivers/video/fbdev/core/cfbimgblt.o
  CC      kernel/cgroup/pids.o
  CC      drivers/virtio/virtio_pci_legacy_dev.o
  CC      net/bridge/br_stp.o
  CC [M]  sound/hda/hdac_regmap.o
  CC      drivers/acpi/acpica/evgpeutil.o
  CC      drivers/pci/mmap.o
  CC      net/core/rtnetlink.o
  CC      drivers/char/tpm/tpm-dev.o
  CC [M]  drivers/dma/ioat/dma.o
  CC      drivers/dma/dmaengine.o
  CC      fs/proc/kmsg.o
  CC      arch/x86/kernel/irq.o
  CC      fs/ntfs/namei.o
  CC      block/bsg.o
  CC [M]  net/8021q/vlan_netlink.o
  CC [M]  drivers/dma/ioat/prep.o
  CC      crypto/pcrypt.o
  COPY    drivers/tty/vt/defkeymap.c
  CC [M]  drivers/video/fbdev/simplefb.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_mech.o
  AR      net/dcb/built-in.a
  CC      drivers/char/misc.o
  CC      net/l3mdev/l3mdev.o
  CC [M]  net/netfilter/ipvs/ip_vs_rr.o
  CC      drivers/acpi/acpica/evglock.o
  CC      fs/lockd/procfs.o
  CC      kernel/cgroup/cpuset.o
  CC      net/ipv6/udp.o
  CC      drivers/virtio/virtio_pci_modern.o
  CC      drivers/char/agp/intel-agp.o
  CC      net/bridge/br_stp_bpdu.o
  AR      io_uring/built-in.a
  CC      net/ethtool/coalesce.o
  CC      arch/x86/kernel/irq_64.o
  CC      fs/proc/page.o
  CC      drivers/pci/setup-irq.o
  CC      fs/tracefs/inode.o
  CC [M]  sound/hda/hdac_controller.o
  CC      drivers/char/tpm/tpm-interface.o
  CC [M]  sound/hda/hdac_stream.o
  CC      kernel/time/clockevents.o
  CC [M]  sound/hda/array.o
  CC      lib/decompress_bunzip2.o
  CC [M]  drivers/video/fbdev/core/sysfillrect.o
  CC      fs/ntfs/runlist.o
  CC      crypto/cryptd.o
  CC      fs/ntfs/super.o
  CC      drivers/acpi/acpica/evhandler.o
  CC      block/bsg-lib.o
  CC      drivers/acpi/acpica/evmisc.o
  CC      mm/percpu.o
  LD [M]  net/netfilter/ipvs/ip_vs.o
  CC      fs/ext4/inline.o
  CC      net/ipv6/udplite.o
  CC      fs/nfs/symlink.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_seal.o
  CC [M]  net/8021q/vlanproc.o
  CC      arch/x86/kernel/dumpstack_64.o
  CC      net/bridge/br_stp_if.o
  CC      block/blk-cgroup.o
  CC      arch/x86/kernel/time.o
  AR      fs/lockd/built-in.a
  CC      fs/ext4/inode.o
  CC      drivers/pci/proc.o
  CC      fs/ext4/ioctl.o
  AR      net/l3mdev/built-in.a
  CC      net/handshake/alert.o
  CC      drivers/virtio/virtio_pci_common.o
  CC [M]  drivers/dma/ioat/dca.o
  CC      net/handshake/genl.o
  CC      drivers/pci/slot.o
  CC      kernel/time/tick-common.o
  CC      drivers/acpi/acpica/evregion.o
  CC      fs/tracefs/event_inode.o
  CC      drivers/pci/pci-acpi.o
  CC      drivers/char/agp/intel-gtt.o
  CC      lib/decompress_inflate.o
  CC      drivers/char/tpm/tpm1-cmd.o
  CC [M]  net/netfilter/nf_conntrack_core.o
  CC      net/core/utils.o
  CC [M]  sound/hda/hdmi_chmap.o
  CC      net/ipv4/inet_connection_sock.o
  AR      fs/proc/built-in.a
  CONMK   drivers/tty/vt/consolemap_deftbl.c
  CC      drivers/tty/vt/defkeymap.o
  CC [M]  drivers/video/fbdev/core/syscopyarea.o
  CC [M]  drivers/dma/ioat/sysfs.o
  CC      net/ethtool/pause.o
  CC      drivers/virtio/virtio_pci_legacy.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_unseal.o
  CC      arch/x86/kernel/ioport.o
  CC      lib/zstd/zstd_decompress_module.o
  CC      fs/btrfs/super.o
  CC      block/blk-cgroup-rwstat.o
  CC      net/ipv4/tcp.o
  CC      drivers/tty/vt/consolemap_deftbl.o
  AR      drivers/tty/vt/built-in.a
  CC      drivers/tty/serial/8250/8250_core.o
  CC      crypto/des_generic.o
  CC      drivers/tty/serial/8250/8250_pnp.o
  CC      drivers/tty/serial/serial_core.o
  AR      net/8021q/built-in.a
  CC      drivers/acpi/acpica/evrgnini.o
  LD [M]  net/8021q/8021q.o
  CC      net/ipv4/tcp_input.o
  CC      drivers/acpi/acpica/evsci.o
  CC      net/ipv4/tcp_output.o
  CC      fs/nfs/unlink.o
  CC      net/ipv4/tcp_timer.o
  CC      crypto/aes_generic.o
  CC      net/ethtool/eee.o
  CC      lib/zstd/decompress/huf_decompress.o
  CC      drivers/acpi/acpica/evxface.o
  CC      lib/zstd/decompress/zstd_ddict.o
  CC      fs/ntfs/sysctl.o
  CC      fs/ntfs/unistr.o
  AR      fs/tracefs/built-in.a
  CC      fs/pstore/inode.o
  LD [M]  drivers/dma/ioat/ioatdma.o
  CC      fs/efivarfs/inode.o
  CC      kernel/time/tick-broadcast.o
  CC      drivers/dma/virt-dma.o
  CC      drivers/char/tpm/tpm2-cmd.o
  CC      net/bridge/br_stp_timer.o
  CC [M]  drivers/virtio/virtio_mem.o
  CC      fs/efivarfs/file.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_wrap.o
  CC      drivers/clk/clk-divider.o
  CC      fs/pstore/platform.o
  CC      arch/x86/kernel/dumpstack.o
  AR      drivers/char/agp/built-in.a
  CC      drivers/acpi/acpica/evxfevnt.o
  CC      net/handshake/netlink.o
  CC      drivers/char/virtio_console.o
  CC      fs/efivarfs/super.o
  CC      drivers/pci/quirks.o
  CC [M]  sound/hda/trace.o
  CC [M]  drivers/video/fbdev/core/sysimgblt.o
  CC      net/ipv6/raw.o
  CC      net/ethtool/tsinfo.o
  CC      drivers/char/tpm/tpmrm-dev.o
  CC      net/core/link_watch.o
  CC      net/ipv4/tcp_ipv4.o
  CC      drivers/acpi/acpica/evxfgpe.o
  CC      fs/ntfs/upcase.o
  CC      drivers/acpi/acpica/evxfregn.o
  CC      fs/efivarfs/vars.o
  CC      crypto/crc32c_generic.o
  CC      drivers/tty/serial/8250/8250_port.o
  CC      drivers/char/tpm/tpm2-space.o
  CC      kernel/time/tick-broadcast-hrtimer.o
  CC      kernel/time/tick-oneshot.o
  CC      drivers/dma/acpi-dma.o
  CC      net/ipv6/icmp.o
  CC      block/blk-throttle.o
  CC [M]  sound/hda/hdac_component.o
  CC      drivers/clk/clk-fixed-factor.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_crypto.o
  CC      fs/btrfs/ctree.o
  CC      drivers/clk/clk-fixed-rate.o
  CC      arch/x86/kernel/nmi.o
  CC      fs/pstore/pmsg.o
  CC      drivers/char/tpm/tpm-sysfs.o
  AR      kernel/cgroup/built-in.a
  AR      drivers/tty/ipwireless/built-in.a
  AR      drivers/acpi/pmic/built-in.a
  CC      drivers/acpi/acpica/exconcat.o
  CC      drivers/tty/tty_io.o
  CC      drivers/acpi/dptf/int340x_thermal.o
  CC      drivers/tty/n_tty.o
  CC      fs/btrfs/extent-tree.o
  CC      crypto/crct10dif_common.o
  CC [M]  net/sunrpc/auth_gss/gss_krb5_keys.o
  CC      mm/slab_common.o
  CC      net/bridge/br_netlink.o
  CC      fs/nfs/write.o
  CC [M]  drivers/video/fbdev/core/fb_sys_fops.o
  CC      kernel/time/tick-sched.o
  AR      fs/ntfs/built-in.a
  AR      sound/xen/built-in.a
  CC      drivers/acpi/tables.o
  CC      net/ethtool/cabletest.o
  CC      drivers/acpi/blacklist.o
  CC      net/handshake/request.o
  CC      drivers/clk/clk-gate.o
  CC      drivers/tty/tty_ioctl.o
  AR      fs/efivarfs/built-in.a
  AR      fs/pstore/built-in.a
  CC      drivers/tty/serial/serial_base_bus.o
  CC [M]  fs/netfs/buffered_read.o
  CC      net/ipv4/tcp_minisocks.o
  CC [M]  fs/netfs/io.o
  CC      crypto/crct10dif_generic.o
  AR      drivers/dma/built-in.a
  CC      drivers/acpi/acpica/exconfig.o
  CC [M]  fs/netfs/iterator.o
  AR      drivers/acpi/dptf/built-in.a
  CC      drivers/clk/clk-multiplier.o
  CC [M]  sound/hda/hdac_i915.o
  CC      kernel/time/vsyscall.o
  AR      sound/virtio/built-in.a
  CC      fs/ext4/mballoc.o
  CC      arch/x86/kernel/ldt.o
  CC [M]  net/netfilter/nf_conntrack_standalone.o
  CC      net/core/filter.o
  CC      drivers/acpi/osi.o
  AS      arch/x86/kernel/ibt_selftest.o
  CC      drivers/char/tpm/eventlog/common.o
  CC      arch/x86/kernel/setup.o
  CC      drivers/acpi/acpica/exconvrt.o
  AR      drivers/virtio/built-in.a
  AR      drivers/iommu/amd/built-in.a
  LD [M]  net/sunrpc/auth_gss/auth_rpcgss.o
  CC      drivers/iommu/intel/dmar.o
  CC      crypto/authenc.o
  AR      drivers/gpu/host1x/built-in.a
  LD [M]  net/sunrpc/auth_gss/rpcsec_gss_krb5.o
  AR      drivers/gpu/vga/built-in.a
  CC      net/sunrpc/clnt.o
  AR      drivers/gpu/drm/tests/built-in.a
  CC [M]  drivers/gpu/drm/tests/drm_kunit_helpers.o
  LD [M]  drivers/video/fbdev/core/fb.o
  CC      drivers/connector/cn_queue.o
  CC      drivers/clk/clk-mux.o
  CC      drivers/connector/connector.o
  AR      drivers/video/fbdev/core/built-in.a
  AR      drivers/video/fbdev/built-in.a
  AR      drivers/video/built-in.a
  CC      net/sunrpc/xprt.o
  CC      drivers/tty/serial/8250/8250_dma.o
  CC      arch/x86/kernel/x86_init.o
  CC [M]  fs/netfs/main.o
  CC      block/mq-deadline.o
  CC [M]  sound/hda/intel-dsp-config.o
  AR      drivers/gpu/drm/arm/built-in.a
  CC      drivers/char/hpet.o
  CC      kernel/time/timekeeping_debug.o
  CC      drivers/tty/serial/8250/8250_dwlib.o
  CC      drivers/pci/ats.o
  CC [M]  drivers/gpu/drm/tests/drm_buddy_test.o
  CC      drivers/acpi/acpica/excreate.o
  CC      net/ipv6/mcast.o
  CC      net/ethtool/tunnels.o
  CC      drivers/char/tpm/eventlog/tpm1.o
  CC      net/ipv6/reassembly.o
  CC [M]  fs/netfs/objects.o
  CC      kernel/time/namespace.o
  CC [M]  sound/hda/intel-nhlt.o
  CC      drivers/clk/clk-composite.o
  CC      net/ethtool/fec.o
  CC [M]  drivers/gpu/drm/tests/drm_cmdline_parser_test.o
  CC      net/handshake/tlshd.o
  CC [M]  arch/x86/kvm/irq.o
  CC      mm/compaction.o
  CC      arch/x86/kernel/i8259.o
  CC [M]  drivers/gpu/drm/tests/drm_connector_test.o
  CC      drivers/acpi/acpica/exdebug.o
  CC      drivers/acpi/acpica/exdump.o
  CC [M]  arch/x86/kvm/lapic.o
  CC      drivers/tty/serial/8250/8250_pcilib.o
  CC      crypto/authencesn.o
  CC      arch/x86/kernel/irqinit.o
  CC      arch/x86/kernel/jump_label.o
  CC      drivers/tty/tty_ldisc.o
  CC [M]  net/bluetooth/af_bluetooth.o
  CC [M]  net/netfilter/nf_conntrack_expect.o
  CC      net/ipv4/tcp_cong.o
  CC      net/bridge/br_netlink_tunnel.o
  CC      drivers/tty/tty_buffer.o
  CC      drivers/connector/cn_proc.o
  CC      drivers/pci/iov.o
  CC      net/ipv4/tcp_metrics.o
  CC      net/ethtool/eeprom.o
  CC      drivers/clk/clk-fractional-divider.o
  CC [M]  sound/hda/intel-sdw-acpi.o
  CC      drivers/char/nvram.o
  CC      drivers/char/tpm/eventlog/tpm2.o
  CC [M]  net/dns_resolver/dns_key.o
  CC      drivers/clk/clk-gpio.o
  CC      net/devres.o
  AR      kernel/time/built-in.a
  CC      drivers/acpi/acpica/exfield.o
  LD [M]  fs/netfs/netfs.o
  CC      net/handshake/trace.o
  CC      kernel/trace/trace_clock.o
  CC      net/sunrpc/socklib.o
  CC      kernel/trace/ftrace.o
  CC      drivers/base/power/sysfs.o
  CC      block/kyber-iosched.o
  CC      drivers/tty/serial/8250/8250_pci.o
  CC      drivers/base/regmap/regmap.o
  CC      drivers/base/firmware_loader/builtin/main.o
  CC      drivers/base/firmware_loader/main.o
  AR      drivers/base/test/built-in.a
  CC      drivers/iommu/intel/iommu.o
  CC      drivers/tty/serial/8250/8250_exar.o
  CC      fs/nfs/namespace.o
  CC      arch/x86/kernel/irq_work.o
  CC      drivers/iommu/intel/pasid.o
  CC      drivers/base/regmap/regcache.o
  CC      net/sunrpc/xprtsock.o
  CC      block/bfq-iosched.o
  LD [M]  sound/hda/snd-hda-core.o
  CC      drivers/acpi/acpica/exfldio.o
  LD [M]  sound/hda/snd-intel-dspcfg.o
  LD [M]  sound/hda/snd-intel-sdw-acpi.o
  CC      sound/sound_core.o
  CC      sound/last.o
  CC [M]  net/dns_resolver/dns_query.o
  CC      crypto/lzo.o
  AR      drivers/clk/built-in.a
  AR      drivers/base/firmware_loader/builtin/built-in.a
  CC      drivers/char/tpm/tpm_ppi.o
  CC      drivers/acpi/acpica/exmisc.o
  CC [M]  fs/fscache/cache.o
  CC      drivers/char/tpm/eventlog/acpi.o
  CC [M]  fs/fscache/cookie.o
  CC      lib/zstd/decompress/zstd_decompress.o
  CC      drivers/base/power/generic_ops.o
  CC      drivers/char/tpm/eventlog/efi.o
  CC      drivers/pci/pci-label.o
  CC      net/ethtool/stats.o
  CC      net/bridge/br_arp_nd_proxy.o
  AR      drivers/connector/built-in.a
  CC      net/ipv6/tcp_ipv6.o
  CC      drivers/block/loop.o
  AR      sound/built-in.a
  CC      kernel/trace/ring_buffer.o
  CC [M]  net/bluetooth/hci_core.o
  CC      drivers/tty/tty_port.o
  CC      net/ipv6/ping.o
  CC [M]  drivers/gpu/drm/tests/drm_damage_helper_test.o
  CC      mm/show_mem.o
  CC      drivers/acpi/acpica/exmutex.o
  CC      net/bridge/br_sysfs_if.o
  CC      crypto/lzo-rle.o
  CC [M]  net/netfilter/nf_conntrack_helper.o
  CC      arch/x86/kernel/probe_roms.o
  CC [M]  fs/smb/common/cifs_arc4.o
  CC      net/sunrpc/sched.o
  LD [M]  net/dns_resolver/dns_resolver.o
  CC [M]  fs/smb/common/cifs_md4.o
  CC      drivers/base/power/common.o
  CC      lib/decompress_unlz4.o
  CC      lib/decompress_unlzma.o
  CC [M]  fs/smb/client/trace.o
  AR      drivers/base/firmware_loader/built-in.a
  CC      drivers/char/tpm/tpm_crb.o
  CC      net/ipv4/tcp_fastopen.o
  CC      fs/ext4/migrate.o
  AR      net/handshake/built-in.a
  CC      net/socket.o
  CC [M]  fs/smb/client/cifsfs.o
  CC [M]  drivers/block/nbd.o
  CC      net/ipv4/tcp_rate.o
  CC      fs/nfs/mount_clnt.o
  CC      drivers/tty/serial/8250/8250_early.o
  CC      lib/zstd/decompress/zstd_decompress_block.o
  CC      drivers/acpi/acpica/exnames.o
  CC      drivers/pci/pci-stub.o
  CC      fs/btrfs/print-tree.o
  CC      drivers/pci/vgaarb.o
  CC [M]  fs/fscache/io.o
  CC [M]  fs/fscache/main.o
  CC      crypto/lz4.o
  CC      net/ipv4/tcp_recovery.o
  CC      drivers/tty/tty_mutex.o
  CC      drivers/base/power/qos.o
  CC      net/sunrpc/auth.o
  AR      drivers/gpu/drm/display/built-in.a
  CC [M]  drivers/gpu/drm/display/drm_display_helper_mod.o
  CC      arch/x86/kernel/sys_ia32.o
  CC      net/ethtool/phc_vclocks.o
  CC      mm/shmem_quota.o
  CC      drivers/acpi/acpica/exoparg1.o
  CC      fs/nfs/nfstrace.o
  CC      drivers/acpi/acpica/exoparg2.o
  CC      net/sunrpc/auth_null.o
  CC [M]  drivers/gpu/drm/tests/drm_dp_mst_helper_test.o
  CC      drivers/acpi/acpica/exoparg3.o
  CC [M]  arch/x86/kvm/i8254.o
  CC      drivers/tty/serial/8250/8250_dw.o
  CC      drivers/acpi/acpica/exoparg6.o
  CC      crypto/lz4hc.o
  AR      drivers/char/tpm/built-in.a
  AR      drivers/char/built-in.a
  CC [M]  drivers/gpu/drm/display/drm_dp_dual_mode_helper.o
  CC [M]  fs/smb/client/cifs_debug.o
  CC      net/bridge/br_sysfs_br.o
  CC      net/bridge/br_nf_core.o
  CC      drivers/iommu/intel/trace.o
  CC [M]  drivers/gpu/drm/tests/drm_format_helper_test.o
  CC      drivers/tty/tty_ldsem.o
  CC [M]  net/netfilter/nf_conntrack_proto.o
  CC [M]  drivers/gpu/drm/display/drm_dp_helper.o
  CC [M]  net/bluetooth/hci_conn.o
  AR      drivers/block/built-in.a
  CC      drivers/acpi/acpica/exprep.o
  CC      drivers/acpi/acpica/exregion.o
  CC      drivers/acpi/acpica/exresnte.o
  CC      lib/zstd/zstd_common_module.o
  CC      lib/zstd/common/debug.o
  CC [M]  drivers/gpu/drm/display/drm_dp_mst_topology.o
  CC      mm/interval_tree.o
  CC      crypto/xxhash_generic.o
  CC [M]  net/netfilter/nf_conntrack_proto_generic.o
  AR      drivers/pci/built-in.a
  CC      fs/nfs/export.o
  CC      arch/x86/kernel/signal_32.o
  CC      net/compat.o
  CC      drivers/base/power/runtime.o
  CC      fs/btrfs/root-tree.o
  CC [M]  fs/fscache/volume.o
  CC      net/ipv4/tcp_ulp.o
  CC      net/ethtool/mm.o
  CC [M]  drivers/gpu/drm/tests/drm_format_test.o
  CC      drivers/base/regmap/regcache-rbtree.o
  CC      mm/list_lru.o
  CC [M]  drivers/gpu/drm/display/drm_dsc_helper.o
  CC [M]  drivers/gpu/drm/display/drm_hdcp_helper.o
  CC      drivers/acpi/acpica/exresolv.o
  CC      drivers/acpi/acpica/exresop.o
  CC      drivers/tty/serial/8250/8250_lpss.o
  CC [M]  drivers/gpu/drm/tests/drm_framebuffer_test.o
  CC      crypto/rng.o
  CC [M]  arch/x86/kvm/ioapic.o
  CC      fs/ext4/mmp.o
  CC [M]  drivers/gpu/drm/tests/drm_managed_test.o
  CC      drivers/base/component.o
  CC      drivers/iommu/intel/cap_audit.o
  CC      net/sunrpc/auth_tls.o
  CC      net/sysctl_net.o
  CC [M]  fs/smb/client/connect.o
  CC      net/ipv6/exthdrs.o
  CC [M]  drivers/gpu/drm/tests/drm_mm_test.o
  CC      block/bfq-wf2q.o
  CC [M]  fs/fscache/proc.o
  CC      net/ipv4/tcp_offload.o
  CC      kernel/trace/trace.o
  CC      net/ipv4/tcp_plb.o
  CC      arch/x86/kernel/sys_x86_64.o
  CC      drivers/acpi/acpica/exserial.o
  CC      drivers/tty/tty_baudrate.o
  CC [M]  net/bluetooth/hci_event.o
  CC      drivers/tty/tty_jobctrl.o
  CC      net/bridge/br_multicast.o
  CC [M]  net/bluetooth/mgmt.o
  CC      drivers/base/regmap/regcache-flat.o
  CC      drivers/acpi/osl.o
  CC      drivers/base/core.o
  CC [M]  net/netfilter/nf_conntrack_proto_tcp.o
  CC      net/sunrpc/auth_unix.o
  CC      drivers/tty/serial/8250/8250_mid.o
  CC      drivers/acpi/acpica/exstore.o
  CC      net/ethtool/module.o
  CC      crypto/drbg.o
  CC      net/sunrpc/svc.o
  CC      fs/ext4/move_extent.o
  CC [M]  drivers/gpu/drm/tests/drm_modes_test.o
  CC      drivers/base/power/wakeirq.o
  CC      mm/workingset.o
  CC      block/bfq-cgroup.o
  AR      drivers/misc/eeprom/built-in.a
  CC      fs/btrfs/dir-item.o
  AR      drivers/misc/cb710/built-in.a
  AR      drivers/gpu/drm/renesas/rcar-du/built-in.a
  AR      drivers/misc/ti-st/built-in.a
  AR      drivers/gpu/drm/renesas/built-in.a
  CC      mm/debug.o
  AR      drivers/misc/lis3lv02d/built-in.a
  AR      drivers/gpu/drm/omapdrm/built-in.a
  AR      drivers/gpu/drm/tilcdc/built-in.a
  CC      mm/gup.o
  CC [M]  fs/smb/client/dir.o
  AR      drivers/misc/cardreader/built-in.a
  AR      drivers/gpu/drm/imx/built-in.a
  CC      mm/mmap_lock.o
  CC [M]  drivers/misc/mei/hdcp/mei_hdcp.o
  CC      fs/nfs/sysfs.o
  CC [M]  fs/smb/client/file.o
  LD [M]  fs/fscache/fscache.o
  CC      drivers/tty/n_null.o
  CC      net/sunrpc/svcsock.o
  CC      arch/x86/kernel/espfix_64.o
  CC      drivers/base/regmap/regcache-maple.o
  CC      drivers/iommu/intel/irq_remapping.o
  CC      crypto/jitterentropy.o
  CC [M]  arch/x86/kvm/irq_comm.o
  CC      drivers/acpi/acpica/exstoren.o
  CC      drivers/tty/serial/serial_ctrl.o
  CC      crypto/jitterentropy-kcapi.o
  CC [M]  fs/fuse/dev.o
  CC      drivers/iommu/intel/perfmon.o
  CC      drivers/tty/pty.o
  CC      drivers/base/power/main.o
  CC      drivers/tty/serial/8250/8250_pericom.o
  CC      net/ipv4/datagram.o
  CC      lib/zstd/common/entropy_common.o
  CC [M]  drivers/gpu/drm/tests/drm_plane_helper_test.o
  CC      drivers/acpi/acpica/exstorob.o
  CC      net/sunrpc/svcauth.o
  CC      net/ethtool/pse-pd.o
  CC      drivers/tty/serial/serial_port.o
  CC      lib/zstd/common/error_private.o
  CC      drivers/tty/sysrq.o
  CC      net/sunrpc/svcauth_unix.o
  CC      drivers/base/regmap/regmap-debugfs.o
  CC      fs/ext4/namei.o
  CC      lib/zstd/common/fse_decompress.o
  CC      lib/zstd/common/zstd_common.o
  CC      net/ipv4/raw.o
  CC      drivers/tty/serial/earlycon.o
  CC      block/blk-mq-pci.o
  CC      drivers/base/regmap/regmap-i2c.o
  CC [M]  fs/fuse/dir.o
  CC      arch/x86/kernel/ksysfs.o
  CC [M]  drivers/misc/mei/pxp/mei_pxp.o
  CC      block/blk-mq-virtio.o
  CC      lib/decompress_unlzo.o
  CC      drivers/acpi/acpica/exsystem.o
  AR      drivers/tty/serial/8250/built-in.a
  CC [M]  fs/fuse/file.o
  CC      crypto/ghash-generic.o
  CC      drivers/acpi/acpica/extrace.o
  CC      net/bridge/br_mdb.o
  CC      crypto/af_alg.o
  CC      fs/btrfs/file-item.o
  CC [M]  drivers/gpu/drm/tests/drm_probe_helper_test.o
  CC [M]  arch/x86/kvm/cpuid.o
  CC      net/ipv6/datagram.o
  CC      net/bridge/br_multicast_eht.o
  CC [M]  net/netfilter/nf_conntrack_proto_udp.o
  CC      net/core/sock_diag.o
  CC      drivers/tty/serial/serial_mctrl_gpio.o
  CC [M]  drivers/gpu/drm/display/drm_hdmi_helper.o
  CC      drivers/acpi/acpica/exutils.o
  CC      arch/x86/kernel/bootflag.o
  CC      block/blk-mq-debugfs.o
  CC [M]  net/bluetooth/hci_sock.o
  AR      lib/zstd/built-in.a
  CC      fs/btrfs/inode-item.o
  AR      drivers/iommu/intel/built-in.a
  CC      arch/x86/kernel/e820.o
  CC      net/ethtool/plca.o
  CC [M]  drivers/misc/mei/init.o
  CC      drivers/acpi/acpica/hwacpi.o
  AR      drivers/iommu/arm/arm-smmu/built-in.a
  AR      drivers/iommu/iommufd/built-in.a
  CC [M]  drivers/gpu/drm/display/drm_scdc_helper.o
  CC      lib/decompress_unxz.o
  AR      drivers/iommu/arm/arm-smmu-v3/built-in.a
  CC      drivers/mfd/mfd-core.o
  AR      drivers/iommu/arm/built-in.a
  CC      drivers/iommu/iommu.o
  AR      drivers/nfc/built-in.a
  CC      block/blk-pm.o
  CC      drivers/base/regmap/regmap-irq.o
  CC      drivers/iommu/iommu-traces.o
  CC      net/core/dev_ioctl.o
  CC      net/sunrpc/addr.o
  CC [M]  drivers/gpu/drm/tests/drm_rect_test.o
  CC      drivers/acpi/acpica/hwesleep.o
  CC      fs/nfs/fs_context.o
  CC      net/sunrpc/rpcb_clnt.o
  CC [M]  fs/fuse/inode.o
  CC [M]  fs/smb/client/inode.o
  CC [M]  net/bluetooth/hci_sysfs.o
  CC [M]  net/netfilter/nf_conntrack_proto_icmp.o
  CC      lib/decompress_unzstd.o
  CC      drivers/base/power/wakeup.o
  AR      drivers/tty/serial/built-in.a
  AR      drivers/tty/built-in.a
  CC [M]  net/netfilter/nf_conntrack_extend.o
  CC      lib/dump_stack.o
  CC [M]  drivers/misc/mei/hbm.o
  CC      block/holder.o
  CC      mm/highmem.o
  CC [M]  drivers/gpu/drm/display/drm_dp_aux_dev.o
  CC      lib/earlycpio.o
  CC      drivers/mfd/intel-lpss.o
  CC      drivers/mfd/intel-lpss-pci.o
  CC      drivers/acpi/acpica/hwgpe.o
  CC [M]  fs/fuse/control.o
  CC      net/ipv4/udp.o
  CC      kernel/trace/trace_output.o
  CC      net/sunrpc/timer.o
  CC      net/core/tso.o
  AR      net/ethtool/built-in.a
  CC      mm/memory.o
  CC [M]  drivers/gpu/drm/tests/drm_exec_test.o
  CC      arch/x86/kernel/pci-dma.o
  CC      net/ipv6/ip6_flowlabel.o
  CC [M]  fs/smb/client/link.o
  CC      arch/x86/kernel/quirks.o
  CC      lib/extable.o
  CC      crypto/algif_hash.o
  CC      drivers/base/power/wakeup_stats.o
  CC      drivers/mfd/intel-lpss-acpi.o
  AR      drivers/base/regmap/built-in.a
  CC      drivers/base/bus.o
  CC      drivers/iommu/iommu-sysfs.o
  CC      arch/x86/kernel/topology.o
  AR      drivers/dax/hmem/built-in.a
  CC      drivers/dax/super.o
  CC      fs/btrfs/disk-io.o
  AR      block/built-in.a
  CC      fs/ext4/page-io.o
  CC      drivers/acpi/utils.o
  CC      drivers/acpi/acpica/hwregs.o
  CC      net/bridge/br_vlan.o
  CC      drivers/acpi/reboot.o
  CC      drivers/acpi/acpica/hwsleep.o
  CC      drivers/mfd/intel_soc_pmic_crc.o
  CC [M]  fs/overlayfs/super.o
  CC      lib/flex_proportions.o
  CC [M]  net/netfilter/nf_conntrack_acct.o
  CC [M]  fs/fuse/xattr.o
  CC [M]  net/bluetooth/l2cap_core.o
  CC      drivers/dax/bus.o
  CC      lib/idr.o
  LD [M]  drivers/gpu/drm/display/drm_display_helper.o
  CC      fs/open.o
  CC [M]  drivers/misc/mei/interrupt.o
  CC [M]  drivers/misc/mei/client.o
  CC      drivers/base/power/domain.o
  CC      net/sunrpc/xdr.o
  AR      drivers/gpu/drm/i2c/built-in.a
  CC      drivers/base/dd.o
  CC [M]  arch/x86/kvm/pmu.o
  AR      drivers/gpu/drm/panel/built-in.a
  CC [M]  arch/x86/kvm/mtrr.o
  AR      drivers/gpu/drm/bridge/analogix/built-in.a
  AR      drivers/gpu/drm/bridge/cadence/built-in.a
  AR      drivers/gpu/drm/bridge/imx/built-in.a
  AR      drivers/gpu/drm/hisilicon/built-in.a
  CC      arch/x86/kernel/kdebugfs.o
  AR      drivers/gpu/drm/mxsfb/built-in.a
  CC      kernel/bpf/core.o
  CC      arch/x86/kernel/alternative.o
  AR      drivers/gpu/drm/bridge/synopsys/built-in.a
  AR      drivers/gpu/drm/bridge/built-in.a
  CC      kernel/events/core.o
  AR      drivers/gpu/drm/tiny/built-in.a
  CC      kernel/events/ring_buffer.o
  CC      kernel/events/callchain.o
  AR      drivers/gpu/drm/xlnx/built-in.a
  CC      drivers/acpi/acpica/hwvalid.o
  CC      net/core/sock_reuseport.o
  AR      drivers/gpu/drm/gud/built-in.a
  CC      drivers/iommu/dma-iommu.o
  AR      drivers/gpu/drm/solomon/built-in.a
  CC      kernel/events/hw_breakpoint.o
  CC [M]  drivers/gpu/drm/ttm/ttm_tt.o
  CC      drivers/iommu/iova.o
  CC [M]  drivers/gpu/drm/ttm/ttm_bo.o
  CC      drivers/acpi/nvs.o
  CC [M]  drivers/misc/mei/main.o
  CC      lib/irq_regs.o
  CC      net/bridge/br_vlan_tunnel.o
  CC [M]  fs/overlayfs/namei.o
  CC [M]  fs/overlayfs/util.o
  CC      crypto/algif_skcipher.o
  CC      kernel/trace/trace_seq.o
  CC      fs/nfs/sysctl.o
  CC [M]  drivers/mfd/lpc_sch.o
  CC      kernel/fork.o
  CC [M]  fs/fuse/acl.o
  CC      lib/is_single_threaded.o
  CC      drivers/acpi/acpica/hwxface.o
  CC      fs/ext4/readpage.o
  CC      arch/x86/kernel/i8253.o
  CC      net/ipv6/inet6_connection_sock.o
  CC [M]  fs/fuse/readdir.o
  CC [M]  drivers/misc/mei/dma-ring.o
  CC [M]  net/netfilter/nf_conntrack_seqadj.o
  CC      fs/nfs/nfs2super.o
  CC      lib/klist.o
  CC      kernel/events/uprobes.o
  CC      drivers/base/syscore.o
  CC [M]  fs/overlayfs/inode.o
  CC      drivers/acpi/wakeup.o
  CC      kernel/trace/trace_stat.o
  AR      drivers/dax/built-in.a
  CC      kernel/trace/trace_printk.o
  CC      drivers/acpi/acpica/hwxfsleep.o
  CC      kernel/exec_domain.o
  CC [M]  drivers/mfd/lpc_ich.o
  CC      kernel/panic.o
  CC      kernel/cpu.o
  CC      drivers/acpi/sleep.o
  CC [M]  arch/x86/kvm/hyperv.o
  CC [M]  fs/fuse/ioctl.o
  CC [M]  drivers/gpu/drm/ttm/ttm_bo_util.o
  CC [M]  drivers/misc/mei/bus.o
  CC      drivers/dma-buf/dma-buf.o
  CC      lib/kobject.o
  CC      drivers/dma-buf/dma-fence.o
  CC      crypto/xor.o
  AR      drivers/cxl/core/built-in.a
  AR      drivers/cxl/built-in.a
  AR      drivers/macintosh/built-in.a
  CC      net/core/fib_notifier.o
  CC      kernel/exit.o
  CC [M]  arch/x86/kvm/debugfs.o
  CC      lib/kobject_uevent.o
  CC      arch/x86/kernel/hw_breakpoint.o
  CC [M]  fs/overlayfs/file.o
  CC      drivers/base/power/domain_governor.o
  CC      drivers/iommu/irq_remapping.o
  CC      drivers/acpi/acpica/hwpci.o
  CC [M]  drivers/gpu/drm/ttm/ttm_bo_vm.o
  CC      fs/ext4/resize.o
  CC      net/core/xdp.o
  CC      fs/ext4/super.o
  AR      drivers/misc/built-in.a
  CC [M]  fs/smb/client/misc.o
  CC      crypto/hash_info.o
  CC      drivers/acpi/acpica/nsaccess.o
  CC      drivers/acpi/device_sysfs.o
  CC [M]  drivers/gpu/drm/ttm/ttm_module.o
  CC      fs/nfs/proc.o
  CC      net/sunrpc/sunrpc_syms.o
  CC [M]  fs/smb/client/netmisc.o
  CC      kernel/trace/pid_list.o
  CC      net/ipv6/udp_offload.o
  CC [M]  fs/smb/client/smbencrypt.o
  CC      drivers/base/driver.o
  CC      crypto/simd.o
  CC      kernel/softirq.o
  AR      drivers/mfd/built-in.a
  LD [M]  fs/fuse/fuse.o
  CC      net/sunrpc/cache.o
  CC [M]  net/netfilter/nf_conntrack_proto_icmpv6.o
  CC      drivers/base/power/clock_ops.o
  CC      drivers/scsi/scsi.o
  CC      drivers/acpi/acpica/nsalloc.o
  CC [M]  drivers/gpu/drm/ttm/ttm_execbuf_util.o
  CC      drivers/scsi/hosts.o
  CC      fs/read_write.o
  CC      net/bridge/br_vlan_options.o
  AR      drivers/iommu/built-in.a
  CC      drivers/acpi/device_pm.o
  CC      drivers/nvme/host/core.o
  AR      drivers/nvme/target/built-in.a
  CC      net/sunrpc/rpc_pipe.o
  CC [M]  drivers/gpu/drm/ttm/ttm_range_manager.o
  CC      kernel/resource.o
  CC      net/ipv4/udplite.o
  CC      arch/x86/kernel/tsc.o
  CC [M]  fs/smb/client/transport.o
  CC [M]  drivers/misc/mei/bus-fixup.o
  CC [M]  crypto/md4.o
  CC [M]  arch/x86/kvm/mmu/mmu.o
  CC      kernel/sysctl.o
  CC [M]  drivers/misc/mei/debugfs.o
  CC [M]  fs/overlayfs/dir.o
  CC      arch/x86/kernel/tsc_msr.o
  CC [M]  crypto/ccm.o
  CC      drivers/acpi/acpica/nsarguments.o
  CC      drivers/dma-buf/dma-fence-array.o
  CC      lib/logic_pio.o
  AR      kernel/bpf/built-in.a
  CC      kernel/capability.o
  CC      kernel/trace/trace_sched_switch.o
  CC [M]  crypto/arc4.o
  AR      drivers/base/power/built-in.a
  CC      net/sunrpc/sysfs.o
  CC      drivers/base/class.o
  CC [M]  drivers/gpu/drm/ttm/ttm_resource.o
  CC      fs/btrfs/transaction.o
  CC      arch/x86/kernel/io_delay.o
  CC      drivers/scsi/scsi_ioctl.o
  CC      net/ipv6/seg6.o
  CC [M]  crypto/ecc.o
  CC [M]  crypto/essiv.o
  CC      arch/x86/kernel/rtc.o
  CC      fs/nfs/nfs2xdr.o
  CC      drivers/acpi/proc.o
  CC      drivers/acpi/acpica/nsconvert.o
  CC [M]  fs/smb/client/cached_dir.o
  CC      drivers/dma-buf/dma-fence-chain.o
  CC [M]  net/bluetooth/l2cap_sock.o
  CC [M]  drivers/misc/mei/mei-trace.o
  CC      net/core/flow_offload.o
  CC [M]  arch/x86/kvm/mmu/page_track.o
  CC [M]  net/netfilter/nf_conntrack_proto_dccp.o
  CC [M]  crypto/ecdh.o
  CC      net/ipv4/udp_offload.o
  CC      arch/x86/kernel/resource.o
  CC [M]  crypto/ecdh_helper.o
  CC      lib/maple_tree.o
  CC      drivers/acpi/bus.o
  CC [M]  drivers/gpu/drm/ttm/ttm_pool.o
  CC      drivers/acpi/glue.o
  CC      kernel/trace/trace_functions.o
  CC      kernel/ptrace.o
  CC [M]  fs/smb/client/cifs_unicode.o
  CC      drivers/acpi/scan.o
  CC      lib/memcat_p.o
  CC      fs/ext4/symlink.o
  CC      lib/nmi_backtrace.o
  CC      drivers/base/platform.o
  CC      drivers/acpi/acpica/nsdump.o
  CC      kernel/user.o
  CC      fs/nfs/nfs3super.o
  CC [M]  fs/overlayfs/readdir.o
  CC      net/bridge/br_mst.o
  CC      fs/nfs/nfs3client.o
  AS      arch/x86/kernel/irqflags.o
  CC      mm/mincore.o
  CC      arch/x86/kernel/static_call.o
  CC      fs/nfs/nfs3proc.o
  CC      drivers/dma-buf/dma-fence-unwrap.o
  CC [M]  drivers/gpu/drm/ttm/ttm_device.o
  CC      drivers/acpi/resource.o
  CC      drivers/scsi/scsicam.o
  CC      drivers/acpi/acpi_processor.o
  CC [M]  arch/x86/kvm/mmu/spte.o
  CC      kernel/trace/trace_preemptirq.o
  CC      fs/file_table.o
  CC      net/sunrpc/svc_xprt.o
  CC      drivers/acpi/acpica/nseval.o
  CC      net/ipv6/fib6_notifier.o
  CC [M]  drivers/misc/mei/pci-me.o
  CC      net/sunrpc/xprtmultipath.o
  CC [M]  arch/x86/kvm/mmu/tdp_iter.o
  CC      net/ipv4/arp.o
  CC      drivers/acpi/processor_core.o
  CC      arch/x86/kernel/process.o
  CC      drivers/acpi/processor_pdc.o
  CC [M]  net/netfilter/nf_conntrack_proto_sctp.o
  CC      mm/mlock.o
  CC      drivers/dma-buf/dma-resv.o
  CC [M]  net/netfilter/nf_conntrack_netlink.o
  CC      net/core/gro.o
  CC      kernel/signal.o
  CC      drivers/acpi/ec.o
  CC      kernel/trace/trace_nop.o
  CC      drivers/acpi/acpica/nsinit.o
  CC      drivers/scsi/scsi_error.o
  CC      drivers/nvme/host/ioctl.o
  CC      fs/nfs/nfs3xdr.o
  CC [M]  drivers/gpu/drm/ttm/ttm_sys_manager.o
  CC [M]  net/bluetooth/smp.o
  CC      drivers/dma-buf/sync_file.o
  CC [M]  drivers/gpu/drm/ttm/ttm_agp_backend.o
  CC [M]  fs/smb/client/nterr.o
  CC      drivers/base/cpu.o
  CC      drivers/nvme/host/sysfs.o
  CC      drivers/base/firmware.o
  CC      drivers/nvme/host/pr.o
  LD [M]  crypto/ecdh_generic.o
  CC      drivers/dma-buf/sw_sync.o
  CC [M]  fs/overlayfs/copy_up.o
  AR      crypto/built-in.a
  CC      net/core/netdev-genl.o
  CC      kernel/sys.o
  CC      drivers/acpi/dock.o
  CC [M]  drivers/misc/mei/hw-me.o
  CC      drivers/scsi/scsi_lib.o
  CC      fs/super.o
  CC      net/core/netdev-genl-gen.o
  CC [M]  net/bridge/br_netfilter_hooks.o
  CC      kernel/trace/trace_functions_graph.o
  CC      kernel/umh.o
  CC [M]  net/bridge/br_netfilter_ipv6.o
  CC      net/ipv6/rpl.o
  CC      drivers/scsi/scsi_lib_dma.o
  CC      drivers/acpi/acpica/nsload.o
  CC      drivers/acpi/pci_root.o
  CC      drivers/dma-buf/sync_debug.o
  CC      kernel/trace/fgraph.o
  CC [M]  fs/smb/client/cifsencrypt.o
  CC      fs/btrfs/inode.o
  CC [M]  arch/x86/kvm/mmu/tdp_mmu.o
  CC [M]  arch/x86/kvm/smm.o
  LD [M]  drivers/gpu/drm/ttm/ttm.o
  CC      arch/x86/kernel/ptrace.o
  CC [M]  net/bluetooth/lib.o
  CC      drivers/base/init.o
  CC [M]  drivers/gpu/drm/scheduler/sched_main.o
  CC      drivers/acpi/acpica/nsnames.o
  CC      kernel/trace/blktrace.o
  CC [M]  fs/smb/client/readdir.o
  CC      drivers/base/map.o
  CC [M]  net/netfilter/nf_nat_core.o
  CC      drivers/base/devres.o
  CC      drivers/acpi/pci_link.o
  CC      kernel/workqueue.o
  CC      drivers/acpi/acpica/nsobject.o
  CC [M]  drivers/dma-buf/selftest.o
  CC      drivers/nvme/host/trace.o
  CC [M]  drivers/dma-buf/st-dma-fence.o
  CC [M]  net/bluetooth/ecdh_helper.o
  CC      mm/mmap.o
  CC      net/ipv4/icmp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.o
  CC [M]  fs/overlayfs/export.o
  CC      net/ipv6/ioam6.o
  CC      net/ipv4/devinet.o
  CC      mm/mmu_gather.o
  CC      drivers/acpi/acpica/nsparse.o
  CC [M]  net/bluetooth/hci_request.o
  CC      net/core/gso.o
  CC      kernel/trace/trace_events.o
  CC      net/ipv6/sysctl_net_ipv6.o
  CC [M]  drivers/gpu/drm/amd/amdxcp/amdgpu_xcp_drv.o
  CC [M]  fs/overlayfs/params.o
  CC      net/core/net-sysfs.o
  CC      net/sunrpc/stats.o
  AR      net/bridge/built-in.a
  CC      net/sunrpc/sysctl.o
  CC [M]  drivers/dma-buf/st-dma-fence-chain.o
  CC      drivers/base/attribute_container.o
  CC      net/ipv6/xfrm6_policy.o
  CC [M]  drivers/dma-buf/st-dma-fence-unwrap.o
  CC      drivers/scsi/scsi_scan.o
  CC      arch/x86/kernel/tls.o
  CC [M]  net/bluetooth/mgmt_util.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_device.o
  CC      drivers/acpi/acpica/nspredef.o
  CC [M]  drivers/dma-buf/st-dma-resv.o
  AR      fs/nfs/built-in.a
  CC      net/core/page_pool.o
  CC [M]  net/bluetooth/mgmt_config.o
  CC [M]  drivers/misc/mei/gsc-me.o
  CC      arch/x86/kernel/step.o
  LD [M]  drivers/gpu/drm/amd/amdxcp/amdxcp.o
  CC      drivers/nvme/host/fault_inject.o
  CC      net/ipv6/xfrm6_state.o
  CC [M]  fs/smb/client/ioctl.o
  CC [M]  drivers/gpu/drm/scheduler/sched_fence.o
  CC      drivers/base/transport_class.o
  CC      net/core/net-procfs.o
  CC      fs/char_dev.o
  CC [M]  net/netfilter/nf_nat_proto.o
  CC      drivers/base/topology.o
  CC      arch/x86/kernel/i8237.o
  CC      fs/btrfs/file.o
  CC      drivers/acpi/acpica/nsprepkg.o
  AR      drivers/dma-buf/built-in.a
  CC      lib/plist.o
  LD [M]  fs/overlayfs/overlay.o
  CC      drivers/acpi/acpica/nsrepair.o
  AR      kernel/events/built-in.a
  LD [M]  net/bridge/br_netfilter.o
  CC      fs/ext4/sysfs.o
  CC      fs/stat.o
  CC      fs/exec.o
  GEN     drivers/scsi/scsi_devinfo_tbl.c
  CC      net/ipv6/xfrm6_input.o
  CC      drivers/scsi/scsi_devinfo.o
  CC      net/ipv6/xfrm6_output.o
  CC      fs/ext4/xattr.o
  CC      arch/x86/kernel/stacktrace.o
  CC      drivers/acpi/acpica/nsrepair2.o
  CC      fs/pipe.o
  CC [M]  net/netfilter/nf_nat_helper.o
  CC      drivers/scsi/scsi_sysctl.o
  LD [M]  drivers/misc/mei/mei.o
  LD [M]  drivers/dma-buf/dmabuf_selftests.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.o
  CC [M]  drivers/gpu/drm/scheduler/sched_entity.o
  CC      drivers/ata/libata-core.o
  LD [M]  drivers/misc/mei/mei-me.o
  CC [M]  net/bluetooth/hci_codec.o
  CC      drivers/ata/libata-scsi.o
  LD [M]  drivers/misc/mei/mei-gsc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.o
  CC      drivers/nvme/host/pci.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.o
  CC [M]  arch/x86/kvm/vmx/vmx.o
  CC      fs/namei.o
  CC      drivers/base/container.o
  CC      net/ipv4/af_inet.o
  CC      fs/fcntl.o
  CC      kernel/pid.o
  CC      drivers/ata/libata-eh.o
  CC      drivers/acpi/pci_irq.o
  CC      kernel/task_work.o
  CC [M]  net/bluetooth/eir.o
  CC [M]  fs/smb/client/sess.o
  CC      drivers/scsi/scsi_debugfs.o
  CC      drivers/spi/spi.o
  CC      arch/x86/kernel/reboot.o
  CC      drivers/acpi/acpica/nssearch.o
  AR      net/sunrpc/built-in.a
  CC      drivers/acpi/acpica/nsutils.o
  CC      kernel/extable.o
  CC      drivers/base/property.o
  CC      drivers/scsi/scsi_trace.o
  CC      net/ipv4/igmp.o
  CC      drivers/net/phy/mdio-boardinfo.o
  CC      net/core/netpoll.o
  CC      drivers/net/phy/stubs.o
  CC      fs/ioctl.o
  LD [M]  drivers/gpu/drm/scheduler/gpu-sched.o
  AR      drivers/firewire/built-in.a
  CC      drivers/acpi/acpica/nswalk.o
  AR      drivers/cdrom/built-in.a
  CC      net/ipv4/fib_frontend.o
  CC      kernel/params.o
  CC      kernel/trace/trace_export.o
  CC [M]  net/netfilter/nf_nat_redirect.o
  CC      net/ipv4/fib_semantics.o
  CC      net/ipv4/fib_trie.o
  AR      drivers/auxdisplay/built-in.a
  CC      net/ipv4/fib_notifier.o
  CC [M]  drivers/gpu/drm/i915/i915_driver.o
  CC      drivers/acpi/acpica/nsxfeval.o
  CC      arch/x86/kernel/msr.o
  CC      drivers/acpi/acpi_lpss.o
  CC      net/ipv6/xfrm6_protocol.o
  CC      fs/readdir.o
  CC      fs/select.o
  CC      drivers/scsi/scsi_logging.o
  CC      arch/x86/kernel/cpuid.o
  CC      lib/radix-tree.o
  CC      drivers/net/phy/mdio_devres.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_bo_test.o
  CC [M]  net/bluetooth/hci_sync.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_dma_buf_test.o
  CC      mm/mprotect.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_migrate_test.o
  CC      fs/ext4/xattr_hurd.o
  CC [M]  arch/x86/kvm/kvm-asm-offsets.s
  CC      fs/ext4/xattr_trusted.o
  CC      drivers/acpi/acpica/nsxfname.o
  CC      fs/dcache.o
  CC      drivers/net/phy/phy.o
  CC      drivers/base/cacheinfo.o
  CC      kernel/trace/trace_event_perf.o
  CC      drivers/acpi/acpica/nsxfobj.o
  CC      arch/x86/kernel/early-quirks.o
  CC      fs/inode.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_pci_test.o
  CC      net/core/fib_rules.o
  CC      lib/ratelimit.o
  CC      fs/attr.o
  CC      arch/x86/kernel/smp.o
  CC      drivers/base/swnode.o
  CC      fs/bad_inode.o
  CC      net/ipv4/inet_fragment.o
  CC [M]  net/bluetooth/coredump.o
  CC      drivers/scsi/scsi_pm.o
  CC      drivers/acpi/acpica/psargs.o
  CC      drivers/acpi/acpica/psloop.o
  CC [M]  net/netfilter/nf_nat_masquerade.o
  CC      net/ipv4/ping.o
  CC [M]  net/bluetooth/sco.o
  CC      fs/ext4/xattr_user.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_rtp_test.o
  CC      drivers/net/phy/phy-c45.o
  CC      drivers/ata/libata-transport.o
  CC      kernel/trace/trace_events_filter.o
  CC      net/ipv6/netfilter.o
  CC      fs/ext4/fast_commit.o
  CC      lib/rbtree.o
  CC [M]  net/netfilter/x_tables.o
  AR      drivers/nvme/host/built-in.a
  CC      fs/btrfs/defrag.o
  AR      drivers/nvme/built-in.a
  CC      fs/btrfs/extent_map.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/atombios_crtc.o
  CC      fs/btrfs/sysfs.o
  CC [M]  fs/smb/client/export.o
  CC      drivers/net/phy/phy-core.o
  CC      fs/file.o
  CC      fs/filesystems.o
  CC      kernel/trace/trace_events_trigger.o
  CC      net/ipv4/ip_tunnel_core.o
  CC      fs/ext4/orphan.o
  CC      drivers/acpi/acpica/psobject.o
  CC      fs/btrfs/accessors.o
  CC [M]  drivers/gpu/drm/i915/i915_drm_client.o
  CC      lib/seq_buf.o
  CC      drivers/base/auxiliary.o
  CC      drivers/scsi/scsi_bsg.o
  CC      arch/x86/kernel/smpboot.o
  CC      arch/x86/kernel/tsc_sync.o
  CC      net/ipv4/gre_offload.o
  CC      mm/mremap.o
  CC      net/ipv6/fib6_rules.o
  CC      drivers/acpi/acpi_apd.o
  CC      drivers/net/phy/phy_device.o
  CC      drivers/scsi/scsi_common.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_wa_test.o
  CC      lib/siphash.o
  CC      drivers/base/devtmpfs.o
  CC      drivers/ata/libata-trace.o
  CC      drivers/acpi/acpica/psopcode.o
  AR      drivers/spi/built-in.a
  CC [M]  net/bluetooth/iso.o
  CC      fs/namespace.o
  CC      drivers/scsi/sd.o
  CC [M]  drivers/gpu/drm/i915/i915_config.o
  CC [M]  fs/smb/client/unc.o
  CC      drivers/acpi/acpi_platform.o
  CC      fs/btrfs/xattr.o
  CC      arch/x86/kernel/setup_percpu.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_connectors.o
  CC      fs/seq_file.o
  CC      net/core/net-traces.o
  CC      fs/xattr.o
  CC      drivers/ata/libata-sata.o
  CC      fs/btrfs/ordered-data.o
  CC      mm/msync.o
  CC      lib/string.o
  CC      drivers/acpi/acpica/psopinfo.o
  CC      net/ipv4/metrics.o
  CC      drivers/usb/common/common.o
  CC      mm/page_vma_mapped.o
  CC      mm/pagewalk.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/atom.o
  CC      drivers/usb/common/debug.o
  CC      drivers/usb/core/usb.o
  CC      net/ipv6/proc.o
  CC      drivers/usb/core/hub.o
  CC [M]  drivers/gpu/drm/i915/i915_getparam.o
  CC      drivers/ata/libata-sff.o
  CC      drivers/usb/core/hcd.o
  CC      lib/timerqueue.o
  CC      drivers/acpi/acpica/psparse.o
  CC [M]  drivers/gpu/drm/xe/xe_bb.o
  CC      kernel/trace/trace_eprobe.o
  CC      drivers/base/node.o
  CC      net/ipv4/netlink.o
  CC      drivers/acpi/acpica/psscope.o
  CC      lib/vsprintf.o
  CC      net/ipv4/nexthop.o
  CC      drivers/usb/core/urb.o
  CC      lib/win_minmax.o
  CC      drivers/acpi/acpi_pnp.o
  CC      net/ipv4/udp_tunnel_stub.o
  CC      drivers/acpi/power.o
  CC      fs/libfs.o
  CC      drivers/ata/libata-pmp.o
  CC [M]  fs/smb/client/winucase.o
  CC      drivers/usb/core/message.o
  CC [M]  net/netfilter/xt_tcpudp.o
  CC [M]  drivers/gpu/drm/xe/xe_bo.o
  CC      arch/x86/kernel/ftrace.o
  CC [M]  drivers/gpu/drm/xe/xe_bo_evict.o
  CC [M]  net/netfilter/xt_mark.o
  CC      drivers/input/serio/serio.o
  AR      drivers/usb/common/built-in.a
  CC      drivers/acpi/acpica/pstree.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.o
  CC      fs/fs-writeback.o
  CC [M]  drivers/gpu/drm/xe/xe_debugfs.o
  CC [M]  fs/smb/client/smb2ops.o
  CC [M]  fs/smb/client/smb2maperror.o
  AR      fs/ext4/built-in.a
  CC      drivers/acpi/event.o
  CC      drivers/input/keyboard/atkbd.o
  AR      drivers/input/mouse/built-in.a
  CC      mm/pgtable-generic.o
  CC      drivers/acpi/evged.o
  CC      drivers/ata/libata-acpi.o
  CC      fs/btrfs/extent_io.o
  CC      drivers/ata/libata-pata-timings.o
  CC      fs/btrfs/volumes.o
  CC      drivers/input/input.o
  CC [M]  drivers/gpu/drm/xe/xe_devcoredump.o
  CC [M]  drivers/gpu/drm/i915/i915_ioctl.o
  CC      drivers/acpi/acpica/psutils.o
  CC      net/ipv6/syncookies.o
  CC      drivers/net/phy/linkmode.o
  CC      drivers/base/memory.o
  AS      arch/x86/kernel/ftrace_64.o
  CC      drivers/base/module.o
  CC      arch/x86/kernel/trace_clock.o
  CC      drivers/input/input-compat.o
  CC      drivers/base/pinctrl.o
  CC      fs/btrfs/async-thread.o
  CC      net/ipv4/sysctl_net_ipv4.o
  CC      arch/x86/kernel/trace.o
  CC      drivers/input/serio/i8042.o
  CC      net/ipv4/proc.o
  CC      arch/x86/kernel/rethook.o
  CC [M]  net/bluetooth/a2mp.o
  CC [M]  drivers/gpu/drm/i915/i915_irq.o
  AR      drivers/usb/phy/built-in.a
  CC      drivers/acpi/acpica/pswalk.o
  CC      drivers/usb/host/pci-quirks.o
  CC      drivers/acpi/acpica/psxface.o
  CC      kernel/trace/trace_kprobe.o
  CC      drivers/usb/host/ehci-hcd.o
  CC [M]  arch/x86/kvm/vmx/pmu_intel.o
  CC [M]  net/netfilter/xt_nat.o
  CC [M]  arch/x86/kvm/vmx/vmcs12.o
  CC      net/ipv4/syncookies.o
  CC      drivers/base/devcoredump.o
  CC [M]  net/bluetooth/amp.o
  CC      fs/pnode.o
  CC      drivers/base/platform-msi.o
  CC      drivers/ata/ahci.o
  CC      drivers/acpi/acpica/rsaddr.o
  CC      drivers/ata/libahci.o
  CC      mm/rmap.o
  CC      drivers/net/phy/mdio_bus.o
  CC      drivers/scsi/sg.o
  CC      drivers/acpi/acpica/rscalc.o
  CC      drivers/base/physical_location.o
  CC      drivers/usb/core/driver.o
  CC      arch/x86/kernel/crash_core_64.o
  CC      drivers/ata/ata_piix.o
  CC      drivers/acpi/acpica/rscreate.o
  CC [M]  drivers/gpu/drm/i915/i915_mitigations.o
  AR      drivers/net/pse-pd/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.o
  CC      mm/vmalloc.o
  AR      drivers/input/keyboard/built-in.a
  CC      mm/page_alloc.o
  CC      drivers/input/input-mt.o
  CC      drivers/net/phy/mdio_device.o
  CC [M]  drivers/gpu/drm/xe/xe_device.o
  CC      drivers/net/phy/swphy.o
  CC      net/ipv6/mip6.o
  CC      net/ipv6/addrconf_core.o
  CC      fs/btrfs/ioctl.o
  CC [M]  arch/x86/kvm/vmx/hyperv.o
  CC      drivers/acpi/acpica/rsdumpinfo.o
  CC      drivers/usb/host/ehci-pci.o
  CC [M]  fs/smb/client/smb2transport.o
  CC      drivers/base/trace.o
  CC      arch/x86/kernel/module.o
  CC [M]  fs/smb/client/smb2misc.o
  CC      drivers/input/serio/libps2.o
  CC      fs/splice.o
  CC [M]  net/netfilter/xt_REDIRECT.o
  CC [M]  net/netfilter/xt_MASQUERADE.o
  CC [M]  drivers/gpu/drm/xe/xe_device_sysfs.o
  CC      drivers/net/phy/fixed_phy.o
  CC      lib/xarray.o
  CC      fs/sync.o
  CC [M]  net/bluetooth/hci_debugfs.o
  CC      drivers/acpi/acpica/rsinfo.o
  CC      drivers/input/input-poller.o
  CC [M]  arch/x86/kvm/vmx/nested.o
  CC      drivers/acpi/acpica/rsio.o
  CC [M]  drivers/gpu/drm/xe/xe_dma_buf.o
  CC      fs/btrfs/locking.o
  CC      mm/init-mm.o
  CC      arch/x86/kernel/early_printk.o
  CC [M]  arch/x86/kvm/vmx/posted_intr.o
  CC [M]  drivers/gpu/drm/i915/i915_module.o
  CC      net/ipv6/exthdrs_core.o
  CC      net/ipv6/ip6_checksum.o
  CC      fs/btrfs/orphan.o
  CC      lib/lockref.o
  CC      kernel/kthread.o
  AR      drivers/base/built-in.a
  CC      drivers/usb/core/config.o
  CC      drivers/net/mdio/acpi_mdio.o
  AR      drivers/net/pcs/built-in.a
  CC      drivers/usb/host/ohci-hcd.o
  CC      net/ipv4/esp4.o
  CC      drivers/net/mdio/fwnode_mdio.o
  CC      drivers/acpi/acpica/rsirq.o
  CC      drivers/usb/host/ohci-pci.o
  CC      drivers/usb/core/file.o
  AR      drivers/input/serio/built-in.a
  CC [M]  fs/smb/client/smb2pdu.o
  CC      drivers/rtc/lib.o
  CC      arch/x86/kernel/hpet.o
  CC      drivers/rtc/class.o
  CC      kernel/trace/error_report-traces.o
  CC      drivers/input/ff-core.o
  CC      drivers/rtc/interface.o
  CC      drivers/scsi/scsi_sysfs.o
  CC      drivers/input/touchscreen.o
  CC      net/ipv4/esp4_offload.o
  AR      drivers/ata/built-in.a
  CC      drivers/input/ff-memless.o
  AR      drivers/i2c/algos/built-in.a
  CC [M]  drivers/i2c/algos/i2c-algo-bit.o
  CC [M]  drivers/net/phy/phylink.o
  CC [M]  net/netfilter/xt_addrtype.o
  CC      drivers/acpi/acpica/rslist.o
  CC [M]  net/netfilter/xt_conntrack.o
  CC [M]  drivers/gpu/drm/i915/i915_params.o
  CC [M]  drivers/gpu/drm/xe/xe_drm_client.o
  CC      mm/memblock.o
  CC      drivers/usb/core/buffer.o
  CC      mm/memory_hotplug.o
  CC      net/core/selftests.o
  CC [M]  net/netfilter/xt_ipvs.o
  CC      fs/utimes.o
  CC      drivers/input/vivaldi-fmap.o
  CC [M]  drivers/gpu/drm/xe/xe_exec.o
  AR      drivers/net/mdio/built-in.a
  CC      drivers/input/input-leds.o
  CC      drivers/rtc/nvmem.o
  CC      drivers/input/mousedev.o
  CC [M]  fs/smb/client/smb2inode.o
  CC      kernel/trace/power-traces.o
  CC      drivers/input/evdev.o
  CC      drivers/rtc/dev.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_object.o
  CC      mm/madvise.o
  CC      drivers/acpi/acpica/rsmemory.o
  CC      mm/page_io.o
  CC      net/ipv6/ip6_icmp.o
  CC [M]  drivers/gpu/drm/xe/xe_execlist.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.o
  CC      net/ipv4/netfilter.o
  CC      lib/bcd.o
  LD [M]  net/bluetooth/bluetooth.o
  CC      lib/sort.o
  CC      net/ipv4/inet_diag.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_encoders.o
  CC      arch/x86/kernel/amd_nb.o
  CC [M]  drivers/gpu/drm/vgem/vgem_drv.o
  CC      lib/parser.o
  CC [M]  drivers/gpu/drm/vgem/vgem_fence.o
  CC      drivers/i2c/busses/i2c-designware-common.o
  CC      lib/debug_locks.o
  CC      drivers/acpi/acpica/rsmisc.o
  CC      drivers/usb/core/sysfs.o
  CC      drivers/i2c/busses/i2c-designware-master.o
  CC      drivers/i2c/busses/i2c-designware-platdrv.o
  LD [M]  arch/x86/kvm/kvm.o
  CC [M]  drivers/gpu/drm/xe/xe_exec_queue.o
  CC      drivers/acpi/sysfs.o
  CC      mm/swap_state.o
  CC      arch/x86/kernel/kvm.o
  CC      fs/d_path.o
  CC      lib/random32.o
  CC [M]  drivers/gpu/drm/i915/i915_pci.o
  AR      drivers/scsi/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_force_wake.o
  AR      drivers/i3c/built-in.a
  CC      drivers/rtc/proc.o
  CC      drivers/usb/storage/scsiglue.o
  CC      drivers/rtc/sysfs.o
  CC      drivers/usb/serial/usb-serial.o
  CC      drivers/rtc/rtc-mc146818-lib.o
  CC      kernel/sys_ni.o
  AR      drivers/usb/misc/built-in.a
  CC      kernel/nsproxy.o
  LD [M]  net/netfilter/nf_conntrack.o
  CC      arch/x86/kernel/kvmclock.o
  LD [M]  net/netfilter/nf_nat.o
  CC      drivers/acpi/acpica/rsserial.o
  CC      arch/x86/kernel/paravirt.o
  CC      net/ipv6/output_core.o
  AR      net/netfilter/built-in.a
  CC      drivers/i2c/busses/i2c-designware-baytrail.o
  LD [M]  drivers/gpu/drm/vgem/vgem.o
  AR      drivers/input/built-in.a
  CC      drivers/rtc/rtc-cmos.o
  CC      net/core/ptp_classifier.o
  AR      drivers/media/i2c/built-in.a
  AR      drivers/media/tuners/built-in.a
  AR      drivers/net/usb/built-in.a
  AR      drivers/net/ethernet/adi/built-in.a
  CC [M]  drivers/net/usb/pegasus.o
  AR      drivers/media/rc/keymaps/built-in.a
  AR      drivers/net/ethernet/alacritech/built-in.a
  CC      lib/bust_spinlocks.o
  AR      drivers/media/rc/built-in.a
  AR      drivers/net/ethernet/amazon/built-in.a
  CC [M]  drivers/net/usb/rtl8150.o
  CC [M]  drivers/net/usb/r8152.o
  AR      drivers/net/ethernet/aquantia/built-in.a
  AR      drivers/media/common/b2c2/built-in.a
  AR      drivers/net/ethernet/asix/built-in.a
  AR      drivers/media/common/saa7146/built-in.a
  AR      drivers/net/ethernet/cadence/built-in.a
  CC      drivers/usb/core/endpoint.o
  AR      drivers/media/common/siano/built-in.a
  AR      drivers/net/ethernet/broadcom/built-in.a
  CC      drivers/usb/core/devio.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_display.o
  CC [M]  drivers/net/ethernet/broadcom/b44.o
  AR      drivers/media/common/v4l2-tpg/built-in.a
  AR      drivers/media/common/videobuf2/built-in.a
  AR      drivers/media/common/built-in.a
  AR      drivers/media/platform/allegro-dvt/built-in.a
  CC [M]  drivers/net/ethernet/broadcom/bnx2.o
  AR      drivers/net/ethernet/cavium/common/built-in.a
  AR      drivers/media/platform/amlogic/meson-ge2d/built-in.a
  CC      drivers/usb/core/notify.o
  AR      drivers/media/platform/amlogic/built-in.a
  AR      drivers/net/ethernet/cavium/thunder/built-in.a
  AR      drivers/net/ethernet/cavium/liquidio/built-in.a
  AR      drivers/media/platform/amphion/built-in.a
  AR      drivers/media/platform/aspeed/built-in.a
  AR      drivers/net/ethernet/cavium/octeon/built-in.a
  AR      drivers/media/platform/atmel/built-in.a
  AR      drivers/net/ethernet/cavium/built-in.a
  CC      fs/btrfs/export.o
  AR      drivers/media/platform/cadence/built-in.a
  AR      drivers/media/platform/chips-media/built-in.a
  AR      drivers/media/platform/intel/built-in.a
  CC      drivers/usb/core/generic.o
  CC [M]  drivers/i2c/busses/i2c-scmi.o
  CC      net/core/netprio_cgroup.o
  AR      drivers/media/platform/marvell/built-in.a
  CC      mm/swapfile.o
  AR      drivers/media/pci/ttpci/built-in.a
  CC      drivers/acpi/acpica/rsutils.o
  AR      drivers/media/pci/b2c2/built-in.a
  AR      drivers/media/platform/mediatek/jpeg/built-in.a
  AR      drivers/media/pci/pluto2/built-in.a
  AR      drivers/media/platform/mediatek/mdp/built-in.a
  CC      drivers/usb/host/uhci-hcd.o
  AR      drivers/media/pci/dm1105/built-in.a
  AR      drivers/media/platform/mediatek/vcodec/common/built-in.a
  AR      drivers/media/pci/pt1/built-in.a
  CC      kernel/trace/rpm-traces.o
  CC      drivers/acpi/acpica/rsxface.o
  AR      drivers/media/platform/mediatek/vcodec/encoder/built-in.a
  AR      drivers/media/pci/pt3/built-in.a
  CC      lib/kasprintf.o
  CC      drivers/usb/core/quirks.o
  AR      drivers/media/platform/mediatek/vcodec/decoder/built-in.a
  AR      drivers/media/pci/mantis/built-in.a
  AR      drivers/media/platform/mediatek/vcodec/built-in.a
  CC      drivers/usb/host/xhci.o
  AR      drivers/media/pci/ngene/built-in.a
  AR      drivers/media/pci/ddbridge/built-in.a
  CC      kernel/notifier.o
  AR      drivers/media/platform/mediatek/vpu/built-in.a
  AR      drivers/media/pci/saa7146/built-in.a
  AR      drivers/media/platform/mediatek/mdp3/built-in.a
  AR      drivers/media/platform/mediatek/built-in.a
  AR      drivers/media/pci/smipcie/built-in.a
  AR      drivers/media/pci/netup_unidvb/built-in.a
  CC      fs/btrfs/tree-log.o
  CC [M]  drivers/i2c/busses/i2c-ccgx-ucsi.o
  AR      drivers/media/platform/microchip/built-in.a
  AR      drivers/media/pci/intel/ipu3/built-in.a
  AR      drivers/media/platform/nvidia/tegra-vde/built-in.a
  AR      drivers/media/pci/intel/ivsc/built-in.a
  AR      drivers/media/platform/nvidia/built-in.a
  AR      drivers/media/pci/intel/built-in.a
  CC      lib/bitmap.o
  AR      drivers/media/pci/built-in.a
  CC [M]  drivers/net/phy/aquantia_main.o
  AR      drivers/ptp/built-in.a
  AR      drivers/media/platform/nxp/dw100/built-in.a
  CC      drivers/usb/storage/protocol.o
  CC [M]  drivers/ptp/ptp_clock.o
  AR      drivers/media/platform/nxp/imx-jpeg/built-in.a
  CC      drivers/usb/storage/transport.o
  AR      drivers/media/platform/nxp/imx8-isi/built-in.a
  CC      drivers/usb/core/devices.o
  CC [M]  drivers/i2c/busses/i2c-i801.o
  AR      drivers/media/platform/nxp/built-in.a
  CC      drivers/usb/storage/usb.o
  AR      drivers/media/platform/qcom/camss/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_ggtt.o
  AR      drivers/media/platform/qcom/venus/built-in.a
  CC [M]  drivers/gpu/drm/i915/i915_scatterlist.o
  AR      drivers/media/platform/qcom/built-in.a
  AR      drivers/media/platform/renesas/rcar-vin/built-in.a
  AR      drivers/media/platform/renesas/rzg2l-cru/built-in.a
  AR      drivers/media/platform/renesas/vsp1/built-in.a
  CC      fs/stack.o
  AR      drivers/media/platform/renesas/built-in.a
  CC [M]  drivers/net/ipvlan/ipvlan_core.o
  AR      drivers/media/platform/rockchip/rga/built-in.a
  CC [M]  drivers/net/ipvlan/ipvlan_main.o
  AR      drivers/media/platform/rockchip/rkisp1/built-in.a
  AR      drivers/media/platform/samsung/exynos-gsc/built-in.a
  CC      fs/fs_struct.o
  AR      drivers/media/platform/rockchip/built-in.a
  CC      arch/x86/kernel/pvclock.o
  AR      drivers/media/platform/samsung/exynos4-is/built-in.a
  CC      fs/statfs.o
  AR      drivers/media/platform/samsung/s3c-camif/built-in.a
  AR      drivers/media/platform/samsung/s5p-g2d/built-in.a
  CC      drivers/usb/storage/initializers.o
  AR      drivers/media/platform/samsung/s5p-jpeg/built-in.a
  CC      arch/x86/kernel/pcspeaker.o
  AR      drivers/media/platform/samsung/s5p-mfc/built-in.a
  CC      drivers/acpi/acpica/tbdata.o
  AR      drivers/media/platform/samsung/built-in.a
  CC      net/ipv6/protocol.o
  CC      net/ipv4/tcp_diag.o
  CC      net/ipv4/udp_diag.o
  AR      drivers/media/platform/st/sti/bdisp/built-in.a
  AR      drivers/rtc/built-in.a
  CC      fs/fs_pin.o
  CC      net/ipv4/tcp_cubic.o
  CC      fs/nsfs.o
  CC      kernel/trace/trace_dynevent.o
  AR      drivers/media/platform/st/sti/c8sectpfe/built-in.a
  AR      drivers/media/platform/st/sti/delta/built-in.a
  AR      drivers/media/platform/st/sti/hva/built-in.a
  AR      drivers/media/platform/st/stm32/built-in.a
  AR      drivers/media/platform/st/built-in.a
  AR      drivers/media/platform/sunxi/sun4i-csi/built-in.a
  AR      drivers/media/platform/sunxi/sun6i-csi/built-in.a
  AR      drivers/media/platform/sunxi/sun6i-mipi-csi2/built-in.a
  CC      fs/fs_types.o
  AR      drivers/media/platform/sunxi/sun8i-a83t-mipi-csi2/built-in.a
  CC      drivers/usb/serial/generic.o
  AR      drivers/media/platform/sunxi/sun8i-di/built-in.a
  CC [M]  drivers/net/vxlan/vxlan_core.o
  AR      drivers/media/platform/sunxi/sun8i-rotate/built-in.a
  AR      drivers/media/platform/sunxi/built-in.a
  AR      drivers/power/reset/built-in.a
  AR      drivers/media/platform/ti/am437x/built-in.a
  AR      drivers/media/platform/ti/cal/built-in.a
  CC      drivers/power/supply/power_supply_core.o
  AR      drivers/media/platform/ti/vpe/built-in.a
  AR      drivers/media/platform/ti/davinci/built-in.a
  CC      drivers/hwmon/hwmon.o
  CC      fs/fs_context.o
  UPD     arch/x86/kvm/kvm-asm-offsets.h
  AR      drivers/media/platform/ti/omap/built-in.a
  CC      net/ipv4/xfrm4_policy.o
  CC      fs/fs_parser.o
  AR      drivers/media/platform/ti/omap3isp/built-in.a
  AR      drivers/media/platform/ti/built-in.a
  AR      drivers/media/platform/verisilicon/built-in.a
  AR      drivers/media/platform/via/built-in.a
  CC      fs/fsopen.o
  AR      drivers/media/platform/xilinx/built-in.a
  AR      drivers/media/platform/built-in.a
  CC      net/core/dst_cache.o
  CC      drivers/acpi/acpica/tbfadt.o
  CC      lib/scatterlist.o
  CC      lib/list_sort.o
  CC      kernel/trace/trace_probe.o
  AR      drivers/media/usb/b2c2/built-in.a
  CC      arch/x86/kernel/check.o
  CC [M]  drivers/ptp/ptp_chardev.o
  AR      drivers/media/usb/dvb-usb/built-in.a
  CC [M]  drivers/net/phy/aquantia_hwmon.o
  AR      drivers/media/usb/dvb-usb-v2/built-in.a
  CC [M]  drivers/net/phy/ax88796b.o
  CC      arch/x86/kernel/uprobes.o
  AR      drivers/media/usb/s2255/built-in.a
  AR      drivers/media/usb/siano/built-in.a
  AR      drivers/media/usb/ttusb-budget/built-in.a
  CC      kernel/trace/trace_uprobe.o
  AR      drivers/media/usb/ttusb-dec/built-in.a
  CC      kernel/trace/rethook.o
  CC      drivers/acpi/property.o
  AR      drivers/media/usb/built-in.a
  CC      drivers/acpi/acpi_cmos_rtc.o
  AR      drivers/media/mmc/siano/built-in.a
  AR      drivers/media/firewire/built-in.a
  AR      drivers/media/mmc/built-in.a
  CC      net/core/gro_cells.o
  CC      drivers/usb/storage/sierra_ms.o
  AR      drivers/media/spi/built-in.a
  CC      drivers/acpi/x86/apple.o
  CC      drivers/usb/storage/option_ms.o
  AR      drivers/media/test-drivers/built-in.a
  AR      drivers/media/built-in.a
  CC      lib/uuid.o
  CC      net/ipv4/xfrm4_state.o
  CC [M]  drivers/gpu/drm/i915/i915_suspend.o
  CC [M]  drivers/gpu/drm/xe/xe_gt.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_clock.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.o
  CC      net/ipv6/ip6_offload.o
  CC [M]  drivers/net/ethernet/broadcom/cnic.o
  CC      drivers/acpi/acpica/tbfind.o
  CC [M]  drivers/i2c/busses/i2c-isch.o
  CC      fs/init.o
  CC      net/ipv4/xfrm4_input.o
  CC      drivers/usb/gadget/udc/core.o
  CC      drivers/usb/serial/bus.o
  CC      drivers/usb/gadget/udc/trace.o
  CC      arch/x86/kernel/perf_regs.o
  CC      drivers/usb/core/phy.o
  CC      drivers/power/supply/power_supply_sysfs.o
  AR      drivers/usb/gadget/function/built-in.a
  AR      drivers/net/ethernet/cortina/built-in.a
  CC      drivers/acpi/x86/utils.o
  CC      drivers/usb/serial/console.o
  CC      drivers/net/loopback.o
  CC [M]  drivers/net/ipvlan/ipvlan_l3s.o
  AR      drivers/net/ethernet/engleder/built-in.a
  CC      fs/kernel_read_file.o
  CC      drivers/acpi/x86/s2idle.o
  CC      net/ipv6/tcpv6_offload.o
  AR      drivers/thermal/broadcom/built-in.a
  AR      drivers/thermal/samsung/built-in.a
  CC [M]  drivers/net/phy/bcm7xxx.o
  CC      drivers/thermal/intel/intel_tcc.o
  CC [M]  fs/smb/client/smb2file.o
  CC      fs/mnt_idmapping.o
  CC [M]  drivers/ptp/ptp_sysfs.o
  CC      drivers/acpi/acpica/tbinstal.o
  CC      drivers/watchdog/watchdog_core.o
  CC      drivers/usb/storage/usual-tables.o
  CC      drivers/watchdog/watchdog_dev.o
  AR      drivers/usb/gadget/legacy/built-in.a
  CC      fs/btrfs/free-space-cache.o
  CC      drivers/usb/gadget/usbstring.o
  CC [M]  drivers/hwmon/acpi_power_meter.o
  CC      drivers/thermal/intel/therm_throt.o
  AR      net/core/built-in.a
  CC      drivers/watchdog/softdog.o
  CC      lib/iov_iter.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_debugfs.o
  CC      drivers/usb/host/xhci-mem.o
  CC      drivers/acpi/debugfs.o
  CC      net/ipv4/xfrm4_output.o
  AS [M]  arch/x86/kvm/vmx/vmenter.o
  CC      net/ipv4/xfrm4_protocol.o
  LD [M]  arch/x86/kvm/kvm-intel.o
  CC      fs/btrfs/zlib.o
  CC      drivers/usb/gadget/config.o
  CC      arch/x86/kernel/tracepoint.o
  CC      drivers/power/supply/power_supply_leds.o
  CC      arch/x86/kernel/itmt.o
  CC [M]  drivers/gpu/drm/i915/i915_switcheroo.o
  CC [M]  drivers/gpu/drm/i915/i915_sysfs.o
  CC [M]  drivers/i2c/busses/i2c-ismt.o
  CC      drivers/acpi/acpica/tbprint.o
  CC      fs/btrfs/lzo.o
  CC      drivers/usb/core/port.o
  CC      drivers/acpi/acpi_lpat.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.o
  AR      drivers/usb/storage/built-in.a
  CC      arch/x86/kernel/umip.o
  CC      drivers/power/supply/power_supply_hwmon.o
  CC      mm/swap_slots.o
  CC      drivers/usb/serial/ftdi_sio.o
  CC      arch/x86/kernel/unwind_orc.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/object.o
  CC [M]  net/ipv4/ip_tunnel.o
  AR      drivers/net/ethernet/ezchip/built-in.a
  CC      drivers/usb/serial/pl2303.o
  CC      drivers/usb/gadget/epautoconf.o
  CC [M]  drivers/ptp/ptp_vclock.o
  CC [M]  drivers/net/phy/bcm87xx.o
  CC      drivers/usb/gadget/composite.o
  CC      mm/dmapool.o
  CC      drivers/acpi/acpi_lpit.o
  CC      drivers/net/netconsole.o
  CC [M]  drivers/net/dummy.o
  CC [M]  drivers/hwmon/coretemp.o
  LD [M]  drivers/net/ipvlan/ipvlan.o
  CC      drivers/acpi/acpica/tbutils.o
  CC      drivers/acpi/acpica/tbxface.o
  CC      drivers/acpi/prmt.o
  AR      drivers/watchdog/built-in.a
  CC      arch/x86/kernel/callthunks.o
  AR      drivers/hwmon/built-in.a
  CC      arch/x86/kernel/cet.o
  CC      net/ipv6/exthdrs_offload.o
  AR      kernel/trace/built-in.a
  CC      kernel/ksysfs.o
  CC [M]  drivers/thermal/intel/x86_pkg_temp_thermal.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/client.o
  CC      drivers/usb/gadget/functions.o
  AR      drivers/usb/gadget/udc/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_gt_idle_sysfs.o
  CC      arch/x86/kernel/mmconf-fam10h_64.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_mcr.o
  CC      kernel/cred.o
  AR      drivers/power/supply/built-in.a
  AR      drivers/power/built-in.a
  CC [M]  drivers/i2c/busses/i2c-piix4.o
  CC      arch/x86/kernel/vsmp_64.o
  AR      drivers/net/ethernet/fungible/built-in.a
  CC [M]  fs/smb/client/cifsacl.o
  CC [M]  fs/smb/client/fs_context.o
  CC [M]  fs/smb/client/dns_resolve.o
  CC      drivers/usb/core/hcd-pci.o
  CC      mm/hugetlb.o
  CC [M]  drivers/i2c/busses/i2c-designware-pcidrv.o
  CC [M]  net/ipv4/udp_tunnel_core.o
  CC      drivers/acpi/acpica/tbxfload.o
  CC      drivers/usb/core/usb-acpi.o
  CC      drivers/acpi/acpica/tbxfroot.o
  CC      drivers/acpi/acpica/utaddress.o
  CC [M]  drivers/gpu/drm/i915/i915_utils.o
  CC      drivers/usb/gadget/configfs.o
  CC [M]  drivers/net/macvlan.o
  CC [M]  drivers/net/phy/bcm-phy-lib.o
  CC [M]  drivers/ptp/ptp_kvm_x86.o
  CC      fs/btrfs/zstd.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/conn.o
  CC      kernel/reboot.o
  CC      kernel/async.o
  CC      drivers/usb/gadget/u_f.o
  CC      drivers/acpi/acpi_pcc.o
  CC [M]  drivers/net/mii.o
  CC      drivers/acpi/ac.o
  CC      kernel/range.o
  CC [M]  drivers/ptp/ptp_kvm_common.o
  CC      mm/hugetlb_vmemmap.o
  AR      arch/x86/kernel/built-in.a
  LD [M]  drivers/ptp/ptp.o
  AR      arch/x86/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.o
  AR      drivers/thermal/intel/built-in.a
  CC [M]  drivers/gpu/drm/i915/intel_clock_gating.o
  AR      drivers/thermal/st/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/device.o
  AR      drivers/thermal/qcom/built-in.a
  CC [M]  drivers/gpu/drm/nouveau/nvif/disp.o
  AR      drivers/usb/serial/built-in.a
  AR      drivers/thermal/tegra/built-in.a
  CC [M]  drivers/net/mdio.o
  AR      drivers/thermal/mediatek/built-in.a
  CC      drivers/md/md.o
  CC      drivers/thermal/thermal_core.o
  CC      net/ipv6/inet6_hashtables.o
  CC [M]  drivers/md/persistent-data/dm-array.o
  CC      drivers/thermal/thermal_sysfs.o
  CC      drivers/acpi/acpica/utalloc.o
  CC [M]  drivers/md/persistent-data/dm-bitset.o
  CC      kernel/smpboot.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_pagefault.o
  CC      drivers/usb/host/xhci-ext-caps.o
  CC      drivers/usb/host/xhci-ring.o
  CC      drivers/thermal/thermal_trip.o
  CC      drivers/usb/host/xhci-hub.o
  CC      kernel/ucount.o
  AR      drivers/usb/core/built-in.a
  CC [M]  drivers/gpu/drm/i915/intel_device_info.o
  CC      kernel/regset.o
  CC      drivers/acpi/button.o
  CC      drivers/acpi/fan_core.o
  LD [M]  drivers/i2c/busses/i2c-designware-pci.o
  CC [M]  drivers/net/usb/asix_devices.o
  AR      drivers/i2c/busses/built-in.a
  CC [M]  drivers/net/usb/asix_common.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/driver.o
  LD [M]  drivers/ptp/ptp_kvm.o
  AR      drivers/i2c/muxes/built-in.a
  CC [M]  drivers/i2c/muxes/i2c-mux-gpio.o
  CC [M]  drivers/gpu/drm/i915/intel_memory_region.o
  CC      drivers/acpi/acpica/utascii.o
  ASN.1   fs/smb/client/cifs_spnego_negtokeninit.asn1.[ch]
  CC [M]  fs/smb/client/namespace.o
  CC      fs/remap_range.o
  CC      lib/clz_ctz.o
  CC [M]  drivers/md/persistent-data/dm-block-manager.o
  CC      drivers/usb/host/xhci-dbg.o
  CC      net/ipv6/mcast_snoop.o
  CC [M]  drivers/net/phy/broadcom.o
  CC      mm/mempolicy.o
  CC [M]  drivers/net/phy/lxt.o
  CC      drivers/acpi/acpica/utbuffer.o
  CC      lib/bsearch.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/event.o
  CC      mm/sparse.o
  CC      kernel/ksyms_common.o
  CC [M]  drivers/md/persistent-data/dm-space-map-common.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/fifo.o
  CC [M]  drivers/net/tun.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/head.o
  CC [M]  drivers/gpu/drm/i915/intel_pcode.o
  CC [M]  drivers/net/usb/ax88172a.o
  AR      drivers/usb/gadget/built-in.a
  CC      kernel/groups.o
  CC [M]  drivers/usb/class/usbtmc.o
  CC [M]  net/ipv4/udp_tunnel_nic.o
  CC      drivers/acpi/acpica/utcksum.o
  CC      drivers/usb/host/xhci-trace.o
  CC      fs/btrfs/compression.o
  CC [M]  drivers/net/vxlan/vxlan_multicast.o
  CC      drivers/thermal/thermal_helpers.o
  AR      net/ipv4/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_bios.o
  CC      drivers/opp/core.o
  CC      drivers/acpi/fan_attr.o
  CC [M]  drivers/gpu/drm/i915/intel_region_ttm.o
  CC      fs/btrfs/delayed-ref.o
  CC      lib/find_bit.o
  CC      drivers/thermal/thermal_hwmon.o
  CC [M]  drivers/net/vxlan/vxlan_vnifilter.o
  CC      drivers/acpi/processor_driver.o
  CC [M]  net/ipv6/ip6_udp_tunnel.o
  CC      drivers/i2c/i2c-boardinfo.o
  CC      lib/llist.o
  CC [M]  drivers/net/phy/realtek.o
  CC      drivers/acpi/acpica/utcopy.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_sysfs.o
  CC      kernel/vhost_task.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_tlb_invalidation.o
  CC [M]  drivers/net/veth.o
  CC      drivers/opp/cpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/mem.o
  AR      drivers/net/ethernet/huawei/built-in.a
  CC      drivers/acpi/processor_thermal.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/mmu.o
  CC      lib/memweight.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/outp.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/timer.o
  CC [M]  drivers/net/phy/smsc.o
  CC [M]  drivers/net/usb/ax88179_178a.o
  CC [M]  drivers/md/persistent-data/dm-space-map-disk.o
  CC      lib/kfifo.o
  CC [M]  drivers/gpu/drm/i915/intel_runtime_pm.o
  CC      fs/buffer.o
  CC      drivers/opp/debugfs.o
  CC [M]  drivers/net/ethernet/intel/e1000/e1000_main.o
  CC [M]  fs/smb/client/smb1ops.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.o
  CC [M]  drivers/net/ethernet/broadcom/tg3.o
  CC [M]  drivers/net/usb/cdc_ether.o
  CC [M]  drivers/net/ethernet/intel/e1000e/82571.o
  CC [M]  drivers/net/ethernet/intel/igb/igb_main.o
  CC      drivers/thermal/gov_fair_share.o
  CC [M]  drivers/net/ethernet/intel/igb/igb_ethtool.o
  CC      kernel/kcmp.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_main.o
  CC      drivers/acpi/acpica/utexcep.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_mac.o
  AR      drivers/net/ethernet/i825xx/built-in.a
  AR      drivers/net/ethernet/microsoft/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_gt_topology.o
  CC      drivers/i2c/i2c-core-base.o
  HOSTCC  drivers/gpu/drm/xe/xe_gen_wa_oob
  CC      drivers/acpi/acpica/utdebug.o
  CC      drivers/i2c/i2c-core-smbus.o
  CC [M]  drivers/net/vxlan/vxlan_mdb.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/atombios_dp.o
  CC      kernel/freezer.o
  CC      drivers/thermal/gov_step_wise.o
  CC      mm/sparse-vmemmap.o
  CC [M]  drivers/net/ethernet/intel/igbvf/vf.o
  CC [M]  drivers/md/persistent-data/dm-space-map-metadata.o
  AR      net/ipv6/built-in.a
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_main.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_common.o
  CC      drivers/i2c/i2c-core-acpi.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_82599.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_i225.o
  LD [M]  net/ipv4/udp_tunnel.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_82598.o
  AR      net/built-in.a
  CC      lib/percpu-refcount.o
  CC      drivers/thermal/gov_user_space.o
  CC      fs/btrfs/relocation.o
  CC      drivers/acpi/acpica/utdecode.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_ads.o
  LD [M]  drivers/net/phy/aquantia.o
  AR      drivers/net/phy/built-in.a
  CC      fs/mpage.o
  CC [M]  drivers/net/ethernet/intel/igbvf/mbx.o
  AR      drivers/opp/built-in.a
  AR      drivers/net/ethernet/litex/built-in.a
  CC      drivers/cpufreq/cpufreq.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/vmm.o
  AR      drivers/net/ethernet/microchip/built-in.a
  CC      drivers/cpufreq/freq_table.o
  CC [M]  drivers/net/ethernet/intel/igbvf/ethtool.o
  CC      drivers/cpufreq/cpufreq_performance.o
  CC [M]  drivers/net/ethernet/intel/igbvf/netdev.o
  CC [M]  drivers/gpu/drm/i915/intel_sbi.o
  CC [M]  drivers/gpu/drm/i915/intel_step.o
  CC [M]  drivers/net/ethernet/intel/e1000/e1000_hw.o
  CC [M]  drivers/net/usb/cdc_eem.o
  CC [M]  drivers/net/usb/smsc75xx.o
  AR      drivers/thermal/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_afmt.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_ct.o
  CC      drivers/acpi/acpica/utdelete.o
  CC      kernel/stacktrace.o
  CC [M]  drivers/net/ethernet/intel/e1000e/ich8lan.o
  CC      mm/mmu_notifier.o
  CC [M]  drivers/md/persistent-data/dm-transaction-manager.o
  CC      lib/rhashtable.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_hwconfig.o
  CC      drivers/usb/host/xhci-debugfs.o
  CC      fs/btrfs/delayed-inode.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_base.o
  CC      drivers/acpi/acpica/uterror.o
  CC [M]  drivers/md/persistent-data/dm-btree.o
  CC [M]  drivers/md/persistent-data/dm-btree-remove.o
  CC      drivers/cpuidle/governors/menu.o
  CC [M]  fs/smb/client/cifssmb.o
  CC      drivers/cpuidle/cpuidle.o
  CC      drivers/md/md-bitmap.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/user.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_phy.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.o
  CC      drivers/acpi/acpica/uteval.o
  CC      kernel/dma.o
  CC [M]  drivers/net/ethernet/intel/e1000/e1000_ethtool.o
  CC [M]  drivers/net/usb/smsc95xx.o
  CC [M]  drivers/net/ethernet/intel/e1000/e1000_param.o
  CC [M]  drivers/gpu/drm/i915/intel_uncore.o
  CC      drivers/i2c/i2c-core-slave.o
  CC [M]  drivers/md/persistent-data/dm-btree-spine.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_trace_points.o
  CC      drivers/cpuidle/governors/haltpoll.o
  CC [M]  drivers/gpu/drm/i915/intel_wakeref.o
  CC      mm/ksm.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_log.o
  CC      mm/slub.o
  CC      kernel/smp.o
  CC [M]  drivers/net/usb/mcs7830.o
  CC [M]  drivers/net/ethernet/intel/igb/e1000_82575.o
  CC      fs/proc_namespace.o
  CC      fs/direct-io.o
  CC      drivers/acpi/acpica/utglobal.o
  LD [M]  drivers/net/vxlan/vxlan.o
  CC      kernel/uid16.o
  CC      drivers/acpi/processor_idle.o
  CC      mm/migrate.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_nvm.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_phy.o
  CC      drivers/usb/host/xhci-pci.o
  CC [M]  drivers/gpu/drm/nouveau/nvif/userc361.o
  CC      lib/base64.o
  CC      drivers/acpi/acpica/uthex.o
  LD [M]  drivers/md/persistent-data/dm-persistent-data.o
  CC      drivers/cpuidle/driver.o
  CC [M]  drivers/gpu/drm/i915/vlv_sideband.o
  CC      lib/once.o
  CC      drivers/cpufreq/cpufreq_ondemand.o
  CC      lib/refcount.o
  CC      drivers/acpi/processor_throttling.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_diag.o
  CC      drivers/md/md-autodetect.o
  AR      drivers/cpuidle/governors/built-in.a
  CC      mm/memory-tiers.o
  CC      drivers/i2c/i2c-dev.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_pc.o
  CC      fs/eventpoll.o
  CC      fs/btrfs/scrub.o
  CC      lib/rcuref.o
  CC      drivers/mmc/core/core.o
  LD [M]  drivers/net/ethernet/intel/igbvf/igbvf.o
  CC      drivers/mmc/host/sdhci.o
  AR      drivers/ufs/built-in.a
  CC      drivers/mmc/host/sdhci-pci-core.o
  AR      drivers/leds/trigger/built-in.a
  CC [M]  drivers/leds/trigger/ledtrig-audio.o
  AR      drivers/leds/blink/built-in.a
  CC      drivers/acpi/acpica/utids.o
  AR      drivers/leds/simple/built-in.a
  CC      drivers/acpi/acpica/utinit.o
  CC      drivers/cpuidle/governor.o
  CC      drivers/mmc/core/bus.o
  CC      lib/usercopy.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_ethtool.o
  CC      drivers/acpi/processor_perflib.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/client.o
  CC      kernel/kallsyms.o
  CC      kernel/acct.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_ptp.o
  CC [M]  drivers/net/usb/usbnet.o
  CC      drivers/leds/led-core.o
  CC      fs/anon_inodes.o
  CC      kernel/crash_core.o
  CC      drivers/acpi/container.o
  CC      fs/btrfs/backref.o
  CC      drivers/cpufreq/cpufreq_governor.o
  CC      lib/errseq.o
  CC      drivers/cpufreq/cpufreq_governor_attr_set.o
  CC      drivers/acpi/thermal.o
  CC      fs/signalfd.o
  LD [M]  drivers/net/ethernet/intel/e1000/e1000.o
  CC      drivers/acpi/acpica/utlock.o
  CC      drivers/cpuidle/sysfs.o
  CC      drivers/acpi/acpi_memhotplug.o
  CC      lib/bucket_locks.o
  CC [M]  drivers/net/ethernet/intel/e1000e/80003es2lan.o
  CC      drivers/acpi/ioapic.o
  CC [M]  drivers/net/usb/cdc_ncm.o
  CC      mm/migrate_device.o
  AR      drivers/usb/host/built-in.a
  AR      drivers/firmware/arm_ffa/built-in.a
  AR      drivers/usb/built-in.a
  AR      drivers/firmware/arm_scmi/built-in.a
  CC [M]  drivers/i2c/i2c-smbus.o
  AR      drivers/firmware/broadcom/built-in.a
  CC [M]  drivers/gpu/drm/i915/vlv_suspend.o
  AR      drivers/firmware/cirrus/built-in.a
  AR      drivers/firmware/meson/built-in.a
  CC [M]  drivers/net/ethernet/intel/igb/e1000_mac.o
  CC      fs/btrfs/ulist.o
  CC [M]  drivers/gpu/drm/i915/soc/intel_dram.o
  CC      drivers/mmc/core/host.o
  CC      drivers/firmware/efi/libstub/efi-stub-helper.o
  CC      drivers/acpi/acpica/utmath.o
  CC      drivers/md/dm-uevent.o
  CC      drivers/md/dm.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_submit.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/atombios_encoders.o
  CC      drivers/mmc/core/mmc.o
  CC      drivers/acpi/battery.o
  CC      drivers/acpi/acpica/utmisc.o
  CC      drivers/acpi/hed.o
  CC      drivers/leds/led-class.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/engine.o
  CC      lib/generic-radix-tree.o
  CC      drivers/cpuidle/poll_state.o
  CC      drivers/cpufreq/acpi-cpufreq.o
  CC      kernel/compat.o
  CC [M]  drivers/gpu/drm/xe/xe_heci_gsc.o
  CC      lib/string_helpers.o
  CC      drivers/leds/led-triggers.o
  CC      drivers/acpi/acpica/utmutex.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/enum.o
  CC      drivers/cpufreq/intel_pstate.o
  CC      drivers/cpuidle/cpuidle-haltpoll.o
  CC      drivers/acpi/bgrt.o
  CC      mm/huge_memory.o
  CC      kernel/utsname.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_engine.o
  CC      fs/btrfs/qgroup.o
  CC [M]  drivers/i2c/i2c-mux.o
  CC      lib/hexdump.o
  CC [M]  drivers/gpu/drm/i915/soc/intel_gmch.o
  CC      drivers/firmware/efi/libstub/gop.o
  CC      drivers/firmware/efi/efi-bgrt.o
  CC [M]  drivers/net/ethernet/intel/ixgbevf/vf.o
  CC      kernel/user_namespace.o
  AR      drivers/net/ethernet/intel/built-in.a
  CC      lib/kstrtox.o
  CC      drivers/firmware/efi/efi.o
  CC      drivers/acpi/acpica/utnonansi.o
  AR      drivers/firmware/imx/built-in.a
  CC [M]  drivers/net/ethernet/intel/ixgbevf/mbx.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_dump.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_tsn.o
  CC [M]  drivers/net/ethernet/intel/e1000e/mac.o
  AR      drivers/cpuidle/built-in.a
  CC [M]  drivers/net/ethernet/intel/ixgbevf/ethtool.o
  CC [M]  drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.o
  CC [M]  drivers/net/ethernet/intel/igc/igc_xdp.o
  CC      drivers/mmc/core/mmc_ops.o
  CC      drivers/mmc/host/sdhci-pci-o2micro.o
  CC      drivers/mmc/host/sdhci-pci-arasan.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/event.o
  CC      mm/khugepaged.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_engine_class_sysfs.o
  CC      drivers/acpi/cppc_acpi.o
  AR      drivers/firmware/psci/built-in.a
  AR      drivers/leds/built-in.a
  CC      drivers/acpi/spcr.o
  CC [M]  fs/smb/client/cifs_spnego_negtokeninit.asn1.o
  CC [M]  fs/smb/client/asn1.o
  CC      drivers/firmware/efi/libstub/secureboot.o
  CC      mm/page_counter.o
  CC      drivers/acpi/acpi_pad.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/firmware.o
  CC      drivers/firmware/efi/libstub/tpm.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_fence.o
  CC [M]  drivers/net/ethernet/intel/igb/e1000_nvm.o
  CC      mm/memcontrol.o
  CC      drivers/acpi/acpica/utobject.o
  CC      drivers/mmc/core/sd.o
  CC      lib/debug_info.o
  CC [M]  drivers/net/usb/r8153_ecm.o
  CC      fs/btrfs/send.o
  AR      drivers/firmware/smccc/built-in.a
  AR      drivers/firmware/tegra/built-in.a
  AR      drivers/firmware/xilinx/built-in.a
  AR      drivers/i2c/built-in.a
  CC      drivers/firmware/dmi_scan.o
  AR      drivers/crypto/stm32/built-in.a
  CC      drivers/mmc/core/sd_ops.o
  AR      drivers/crypto/xilinx/built-in.a
  AR      drivers/crypto/hisilicon/built-in.a
  AR      drivers/crypto/intel/keembay/built-in.a
  AR      drivers/crypto/intel/ixp4xx/built-in.a
  AR      drivers/crypto/intel/built-in.a
  AR      drivers/crypto/starfive/built-in.a
  AR      drivers/crypto/built-in.a
  CC      drivers/clocksource/acpi_pm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_sa.o
  CC      drivers/clocksource/i8253.o
  CC [M]  drivers/gpu/drm/i915/soc/intel_pch.o
  CC      drivers/firmware/efi/vars.o
  CC      drivers/firmware/efi/libstub/file.o
  CC      drivers/mmc/host/sdhci-pci-dwc-mshc.o
  CC      kernel/pid_namespace.o
  CC      drivers/acpi/acpica/utosi.o
  UPD     kernel/config_data
  CC      drivers/acpi/acpica/utownerid.o
  CC      mm/vmpressure.o
  CC      drivers/hid/usbhid/hid-core.o
  CC      drivers/hid/hid-core.o
  CC      drivers/hid/usbhid/hiddev.o
  CC [M]  drivers/net/ethernet/intel/ixgbevf/ipsec.o
  CC      drivers/firmware/efi/libstub/mem.o
  LD [M]  drivers/net/ethernet/intel/igc/igc.o
  CC      drivers/mmc/core/sdio.o
  AR      drivers/staging/media/built-in.a
  CC      drivers/firmware/efi/libstub/random.o
  AR      drivers/staging/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_huc.o
  CC      drivers/mmc/core/sdio_ops.o
  CC [M]  drivers/net/ethernet/intel/e100.o
  CC      drivers/firmware/dmi-sysfs.o
  CC      drivers/mmc/host/sdhci-pci-gli.o
  CC      drivers/md/dm-table.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/gpuobj.o
  CC      drivers/md/dm-target.o
  AR      drivers/platform/x86/amd/built-in.a
  AR      drivers/platform/surface/built-in.a
  CC      drivers/platform/x86/p2sb.o
  CC      drivers/platform/x86/intel/pmc/core.o
  LD [M]  drivers/net/usb/asix.o
  CC      drivers/firmware/dmi-id.o
  CC      drivers/firmware/efi/reboot.o
  AR      drivers/clocksource/built-in.a
  CC [M]  drivers/net/ethernet/intel/igb/e1000_phy.o
  CC      drivers/firmware/efi/libstub/randomalloc.o
  CC      mm/swap_cgroup.o
  CC      drivers/mailbox/mailbox.o
  CC      fs/btrfs/dev-replace.o
  LD [M]  fs/smb/client/cifs.o
  CC      drivers/mailbox/pcc.o
  CC [M]  drivers/net/ethernet/intel/e1000e/manage.o
  CC      drivers/acpi/acpica/utpredef.o
  CC      drivers/platform/x86/intel/pmc/core_ssram.o
  AR      drivers/cpufreq/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_huc_debugfs.o
  CC      drivers/mmc/host/sdhci-acpi.o
  CC [M]  drivers/net/ethernet/intel/igb/e1000_mbx.o
  AR      drivers/net/ethernet/mscc/built-in.a
  CC      drivers/acpi/acpica/utresdecode.o
  CC [M]  drivers/net/ethernet/intel/igb/e1000_i210.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_mbx.o
  CC      lib/iomap.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/atombios_i2c.o
  CC      drivers/platform/x86/intel/pmc/spt.o
  CC      mm/hugetlb_cgroup.o
  CC      drivers/firmware/efi/libstub/pci.o
  CC      drivers/firmware/efi/libstub/skip_spaces.o
  CC [M]  drivers/acpi/acpi_video.o
  CC      drivers/platform/x86/intel/pmc/cnp.o
  CC      kernel/stop_machine.o
  CC      drivers/platform/x86/intel/pmc/icl.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/intr.o
  CC      drivers/acpi/acpica/utresrc.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_x540.o
  CC      drivers/firmware/memmap.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_x550.o
  CC      fs/timerfd.o
  CC [M]  drivers/gpu/drm/i915/i915_memcpy.o
  CC      drivers/acpi/acpica/utstate.o
  CC      drivers/mmc/core/sdio_bus.o
  CC      drivers/firmware/efi/libstub/lib-cmdline.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/ioctl.o
  CC      kernel/kprobes.o
  AR      drivers/mailbox/built-in.a
  CC      kernel/hung_task.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_lib.o
  CC      drivers/md/dm-linear.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.o
  CC [M]  drivers/net/ethernet/intel/igb/igb_ptp.o
  CC      kernel/watchdog.o
  CC      drivers/mmc/core/sdio_cis.o
  CC [M]  drivers/gpu/drm/i915/i915_mm.o
  CC      drivers/platform/x86/intel/pmc/tgl.o
  CC      drivers/acpi/acpica/utstring.o
  CC [M]  drivers/gpu/drm/xe/xe_irq.o
  CC [M]  drivers/gpu/drm/xe/xe_lrc.o
  CC      kernel/watchdog_perf.o
  CC      drivers/firmware/efi/libstub/lib-ctype.o
  CC      drivers/firmware/efi/libstub/alignedmem.o
  CC      lib/pci_iomap.o
  CC      drivers/hid/hid-input.o
  CC [M]  drivers/net/ethernet/intel/e1000e/nvm.o
  AR      drivers/hid/usbhid/built-in.a
  CC      drivers/hid/hid-quirks.o
  CC      drivers/acpi/acpica/utstrsuppt.o
  CC      drivers/firmware/efi/memattr.o
  GEN     xe_wa_oob.c xe_wa_oob.h
  CC      drivers/firmware/efi/tpm.o
  CC      drivers/mmc/host/cqhci-core.o
  CC      drivers/platform/x86/pmc_atom.o
  GEN     xe_wa_oob.c xe_wa_oob.h
  CC [M]  drivers/net/ethernet/intel/igb/igb_hwmon.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.o
  CC [M]  drivers/platform/x86/wmi.o
  CC [M]  drivers/net/ethernet/intel/e1000e/phy.o
  CC      drivers/firmware/efi/libstub/relocate.o
  CC      fs/btrfs/raid56.o
  CC      drivers/platform/x86/intel/pmc/adl.o
  CC      drivers/platform/x86/intel/pmc/mtl.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/memory.o
  CC      drivers/md/dm-stripe.o
  CC [M]  drivers/platform/x86/wmi-bmof.o
  CC [M]  drivers/platform/x86/mxm-wmi.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.o
  CC      fs/eventfd.o
  CC      mm/kmemleak.o
  CC      drivers/mmc/core/sdio_io.o
  CC      drivers/acpi/acpica/utstrtoul64.o
  CC      mm/page_isolation.o
  CC      drivers/mmc/core/sdio_irq.o
  CC      fs/btrfs/uuid-tree.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_dcb.o
  CC [M]  drivers/platform/x86/intel_ips.o
  CC      kernel/seccomp.o
  CC      mm/early_ioremap.o
  CC      lib/iomap_copy.o
  CC      drivers/firmware/efi/libstub/printk.o
  CC      kernel/relay.o
  CC      drivers/firmware/efi/memmap.o
  CC      fs/userfaultfd.o
  CC      kernel/utsname_sysctl.o
  CC      drivers/mmc/core/slot-gpio.o
  CC      lib/devres.o
  LD [M]  drivers/net/ethernet/intel/ixgbevf/ixgbevf.o
  CC      drivers/hid/hid-debug.o
  CC [M]  drivers/gpu/drm/i915/i915_sw_fence.o
  CC [M]  drivers/gpu/drm/i915/i915_sw_fence_work.o
  CC [M]  drivers/gpu/drm/i915/i915_syncmap.o
  CC      fs/aio.o
  CC      mm/cma.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_82598.o
  CC      drivers/acpi/acpica/utxface.o
  CC      drivers/acpi/acpica/utxfinit.o
  CC      drivers/firmware/efi/libstub/vsprintf.o
  CC      drivers/platform/x86/intel/pmc/pltdrv.o
  CC      kernel/delayacct.o
  CC      lib/check_signature.o
  CC      lib/interval_tree.o
  CC      drivers/mmc/core/regulator.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/mm.o
  CC      drivers/md/dm-ioctl.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/object.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.o
  CC      kernel/taskstats.o
  LD [M]  drivers/net/ethernet/intel/igb/igb.o
  CC [M]  drivers/net/ethernet/intel/e1000e/param.o
  CC      drivers/firmware/efi/libstub/x86-stub.o
  CC      drivers/mmc/core/debugfs.o
  CC      kernel/tsacct.o
  CC [M]  drivers/mmc/host/sdhci-pltfm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.o
  CC      mm/secretmem.o
  CC      fs/locks.o
  CC      fs/binfmt_script.o
  CC      drivers/firmware/efi/libstub/x86-5lvl.o
  CC      drivers/acpi/acpica/utxferror.o
  CC      drivers/md/dm-io.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_82599.o
  CC      fs/binfmt_elf.o
  CC      drivers/mmc/core/block.o
  CC      drivers/acpi/acpica/utxfmutex.o
  STUBCPY drivers/firmware/efi/libstub/alignedmem.stub.o
  CC      drivers/firmware/efi/esrt.o
  AR      drivers/platform/x86/intel/pmc/built-in.a
  CC      lib/assoc_array.o
  CC [M]  drivers/platform/x86/intel/pmt/class.o
  CC      drivers/firmware/efi/efi-pstore.o
  CC [M]  drivers/platform/x86/intel/pmt/telemetry.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_dcb_nl.o
  CC      drivers/firmware/efi/cper.o
  CC      kernel/tracepoint.o
  CC      drivers/gpu/drm/drm_mipi_dsi.o
  CC [M]  drivers/gpu/drm/ast/ast_drv.o
  CC [M]  drivers/gpu/drm/drm_aperture.o
  CC [M]  drivers/gpu/drm/ast/ast_i2c.o
  CC [M]  drivers/gpu/drm/xe/xe_mmio.o
  CC      fs/compat_binfmt_elf.o
  AR      drivers/acpi/acpica/built-in.a
  CC [M]  drivers/gpu/drm/i915/i915_user_extensions.o
  CC [M]  drivers/gpu/drm/xe/xe_mocs.o
  CC      drivers/platform/x86/intel/turbo_max_3.o
  CC [M]  drivers/acpi/video_detect.o
  CC [M]  drivers/platform/x86/intel/vsec.o
  AR      drivers/mmc/host/built-in.a
  CC      drivers/hid/hidraw.o
  CC      drivers/mmc/core/queue.o
  CC [M]  drivers/gpu/drm/xe/xe_module.o
  CC [M]  drivers/platform/x86/intel/pmt/crashlog.o
  CC [M]  drivers/net/ethernet/intel/e1000e/ethtool.o
  STUBCPY drivers/firmware/efi/libstub/efi-stub-helper.stub.o
  STUBCPY drivers/firmware/efi/libstub/file.stub.o
  CC      mm/userfaultfd.o
  CC [M]  drivers/gpu/drm/drm_atomic.o
  STUBCPY drivers/firmware/efi/libstub/gop.stub.o
  STUBCPY drivers/firmware/efi/libstub/lib-cmdline.stub.o
  STUBCPY drivers/firmware/efi/libstub/lib-ctype.stub.o
  CC      lib/list_debug.o
  STUBCPY drivers/firmware/efi/libstub/mem.stub.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/oproxy.o
  STUBCPY drivers/firmware/efi/libstub/pci.stub.o
  CC [M]  drivers/gpu/drm/xe/xe_pat.o
  CC [M]  drivers/gpu/drm/xe/xe_pci.o
  STUBCPY drivers/firmware/efi/libstub/printk.stub.o
  STUBCPY drivers/firmware/efi/libstub/random.stub.o
  CC [M]  drivers/platform/x86/intel/rst.o
  CC      drivers/firmware/efi/cper_cxl.o
  STUBCPY drivers/firmware/efi/libstub/randomalloc.stub.o
  STUBCPY drivers/firmware/efi/libstub/relocate.stub.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/option.o
  LD [M]  drivers/platform/x86/intel/pmt/pmt_telemetry.o
  STUBCPY drivers/firmware/efi/libstub/secureboot.stub.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/ramht.o
  CC      kernel/latencytop.o
  STUBCPY drivers/firmware/efi/libstub/skip_spaces.stub.o
  CC [M]  drivers/gpu/drm/i915/i915_ioc32.o
  STUBCPY drivers/firmware/efi/libstub/tpm.stub.o
  CC [M]  drivers/net/ethernet/intel/e1000e/netdev.o
  STUBCPY drivers/firmware/efi/libstub/vsprintf.stub.o
  LD [M]  drivers/platform/x86/intel/pmt/pmt_class.o
  STUBCPY drivers/firmware/efi/libstub/x86-5lvl.stub.o
  STUBCPY drivers/firmware/efi/libstub/x86-stub.stub.o
  CC [M]  drivers/gpu/drm/xe/xe_pcode.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ib.o
  AR      drivers/firmware/efi/libstub/lib.a
  CC [M]  drivers/gpu/drm/xe/xe_pm.o
  CC      kernel/irq_work.o
  CC [M]  drivers/gpu/drm/i915/i915_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_preempt_fence.o
  CC      drivers/hid/hid-generic.o
  CC      lib/debugobjects.o
  CC      mm/memremap.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_sysfs.o
  CC [M]  drivers/net/ethernet/intel/e1000e/ptp.o
  CC [M]  drivers/gpu/drm/drm_atomic_uapi.o
  CC      fs/btrfs/props.o
  CC [M]  drivers/gpu/drm/ast/ast_main.o
  CC      drivers/firmware/efi/runtime-wrappers.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/subdev.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_pll.o
  CC      mm/hmm.o
  CC      drivers/md/dm-kcopyd.o
  LD [M]  drivers/platform/x86/intel/pmt/pmt_crashlog.o
  CC      mm/memfd.o
  AR      drivers/acpi/built-in.a
  CC [M]  drivers/gpu/drm/i915/i915_debugfs_params.o
  AR      drivers/platform/x86/intel/built-in.a
  CC [M]  drivers/gpu/drm/i915/display/intel_display_debugfs.o
  CC      mm/bootmem_info.o
  LD [M]  drivers/platform/x86/intel/intel_vsec.o
  LD [M]  drivers/acpi/video.o
  LD [M]  drivers/platform/x86/intel/intel-rst.o
  AR      drivers/platform/x86/built-in.a
  CC      drivers/devfreq/devfreq.o
  AR      drivers/platform/built-in.a
  CC      drivers/hid/hid-a4tech.o
  CC      drivers/hid/hid-apple.o
  CC      drivers/powercap/powercap_sys.o
  CC [M]  drivers/devfreq/governor_simpleondemand.o
  AR      drivers/net/ethernet/neterion/built-in.a
  AR      drivers/net/ethernet/netronome/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_pt.o
  CC [M]  drivers/devfreq/governor_performance.o
  CC      drivers/powercap/intel_rapl_common.o
  CC [M]  drivers/gpu/drm/xe/xe_pt_walk.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_debugfs_params.o
  CC      kernel/static_call.o
  CC      drivers/hid/hid-belkin.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/core/uevent.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/nvfw/fw.o
  CC      drivers/hid/hid-cherry.o
  CC [M]  drivers/gpu/drm/xe/xe_query.o
  CC [M]  drivers/gpu/drm/xe/xe_range_fence.o
  CC      fs/mbcache.o
  AR      drivers/mmc/core/built-in.a
  CC [M]  drivers/gpu/drm/i915/display/intel_pipe_crc.o
  AR      drivers/mmc/built-in.a
  CC [M]  drivers/gpu/drm/i915/i915_pmu.o
  CC      lib/bitrev.o
  AR      drivers/perf/built-in.a
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.o
  CC      kernel/static_call_inline.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_debugfs.o
  CC      drivers/md/dm-sysfs.o
  CC      drivers/firmware/efi/dev-path-parser.o
  CC      drivers/powercap/intel_rapl_msr.o
  CC      drivers/firmware/efi/apple-properties.o
  CC      drivers/hid/hid-chicony.o
  CC      lib/crc16.o
  CC      kernel/user-return-notifier.o
  CC      fs/posix_acl.o
  CC      fs/coredump.o
  CC      drivers/ras/ras.o
  CC      fs/drop_caches.o
  CC      drivers/ras/debugfs.o
  CC      drivers/hid/hid-cypress.o
  CC [M]  drivers/gpu/drm/xe/xe_reg_sr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/nvfw/hs.o
  CC [M]  drivers/gpu/drm/ast/ast_mm.o
  CC [M]  drivers/gpu/drm/i915/gt/gen2_engine_cs.o
  CC      lib/crc-t10dif.o
  CC [M]  drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.o
  CC [M]  drivers/gpu/drm/i915/gt/gen6_engine_cs.o
  AR      drivers/hwtracing/intel_th/built-in.a
  CC [M]  drivers/gpu/drm/i915/gt/gen6_ppgtt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/nvfw/ls.o
  CC      drivers/firmware/efi/earlycon.o
  CC      drivers/md/dm-stats.o
  CC [M]  drivers/gpu/drm/i915/gt/gen7_renderclear.o
  CC      kernel/padata.o
  CC [M]  drivers/gpu/drm/xe/xe_reg_whitelist.o
  CC      drivers/hid/hid-ezkey.o
  CC      kernel/jump_label.o
  AR      mm/built-in.a
  CC      drivers/firmware/efi/cper-x86.o
  CC      drivers/hid/hid-kensington.o
  CC      fs/btrfs/free-space-tree.o
  HOSTCC  lib/gen_crc32table
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.o
  CC [M]  drivers/gpu/drm/drm_auth.o
  CC      lib/libcrc32c.o
  CC      kernel/context_tracking.o
  CC      lib/xxhash.o
  CC [M]  drivers/gpu/drm/drm_blend.o
  CC      drivers/android/binderfs.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/nvfw/acr.o
  CC [M]  drivers/gpu/drm/i915/gt/gen8_engine_cs.o
  CC      drivers/md/dm-rq.o
  AR      drivers/net/ethernet/ni/built-in.a
  AR      drivers/net/ethernet/packetengines/built-in.a
  AR      drivers/powercap/built-in.a
  AR      drivers/net/ethernet/realtek/built-in.a
  CC [M]  drivers/net/ethernet/realtek/8139cp.o
  AR      drivers/net/ethernet/renesas/built-in.a
  CC [M]  drivers/net/ethernet/realtek/8139too.o
  CC      drivers/md/dm-io-rewind.o
  CC [M]  drivers/gpu/drm/xe/xe_rtp.o
  AR      drivers/net/ethernet/sfc/built-in.a
  CC [M]  drivers/net/ethernet/realtek/r8169_main.o
  CC      drivers/md/dm-builtin.o
  CC      drivers/hid/hid-lg.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/nvfw/flcn.o
  CC      lib/genalloc.o
  AR      drivers/nvmem/layouts/built-in.a
  CC      drivers/nvmem/core.o
  AR      drivers/devfreq/built-in.a
  CC [M]  drivers/mtd/chips/chipreg.o
  CC [M]  drivers/gpu/drm/ast/ast_mode.o
  CC      lib/percpu_counter.o
  CC [M]  drivers/uio/uio.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/base.o
  CC      kernel/iomem.o
  CC      fs/sysctls.o
  CC [M]  drivers/md/dm-bufio.o
  CC      drivers/hid/hid-lg-g15.o
  AR      drivers/firmware/efi/built-in.a
  CC      fs/fhandle.o
  AR      drivers/firmware/built-in.a
  AR      drivers/ras/built-in.a
  CC [M]  drivers/net/ethernet/realtek/r8169_firmware.o
  CC      drivers/hid/hid-microsoft.o
  CC [M]  drivers/vfio/pci/vfio_pci_core.o
  CC [M]  drivers/vfio/pci/vfio_pci_intrs.o
  CC [M]  drivers/vfio/pci/vfio_pci_rdwr.o
  CC      kernel/rseq.o
  CC [M]  drivers/vfio/pci/vfio_pci_config.o
  AR      drivers/net/ethernet/smsc/built-in.a
  CC [M]  drivers/net/ethernet/smsc/smsc9420.o
  AR      drivers/net/ethernet/socionext/built-in.a
  CC      drivers/hid/hid-monterey.o
  CC [M]  drivers/net/ethernet/realtek/r8169_phy_config.o
  CC      drivers/android/binder.o
  CC      drivers/android/binder_alloc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_sync.o
  GZIP    kernel/config_data.gz
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/cmdq.o
  CC [M]  drivers/gpu/drm/drm_bridge.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/fw.o
  CC      kernel/configs.o
  CC [M]  drivers/gpu/drm/i915/gt/gen8_ppgtt.o
  CC [M]  drivers/mtd/mtdcore.o
  CC [M]  drivers/pps/pps.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_breadcrumbs.o
  CC [M]  drivers/mtd/mtdsuper.o
  CC [M]  drivers/pps/kapi.o
  CC [M]  drivers/mtd/mtdconcat.o
  CC [M]  drivers/mtd/mtdpart.o
  CC [M]  drivers/bluetooth/btusb.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_context.o
  CC [M]  drivers/gpu/drm/xe/xe_ring_ops.o
  CC      lib/fault-inject.o
  CC [M]  drivers/bluetooth/btintel.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/msgq.o
  CC [M]  drivers/mtd/mtdchar.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/qmgr.o
  LD [M]  drivers/net/ethernet/intel/ixgbe/ixgbe.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/v1.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/gm200.o
  CC      lib/syscall.o
  CC      fs/btrfs/tree-checker.o
  CC [M]  drivers/gpu/drm/xe/xe_sa.o
  CC      fs/btrfs/space-info.o
  CC [M]  drivers/vfio/pci/vfio_pci.o
  CC      fs/btrfs/block-rsv.o
  CC [M]  drivers/bluetooth/btbcm.o
  AR      drivers/nvmem/built-in.a
  CC [M]  drivers/dca/dca-core.o
  AR      drivers/hid/built-in.a
  CC [M]  drivers/ssb/main.o
  CC [M]  drivers/vhost/net.o
  CC [M]  drivers/ssb/scan.o
  AR      kernel/built-in.a
  CC [M]  drivers/vhost/vhost.o
  CC [M]  drivers/dca/dca-sysfs.o
  CC [M]  drivers/ssb/sprom.o
  CC [M]  drivers/pps/sysfs.o
  CC      fs/btrfs/delalloc-space.o
  CC      lib/dynamic_debug.o
  CC [M]  drivers/ssb/pci.o
  CC      fs/btrfs/block-group.o
  CC      fs/btrfs/discard.o
  CC [M]  drivers/gpu/drm/ast/ast_post.o
  CC [M]  drivers/gpu/drm/drm_cache.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/gp102.o
  CC      lib/errname.o
  CC [M]  drivers/md/dm-bio-prison-v1.o
  CC [M]  drivers/ssb/pcihost_wrapper.o
  CC      fs/btrfs/reflink.o
  CC      fs/btrfs/subpage.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.o
  LD [M]  drivers/pps/pps_core.o
  CC      fs/btrfs/tree-mod-log.o
  CC [M]  drivers/gpu/drm/drm_client.o
  CC [M]  drivers/gpu/drm/drm_client_modeset.o
  CC      fs/btrfs/extent-io-tree.o
  CC [M]  drivers/gpu/drm/drm_color_mgmt.o
  CC [M]  drivers/gpu/drm/xe/xe_sched_job.o
  CC [M]  drivers/gpu/drm/xe/xe_step.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_context_sseu.o
  LD [M]  drivers/vfio/pci/vfio-pci.o
  LD [M]  drivers/vfio/pci/vfio-pci-core.o
  CC [M]  drivers/gpu/drm/drm_connector.o
  CC [M]  drivers/vfio/vfio_main.o
  CC      fs/btrfs/fs.o
  CC [M]  drivers/ssb/driver_chipcommon.o
  AR      drivers/net/ethernet/vertexcom/built-in.a
  LD [M]  drivers/dca/dca.o
  CC [M]  drivers/vfio/iova_bitmap.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_engine_cs.o
  CC      fs/btrfs/messages.o
  CC [M]  drivers/ssb/driver_chipcommon_pmu.o
  CC [M]  drivers/ssb/driver_pcicore.o
  CC [M]  drivers/vfio/group.o
  CC [M]  drivers/gpu/drm/drm_crtc.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_engine_heartbeat.o
  LD [M]  drivers/mtd/mtd.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_engine_pm.o
  CC [M]  drivers/gpu/drm/ast/ast_dp501.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.o
  CC      fs/btrfs/bio.o
  LD [M]  drivers/net/ethernet/intel/e1000e/e1000e.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/ga100.o
  CC [M]  drivers/md/dm-bio-prison-v2.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_engine_user.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_execlists_submission.o
  CC [M]  drivers/vfio/container.o
  CC [M]  drivers/gpu/drm/drm_displayid.o
  CC [M]  drivers/vfio/virqfd.o
  CC [M]  drivers/vfio/vfio_iommu_type1.o
  CC [M]  drivers/gpu/drm/xe/xe_sync.o
  CC      fs/btrfs/lru_cache.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ggtt.o
  CC [M]  drivers/bluetooth/btrtl.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ggtt_fencing.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt.o
  LD [M]  drivers/net/ethernet/realtek/r8169.o
  AR      drivers/net/ethernet/wangxun/built-in.a
  AR      drivers/net/ethernet/xilinx/built-in.a
  AR      drivers/net/ethernet/synopsys/built-in.a
  AR      drivers/net/ethernet/pensando/built-in.a
  AR      drivers/net/ethernet/built-in.a
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_buffer_pool.o
  CC [M]  drivers/gpu/drm/ast/ast_dp.o
  CC [M]  drivers/md/dm-crypt.o
  CC      fs/btrfs/acl.o
  AR      drivers/net/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_tile.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_clock_utils.o
  LD [M]  drivers/ssb/ssb.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_debugfs.o
  CC      lib/nlattr.o
  CC      lib/checksum.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vf_error.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_engines_debugfs.o
  CC [M]  drivers/vhost/iotlb.o
  CC [M]  drivers/gpu/drm/drm_drv.o
  LD [M]  drivers/vhost/vhost_net.o
  CC [M]  drivers/gpu/drm/drm_dumb_buffers.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_irq.o
  CC      lib/cpu_rmap.o
  CC      lib/dynamic_queue_limits.o
  CC [M]  drivers/gpu/drm/drm_edid.o
  CC [M]  drivers/gpu/drm/drm_encoder.o
  CC [M]  drivers/md/dm-thin.o
  CC [M]  drivers/md/dm-thin-metadata.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/falcon/ga102.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_sched.o
  LD [M]  drivers/md/dm-bio-prison.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/lsfw.o
  CC      lib/glob.o
  CC [M]  drivers/gpu/drm/drm_file.o
  CC      lib/strncpy_from_user.o
  CC [M]  drivers/gpu/drm/drm_fourcc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gm200.o
  LD [M]  drivers/vfio/vfio.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gm20b.o
  CC [M]  drivers/gpu/drm/drm_framebuffer.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_mcr.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_pm.o
  CC [M]  drivers/gpu/drm/xe/xe_tile_sysfs.o
  CC      lib/strnlen_user.o
  LD [M]  drivers/vhost/vhost_iotlb.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.o
  CC [M]  drivers/gpu/drm/drm_gem.o
  CC [M]  drivers/gpu/drm/xe/xe_trace.o
  CC      lib/net_utils.o
  CC [M]  drivers/gpu/drm/drm_ioctl.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ids.o
  LD [M]  drivers/gpu/drm/ast/ast.o
  CC [M]  drivers/gpu/drm/drm_lease.o
  CC [M]  drivers/gpu/drm/drm_managed.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_sys_mgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_mmhub.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_pm_irq.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_requests.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_stolen_mgr.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_vram_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gp102.o
  CC      lib/sg_pool.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.o
  CC      lib/stackdepot.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.o
  CC [M]  drivers/gpu/drm/drm_mm.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gtt.o
  CC [M]  drivers/gpu/drm/drm_mode_config.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gp108.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gv100.o
  CC [M]  drivers/gpu/drm/drm_mode_object.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/gp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/tu102.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_llc.o
  AR      drivers/md/built-in.a
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/ga100.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_lrc.o
  CC      lib/ucs2_string.o
  CC [M]  drivers/gpu/drm/xe/xe_tuning.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_migrate.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/acr/ga102.o
  AR      fs/btrfs/built-in.a
  AR      fs/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_uc.o
  CC [M]  drivers/gpu/drm/drm_modes.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_mocs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.o
  CC      lib/sbitmap.o
  CC [M]  drivers/gpu/drm/drm_modeset_lock.o
  CC [M]  drivers/gpu/drm/xe/xe_uc_debugfs.o
  CC [M]  drivers/gpu/drm/drm_plane.o
  CC [M]  drivers/gpu/drm/drm_prime.o
  CC      lib/group_cpus.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_csa.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ras.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ppgtt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/base.o
  CC [M]  drivers/gpu/drm/xe/xe_uc_fw.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_rc6.o
  CC [M]  drivers/gpu/drm/drm_print.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_region_lmem.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/g84.o
  CC [M]  drivers/gpu/drm/xe/xe_vm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_cpu.o
  CC [M]  drivers/gpu/drm/drm_property.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/gf100.o
  CC [M]  drivers/gpu/drm/drm_syncobj.o
  CC [M]  drivers/gpu/drm/drm_sysfs.o
  AR      drivers/android/built-in.a
  CC [M]  drivers/gpu/drm/xe/xe_vm_madvise.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/gk20a.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.o
  CC [M]  drivers/gpu/drm/xe/xe_wait_user_fence.o
  CC [M]  drivers/gpu/drm/drm_trace_points.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_renderstate.o
  CC [M]  lib/asn1_decoder.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/gm107.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/gm20b.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_reset.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ring.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bar/tu102.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ring_submission.o
  CC [M]  drivers/gpu/drm/xe/xe_wa.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/base.o
  GEN     lib/oid_registry_data.c
  CC [M]  lib/oid_registry.o
  CC [M]  drivers/gpu/drm/drm_vblank.o
  CC [M]  drivers/gpu/drm/xe/xe_wopcm.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_rps.o
  CC [M]  drivers/gpu/drm/drm_vblank_work.o
  LD [M]  drivers/md/dm-thin-pool.o
  CC [M]  drivers/gpu/drm/drm_vma_manager.o
  CC [M]  drivers/gpu/drm/drm_writeback.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_sa_media.o
  AR      lib/lib.a
  CC [M]  drivers/gpu/drm/lib/drm_random.o
  GEN     lib/crc32table.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/bit.o
  CC      lib/crc32.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/boost.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/conn.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/cstep.o
  CC [M]  drivers/gpu/drm/drm_ioc32.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/dcb.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_nbio.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_sseu.o
  CC [M]  drivers/gpu/drm/drm_panel.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/disp.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/dp.o
  CC [M]  drivers/gpu/drm/xe/xe_hwmon.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_umc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/extdev.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_sseu_debugfs.o
  CC [M]  drivers/gpu/drm/drm_pci.o
  CC [M]  drivers/gpu/drm/xe/xe_display.o
  CC [M]  drivers/gpu/drm/drm_debugfs.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/fan.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_timeline.o
  CC [M]  drivers/gpu/drm/drm_debugfs_crc.o
  CC [M]  drivers/gpu/drm/xe/display/xe_fb_pin.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/gpio.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smu_v11_0_i2c.o
  CC [M]  drivers/gpu/drm/drm_edid_load.o
  CC [M]  drivers/gpu/drm/drm_panel_orientation_quirks.o
  AR      lib/built-in.a
  CC [M]  drivers/gpu/drm/i915/gt/intel_tlb.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_wopcm.o
  CC [M]  drivers/gpu/drm/xe/display/xe_hdcp_gsc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/i2c.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_workarounds.o
  CC [M]  drivers/gpu/drm/drm_exec.o
  CC [M]  drivers/gpu/drm/drm_gpuvm.o
  CC [M]  drivers/gpu/drm/i915/gt/shmem_utils.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/iccsense.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/image.o
  CC [M]  drivers/gpu/drm/i915/gt/sysfs_engines.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_ggtt_gmch.o
  CC [M]  drivers/gpu/drm/i915/gt/gen6_renderstate.o
  CC [M]  drivers/gpu/drm/i915/gt/gen7_renderstate.o
  CC [M]  drivers/gpu/drm/i915/gt/gen8_renderstate.o
  CC [M]  drivers/gpu/drm/drm_buddy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/init.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/mxm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/npde.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/pcir.o
  CC [M]  drivers/gpu/drm/drm_gem_shmem_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/perf.o
  CC [M]  drivers/gpu/drm/i915/gt/gen9_renderstate.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/pll.o
  CC [M]  drivers/gpu/drm/drm_suballoc.o
  CC [M]  drivers/gpu/drm/xe/display/xe_plane_initial.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/pmu.o
  CC [M]  drivers/gpu/drm/drm_gem_ttm_helper.o
  CC [M]  drivers/gpu/drm/drm_atomic_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/power_budget.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.o
  CC [M]  drivers/gpu/drm/drm_atomic_state_helper.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_busy.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_rap.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_clflush.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_context.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_create.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_dmabuf.o
  CC [M]  drivers/gpu/drm/xe/display/xe_display_rps.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_domain.o
  CC [M]  drivers/gpu/drm/xe/display/xe_display_misc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/ramcfg.o
  CC [M]  drivers/gpu/drm/xe/display/ext/i915_irq.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_fw_attestation.o
  CC [M]  drivers/gpu/drm/xe/display/ext/i915_utils.o
  CC [M]  drivers/gpu/drm/drm_bridge_connector.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_execbuffer.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_internal.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/rammap.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadow.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_securedisplay.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_eeprom.o
  CC [M]  drivers/gpu/drm/drm_crtc_helper.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_object.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadowacpi.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_lmem.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadowof.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_mman.o
  CC [M]  drivers/gpu/drm/drm_damage_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadowpci.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_pages.o
  CC [M]  drivers/gpu/drm/drm_encoder_slave.o
  CC [M]  drivers/gpu/drm/drm_flip_work.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_mca.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadowramin.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/shadowrom.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/timing.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/therm.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_phys.o
  CC [M]  drivers/gpu/drm/drm_format_helper.o
  CC [M]  drivers/gpu/drm/drm_gem_atomic_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/vmap.o
  CC [M]  drivers/gpu/drm/xe/i915-soc/intel_dram.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_psp_ta.o
  CC [M]  drivers/gpu/drm/xe/i915-soc/intel_pch.o
  CC [M]  drivers/gpu/drm/drm_gem_framebuffer_helper.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_lsdma.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_pm.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_region.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_shmem.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/volt.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_fdinfo.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/vpstate.o
  CC [M]  drivers/gpu/drm/xe/i915-display/icl_dsi.o
  CC [M]  drivers/gpu/drm/drm_kms_helper_common.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_atomic.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_pmu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/xpio.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_shrinker.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/cik.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/cik_ih.o
  CC [M]  drivers/gpu/drm/drm_modeset_helper.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_stolen.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_throttle.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/M0203.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/M0205.o
  CC [M]  drivers/gpu/drm/drm_plane_helper.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_atomic_plane.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/M0209.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_audio.o
  CC [M]  drivers/gpu/drm/drm_probe_helper.o
  CC [M]  drivers/gpu/drm/drm_rect.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_tiling.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_ttm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bios/P0260.o
  CC [M]  drivers/gpu/drm/drm_self_refresh_helper.o
  CC [M]  drivers/gpu/drm/drm_simple_kms_helper.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_backlight.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/dce_v8_0.o
  CC [M]  drivers/gpu/drm/bridge/panel.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_bios.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v7_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/base.o
  CC [M]  drivers/gpu/drm/drm_fbdev_generic.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/hwsq.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/nv04.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_ttm_move.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_userptr.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gem_wait.o
  CC [M]  drivers/gpu/drm/i915/gem/i915_gemfs.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_bw.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/cik_sdma.o
  CC [M]  drivers/gpu/drm/drm_fb_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/nv31.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/uvd_v4_2.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/nv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vce_v2_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/si.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/g94.o
  CC [M]  drivers/gpu/drm/i915/i915_active.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/bus/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_cdclk.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/base.o
  LD [M]  drivers/gpu/drm/drm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v6_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/nv04.o
  CC [M]  drivers/gpu/drm/i915/i915_cmd_parser.o
  LD [M]  drivers/gpu/drm/drm_shmem_helper.o
  LD [M]  drivers/gpu/drm/drm_suballoc_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/nv40.o
  LD [M]  drivers/gpu/drm/drm_ttm_helper.o
  CC [M]  drivers/gpu/drm/i915/i915_deps.o
  AR      drivers/gpu/drm/built-in.a
  CC [M]  drivers/gpu/drm/i915/i915_gem_evict.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_color.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/g84.o
  CC [M]  drivers/gpu/drm/i915/i915_gem_gtt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/gt215.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/si_ih.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/mcp77.o
  CC [M]  drivers/gpu/drm/i915/i915_gem_ww.o
  CC [M]  drivers/gpu/drm/i915/i915_gem.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_combo_phy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/gf100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/gk104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/gk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/gm20b.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_connector.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/pllnv04.o
  CC [M]  drivers/gpu/drm/i915/i915_query.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_crtc.o
  CC [M]  drivers/gpu/drm/i915/i915_request.o
  CC [M]  drivers/gpu/drm/i915/i915_scheduler.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/clk/pllgt215.o
  CC [M]  drivers/gpu/drm/i915/i915_trace_points.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_crtc_state_dump.o
  CC [M]  drivers/gpu/drm/i915/i915_ttm_buddy_manager.o
  CC [M]  drivers/gpu/drm/i915/i915_vma.o
  CC [M]  drivers/gpu/drm/i915/i915_vma_resource.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/si_dma.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_gsc_fw.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv04.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/dce_v6_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv05.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/uvd_v3_1.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_cursor.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vi.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mxgpu_vi.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_gsc_proxy.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_gsc_uc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv1a.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v6_1.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/soc15.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_cx0_phy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv20.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_gsc_uc_debugfs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/emu_soc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/g84.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_gsc_uc_heci_cmd_submit.o
  LD [M]  drivers/gpu/drm/drm_kms_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/g98.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gt215.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_ddi.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/mcp89.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_ddi_buf_trans.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_debugfs.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_ads.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gf100.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_capture.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_debugfs_params.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm107.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_ct.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm200.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v7_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/tu102.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_fw.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/devinit/ga100.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_device.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_driver.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/user.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vega10_reg_init.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vega20_reg_init.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/gp100.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_hwconfig.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v7_4.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v2_3.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/gp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/gv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nv.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fault/tu102.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_irq.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/base.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_log.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/arct_reg_init.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mxgpu_nv.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_log_debugfs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v7_2.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_params.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_rc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/hdp_v4_0.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv04.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_power.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv1a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv20.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv25.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/hdp_v5_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv30.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv35.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv36.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_guc_submission.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv40.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_power_map.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_huc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv41.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/aldebaran_reg_init.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/aldebaran.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/soc21.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_power_well.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv44.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_huc_debugfs.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_huc_fw.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_trace.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/sienna_cichlid.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_uc.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_uc_debugfs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smu_v13_0_10.o
  CC [M]  drivers/gpu/drm/i915/gt/uc/intel_uc_fw.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_display_wa.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dkl_phy.o
  CC [M]  drivers/gpu/drm/i915/gt/intel_gsc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv46.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v4_3.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dmc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv47.o
  CC [M]  drivers/gpu/drm/i915/i915_hwmon.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/hdp_v6_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v7_7.o
  CC [M]  drivers/gpu/drm/i915/display/hsw_ips.o
  CC [M]  drivers/gpu/drm/i915/display/intel_atomic.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv49.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/hdp_v5_2.o
  CC [M]  drivers/gpu/drm/i915/display/intel_atomic_plane.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/lsdma_v6_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv4e.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/g84.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gt215.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/mcp77.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/nbio_v7_9.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp_aux.o
  CC [M]  drivers/gpu/drm/i915/display/intel_audio.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/df_v1_7.o
  CC [M]  drivers/gpu/drm/i915/display/intel_bios.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/df_v3_6.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/df_v4_3.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp_aux_backlight.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp_hdcp.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp_link_training.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/mcp89.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_bw.o
  CC [M]  drivers/gpu/drm/i915/display/intel_cdclk.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dp_mst.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_color.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dpll.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gf108.o
  CC [M]  drivers/gpu/drm/i915/display/intel_combo_phy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk104.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dpll_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk110.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gk20a.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dpt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gm107.o
  CC [M]  drivers/gpu/drm/i915/display/intel_connector.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_1.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gm200.o
  CC [M]  drivers/gpu/drm/i915/display/intel_crtc.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_drrs.o
  CC [M]  drivers/gpu/drm/i915/display/intel_crtc_state_dump.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v9_4.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gm20b.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dsb.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v2_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v10_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_cursor.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gp100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v2_1.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v2_3.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dsi.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dsi_dcs_backlight.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gp102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gv100.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_dsi_vbt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/tu102.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ga100.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_driver.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v1_7.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_fb.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ga102.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_fbc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_fdi.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v3_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_irq.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v3_0_2.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_fifo_underrun.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_frontbuffer.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ram.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_params.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gmc_v11_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_power.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_global_state.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv04.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_gmbus.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_power_map.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_power_well.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv1a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv20.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv40.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_hdcp.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_hdmi.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_hotplug.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_hotplug_irq.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_reset.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_rps.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv41.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v3_0_1.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0_3.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/umc_v6_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_hti.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/umc_v6_1.o
  CC [M]  drivers/gpu/drm/i915/display/intel_display_wa.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_link_bw.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv44.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dmc.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dpio_phy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv49.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dpll.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv4e.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/umc_v6_7.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgt215.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/umc_v8_7.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dpll_mgr.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_lspcon.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/umc_v8_10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/rammcp77.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_modeset_lock.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ih.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_modeset_setup.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgf100.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_modeset_verify.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_panel.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dpt.o
  CC [M]  drivers/gpu/drm/i915/display/intel_drrs.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_pipe_crc.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_pmdemand.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_pps.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgf108.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/iceland_ih.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_psr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgk104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgm107.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgm200.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_qp_tables.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/tonga_ih.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_quirks.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_snps_phy.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/cz_ih.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dsb.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vega10_ih.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgp100.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_tc.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fb.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vega20_ih.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/navi10_ih.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fb_pin.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_vblank.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/ih_v6_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fbc.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_vdsc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/ih_v6_1.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_vga.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fdi.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_vrr.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fifo_underrun.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_psp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v3_1.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v10_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramgp102.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_wm.o
  CC [M]  drivers/gpu/drm/xe/i915-display/skl_scaler.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/sddr2.o
  CC [M]  drivers/gpu/drm/xe/i915-display/skl_universal_plane.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v11_0.o
  CC [M]  drivers/gpu/drm/xe/i915-display/skl_watermark.o
  CC [M]  drivers/gpu/drm/xe/xe_pmu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/sddr3.o
  CC [M]  drivers/gpu/drm/i915/display/intel_frontbuffer.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_acpi.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_opregion.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v11_0_8.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v12_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v13_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.o
  CC [M]  drivers/gpu/drm/i915/display/intel_global_state.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/dce_v10_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/dce_v11_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gddr3.o
  CC [M]  drivers/gpu/drm/xe/i915-display/intel_fbdev.o
  CC [M]  drivers/gpu/drm/i915/display/intel_hdcp.o
  CC [M]  drivers/gpu/drm/xe/xe_guc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vkms.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fb/gddr5.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.o
  CC [M]  drivers/gpu/drm/xe/xe_migrate.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fuse/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_hdcp_gsc.o
  CC [M]  drivers/gpu/drm/i915/display/intel_hotplug.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v9_4.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_klvs_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_errors_abi.h
  CC [M]  drivers/gpu/drm/i915/display/intel_hotplug_irq.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_actions_slpc_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fuse/nv50.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_communication_mmio_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_actions_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fuse/gf100.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_communication_ctb_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/fuse/gm107.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_messages_abi.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_gem.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_vma_types.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_irq.h
  CC [M]  drivers/gpu/drm/i915/display/intel_hti.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/vlv_sideband_reg.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_wakeref.h
  CC [M]  drivers/gpu/drm/i915/display/intel_link_bw.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_pcode.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_drv.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/base.o
  CC [M]  drivers/gpu/drm/i915/display/intel_load_detect.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v9_4_2.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/nv10.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_reg_defs.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_trace.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_reg.h
  CC [M]  drivers/gpu/drm/i915/display/intel_lpe_audio.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/nv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v10_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_modeset_lock.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/imu_v11_0.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_active_types.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_utils.h
  CC [M]  drivers/gpu/drm/i915/display/intel_modeset_verify.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v11_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/g94.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_config.h
  CC [M]  drivers/gpu/drm/i915/display/intel_modeset_setup.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/gfx_v11_0_3.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/imu_v11_0_3.o
  CC [M]  drivers/gpu/drm/i915/display/intel_overlay.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_vma.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/vlv_sideband.h
  CC [M]  drivers/gpu/drm/i915/display/intel_pch_display.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_gem_stolen.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_mchbar_regs.h
  CC [M]  drivers/gpu/drm/i915/display/intel_pch_refclk.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/gf119.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_debugfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/gk104.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_gpu_error.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/soc/intel_pch.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v2_4.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/soc/intel_dram.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/soc/intel_gmch.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_vgpu.h
  CC [M]  drivers/gpu/drm/i915/display/intel_plane_initial.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_fixed.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_runtime_pm.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_uncore.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_step.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_uc_fw.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gpio/ga102.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_pci_config.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gsp/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gsp/gv100.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/gem/i915_gem_object_frontbuffer.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/gsp/ga102.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/gt/intel_rps.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_clock_gating.h
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/intel_gt_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_pmdemand.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/base.o
  HDRTEST drivers/gpu/drm/xe/compat-i915-headers/i915_active.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/nv04.o
  CC [M]  drivers/gpu/drm/i915/display/intel_psr.o
  CC [M]  drivers/gpu/drm/i915/display/intel_quirks.o
  CC [M]  drivers/gpu/drm/i915/display/intel_sprite.o
  HDRTEST drivers/gpu/drm/xe/instructions/xe_instr_defs.h
  HDRTEST drivers/gpu/drm/xe/instructions/xe_gfxpipe_commands.h
  CC [M]  drivers/gpu/drm/i915/display/intel_sprite_uapi.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/nv4e.o
  CC [M]  drivers/gpu/drm/i915/display/intel_tc.o
  HDRTEST drivers/gpu/drm/xe/instructions/xe_mi_commands.h
  CC [M]  drivers/gpu/drm/i915/display/intel_vblank.o
  CC [M]  drivers/gpu/drm/i915/display/intel_vga.o
  CC [M]  drivers/gpu/drm/i915/display/intel_wm.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_reg_defs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_guc_regs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_gt_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v3_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v4_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v4_4.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/nv50.o
  CC [M]  drivers/gpu/drm/i915/display/i9xx_plane.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_gpu_commands.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/g94.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/gf117.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_lrc_layout.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/gf119.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_mchbar_regs.h
  CC [M]  drivers/gpu/drm/i915/display/i9xx_wm.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_engine_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v5_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v5_2.o
  CC [M]  drivers/gpu/drm/i915/display/skl_scaler.o
  HDRTEST drivers/gpu/drm/xe/tests/xe_test.h
  CC [M]  drivers/gpu/drm/i915/display/skl_universal_plane.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/sdma_v6_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/gk104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/gk110.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/gm200.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/pad.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padnv04.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padnv4e.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padnv50.o
  HDRTEST drivers/gpu/drm/xe/tests/xe_pci_test.h
  CC [M]  drivers/gpu/drm/i915/display/skl_watermark.o
  CC [M]  drivers/gpu/drm/i915/display/intel_acpi.o
  CC [M]  drivers/gpu/drm/i915/display/intel_opregion.o
  HDRTEST drivers/gpu/drm/xe/tests/xe_migrate_test.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_mes.o
  HDRTEST drivers/gpu/drm/xe/tests/xe_dma_buf_test.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/mes_v10_1.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/mes_v11_0.o
  HDRTEST drivers/gpu/drm/xe/tests/xe_bo_test.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padg94.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padgf119.o
  HDRTEST drivers/gpu/drm/xe/xe_assert.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/padgm200.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/bus.o
  CC [M]  drivers/gpu/drm/i915/display/intel_fbdev.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/busnv04.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_ch7017.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_ch7xxx.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_ivch.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_ns2501.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/busnv4e.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/uvd_v5_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/busnv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/busgf119.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/bit.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_sil164.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/auxg94.o
  CC [M]  drivers/gpu/drm/i915/display/dvo_tfp410.o
  CC [M]  drivers/gpu/drm/i915/display/g4x_dp.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/auxgf119.o
  HDRTEST drivers/gpu/drm/xe/xe_bb.h
  HDRTEST drivers/gpu/drm/xe/xe_bb_types.h
  HDRTEST drivers/gpu/drm/xe/xe_bo.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_doc.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_evict.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/auxgm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/uvd_v6_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/i2c/anx9805.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/uvd_v7_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/iccsense/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/iccsense/gf100.o
  CC [M]  drivers/gpu/drm/i915/display/g4x_hdmi.o
  CC [M]  drivers/gpu/drm/i915/display/icl_dsi.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv04.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv40.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/nv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vce.o
  CC [M]  drivers/gpu/drm/i915/display/intel_backlight.o
  HDRTEST drivers/gpu/drm/xe/xe_debugfs.h
  CC [M]  drivers/gpu/drm/i915/display/intel_crt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/instmem/gk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gf100.o
  HDRTEST drivers/gpu/drm/xe/xe_devcoredump.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gk104.o
  CC [M]  drivers/gpu/drm/i915/display/intel_cx0_phy.o
  HDRTEST drivers/gpu/drm/xe/xe_devcoredump_types.h
  HDRTEST drivers/gpu/drm/xe/xe_device.h
  CC [M]  drivers/gpu/drm/i915/display/intel_ddi.o
  HDRTEST drivers/gpu/drm/xe/xe_device_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_device_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gm107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vce_v3_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vce_v4_0.o
  HDRTEST drivers/gpu/drm/xe/xe_display.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_sw_ring.o
  CC [M]  drivers/gpu/drm/i915/display/intel_ddi_buf_trans.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v1_0.o
  HDRTEST drivers/gpu/drm/xe/xe_dma_buf.h
  HDRTEST drivers/gpu/drm/xe/xe_drm_client.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v2_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v2_5.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v3_0.o
  HDRTEST drivers/gpu/drm/xe/xe_drv.h
  CC [M]  drivers/gpu/drm/i915/display/intel_display_device.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v4_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gp100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.o
  HDRTEST drivers/gpu/drm/xe/xe_exec.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gp102.o
  HDRTEST drivers/gpu/drm/xe/xe_exec_queue.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/gp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/ltc/ga102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/nv04.o
  HDRTEST drivers/gpu/drm/xe/xe_exec_queue_types.h
  HDRTEST drivers/gpu/drm/xe/xe_execlist.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/nv11.o
  HDRTEST drivers/gpu/drm/xe/xe_execlist_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/nv17.o
  HDRTEST drivers/gpu/drm/xe/xe_force_wake.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/nv44.o
  HDRTEST drivers/gpu/drm/xe/xe_force_wake_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_display_trace.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/g84.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/g98.o
  HDRTEST drivers/gpu/drm/xe/xe_ggtt.h
  HDRTEST drivers/gpu/drm/xe/xe_ggtt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_gt.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_jpeg.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_clock.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gt215.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_debugfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v1_0.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_idle_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_idle_sysfs_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dkl_phy.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_mcr.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_pagefault.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_printk.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dp_aux.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dp_aux_backlight.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/athub_v1_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gk20a.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_sysfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gp100.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_sysfs_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dp_hdcp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/athub_v2_0.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/athub_v2_1.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/gp10b.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_tlb_invalidation_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/athub_v3_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v9_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v11_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mc/ga100.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dp_link_training.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/base.o
  HDRTEST drivers/gpu/drm/xe/xe_gt_topology.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dp_mst.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_ads.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_ads_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dsi.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_ct.h
  CC [M]  drivers/gpu/drm/i915/display/intel_dsi_dcs_backlight.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/nv04.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dsi_vbt.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_ct_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v11_0_6.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v13_0.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v13_0_3.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/nv41.o
  CC [M]  drivers/gpu/drm/i915/display/intel_dvo.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_debugfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/nv44.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/smuio_v13_0_6.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_fwif.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_reset.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_hwconfig.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/nv50.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_log.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/g84.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_log_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/mca_v3_0.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_pc.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_pc_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/mcp77.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_module.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_chardev.o
  HDRTEST drivers/gpu/drm/xe/xe_guc_submit.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_submit_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_topology.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_pasid.o
  HDRTEST drivers/gpu/drm/xe/xe_heci_gsc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_doorbell.o
  HDRTEST drivers/gpu/drm/xe/xe_huc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_flat_memory.o
  HDRTEST drivers/gpu/drm/xe/xe_huc_debugfs.h
  CC [M]  drivers/gpu/drm/i915/display/intel_gmbus.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_process.o
  HDRTEST drivers/gpu/drm/xe/xe_huc_types.h
  CC [M]  drivers/gpu/drm/i915/display/intel_hdmi.o
  CC [M]  drivers/gpu/drm/i915/display/intel_lspcon.o
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gm200.o
  CC [M]  drivers/gpu/drm/i915/display/intel_lvds.o
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine_class_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gm20b.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_queue.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gp100.o
  HDRTEST drivers/gpu/drm/xe/xe_hw_fence.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager_cik.o
  HDRTEST drivers/gpu/drm/xe/xe_hw_fence_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gp10b.o
  CC [M]  drivers/gpu/drm/i915/display/intel_panel.o
  HDRTEST drivers/gpu/drm/xe/xe_hwmon.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/gv100.o
  CC [M]  drivers/gpu/drm/i915/display/intel_pps.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager_vi.o
  HDRTEST drivers/gpu/drm/xe/xe_irq.h
  HDRTEST drivers/gpu/drm/xe/xe_lrc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager_v9.o
  CC [M]  drivers/gpu/drm/i915/display/intel_qp_tables.o
  HDRTEST drivers/gpu/drm/xe/xe_lrc_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager_v10.o
  CC [M]  drivers/gpu/drm/i915/display/intel_sdvo.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/tu102.o
  HDRTEST drivers/gpu/drm/xe/xe_macros.h
  HDRTEST drivers/gpu/drm/xe/xe_map.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_mqd_manager_v11.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_kernel_queue.o
  HDRTEST drivers/gpu/drm/xe/xe_migrate.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_packet_manager.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_packet_manager_vi.o
  CC [M]  drivers/gpu/drm/i915/display/intel_snps_phy.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/mem.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_packet_manager_v9.o
  HDRTEST drivers/gpu/drm/xe/xe_migrate_doc.h
  CC [M]  drivers/gpu/drm/i915/display/intel_tv.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_process_queue_manager.o
  HDRTEST drivers/gpu/drm/xe/xe_mmio.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager.o
  HDRTEST drivers/gpu/drm/xe/xe_mocs.h
  CC [M]  drivers/gpu/drm/i915/display/intel_vdsc.o
  CC [M]  drivers/gpu/drm/i915/display/intel_vrr.o
  HDRTEST drivers/gpu/drm/xe/xe_module.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager_cik.o
  CC [M]  drivers/gpu/drm/i915/display/vlv_dsi.o
  HDRTEST drivers/gpu/drm/xe/xe_pat.h
  CC [M]  drivers/gpu/drm/i915/display/vlv_dsi_pll.o
  CC [M]  drivers/gpu/drm/i915/i915_perf.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/memnv04.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/memnv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/memgf100.o
  HDRTEST drivers/gpu/drm/xe/xe_pci.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager_vi.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager_v9.o
  HDRTEST drivers/gpu/drm/xe/xe_pci_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.o
  HDRTEST drivers/gpu/drm/xe/xe_pcode.h
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager_v10.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_tee.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv04.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_huc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv41.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_device_queue_manager_v11.o
  HDRTEST drivers/gpu/drm/xe/xe_pcode_api.h
  HDRTEST drivers/gpu/drm/xe/xe_platform_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_interrupt.o
  HDRTEST drivers/gpu/drm/xe/xe_pm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv44.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmnv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_events.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmmcp77.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgf100.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_cmd.o
  HDRTEST drivers/gpu/drm/xe/xe_pmu.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/cik_event_interrupt.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_int_process_v9.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_int_process_v10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgk104.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_debugfs.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_int_process_v11.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_smi_events.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_gsccs.o
  HDRTEST drivers/gpu/drm/xe/xe_pmu_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_crat.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_irq.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_debug.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgm20b.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_debugfs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_svm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../amdkfd/kfd_migrate.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_pm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_arcturus.o
  CC [M]  drivers/gpu/drm/i915/pxp/intel_pxp_session.o
  HDRTEST drivers/gpu/drm/xe/xe_preempt_fence.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_aldebaran.o
  CC [M]  drivers/gpu/drm/i915/i915_gpu_error.o
  CC [M]  drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgp100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gc_9_4_3.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgv100.o
  HDRTEST drivers/gpu/drm/xe/xe_preempt_fence_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v10.o
  CC [M]  drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.o
  CC [M]  drivers/gpu/drm/i915/selftests/intel_scheduler_helpers.o
  HDRTEST drivers/gpu/drm/xe/xe_pt.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v10_3.o
  HDRTEST drivers/gpu/drm/xe/xe_pt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_pt_walk.h
  CC [M]  drivers/gpu/drm/i915/selftests/i915_random.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmtu102.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v11.o
  HDRTEST drivers/gpu/drm/xe/xe_query.h
  HDRTEST drivers/gpu/drm/xe/xe_range_fence.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/umem.o
  CC [M]  drivers/gpu/drm/i915/selftests/i915_selftest.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/ummu.o
  CC [M]  drivers/gpu/drm/i915/selftests/igt_atomic.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.o
  CC [M]  drivers/gpu/drm/i915/selftests/igt_flush_test.o
  HDRTEST drivers/gpu/drm/xe/xe_reg_sr.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.o
  HDRTEST drivers/gpu/drm/xe/xe_reg_sr_types.h
  CC [M]  drivers/gpu/drm/i915/selftests/igt_live_test.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_job.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_acp.o
  CC [M]  drivers/gpu/drm/i915/selftests/igt_mmap.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mxm/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../acp/acp_hw.o
  HDRTEST drivers/gpu/drm/xe/xe_reg_whitelist.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_ioc32.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.o
  HDRTEST drivers/gpu/drm/xe/xe_res_cursor.h
  CC [M]  drivers/gpu/drm/i915/selftests/igt_reset.o
  CC [M]  drivers/gpu/drm/i915/selftests/igt_spinner.o
  HDRTEST drivers/gpu/drm/xe/xe_ring_ops.h
  CC [M]  drivers/gpu/drm/i915/selftests/librapl.o
  HDRTEST drivers/gpu/drm/xe/xe_ring_ops_types.h
  HDRTEST drivers/gpu/drm/xe/xe_rtp.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mxm/mxms.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.o
  HDRTEST drivers/gpu/drm/xe/xe_rtp_types.h
  CC [M]  drivers/gpu/drm/i915/i915_vgpu.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/arcturus_ppt.o
  HDRTEST drivers/gpu/drm/xe/xe_sa.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/mxm/nv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/navi10_ppt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/agp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/sienna_cichlid_ppt.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/vangogh_ppt.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dkl_phy_regs.h
  HDRTEST drivers/gpu/drm/xe/xe_sa_types.h
  HDRTEST drivers/gpu/drm/xe/xe_sched_job.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.o
  HDRTEST drivers/gpu/drm/i915/display/intel_crtc_state_dump.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/pcie.o
  HDRTEST drivers/gpu/drm/i915/display/hsw_ips.h
  HDRTEST drivers/gpu/drm/xe/xe_sched_job_types.h
  HDRTEST drivers/gpu/drm/i915/display/g4x_hdmi.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/cyan_skillfish_ppt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/nv04.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu11/smu_v11_0.o
  HDRTEST drivers/gpu/drm/i915/display/intel_hdcp_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu12/renoir_ppt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/nv40.o
  HDRTEST drivers/gpu/drm/i915/display/intel_overlay.h
  HDRTEST drivers/gpu/drm/i915/display/intel_display.h
  HDRTEST drivers/gpu/drm/xe/xe_step.h
  HDRTEST drivers/gpu/drm/xe/xe_step_types.h
  HDRTEST drivers/gpu/drm/i915/display/skl_watermark_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dmc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu12/smu_v12_0.o
  HDRTEST drivers/gpu/drm/xe/xe_sync.h
  HDRTEST drivers/gpu/drm/i915/display/intel_vga.h
  HDRTEST drivers/gpu/drm/xe/xe_sync_types.h
  HDRTEST drivers/gpu/drm/xe/xe_tile.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0.o
  HDRTEST drivers/gpu/drm/xe/xe_tile_sysfs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_audio.h
  HDRTEST drivers/gpu/drm/i915/display/intel_lvds.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/aldebaran_ppt.o
  HDRTEST drivers/gpu/drm/xe/xe_tile_sysfs_types.h
  HDRTEST drivers/gpu/drm/i915/display/intel_modeset_setup.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/nv46.o
  HDRTEST drivers/gpu/drm/xe/xe_trace.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/nv4c.o
  HDRTEST drivers/gpu/drm/i915/display/intel_cdclk.h
  HDRTEST drivers/gpu/drm/i915/display/intel_display_limits.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_stolen_mgr.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/yellow_carp_ppt.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0_0_ppt.o
  HDRTEST drivers/gpu/drm/xe/xe_ttm_sys_mgr.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/g84.o
  HDRTEST drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0_4_ppt.o
  HDRTEST drivers/gpu/drm/i915/display/intel_hotplug.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0_5_ppt.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0_7_ppt.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/g92.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu13/smu_v13_0_6_ppt.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dkl_phy.h
  HDRTEST drivers/gpu/drm/i915/display/intel_atomic.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/g94.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/amdgpu_smu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/gf100.o
  HDRTEST drivers/gpu/drm/i915/display/intel_color_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/swsmu/smu_cmn.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_driver.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dpll.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/gf106.o
  HDRTEST drivers/gpu/drm/xe/xe_tuning.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/gk104.o
  HDRTEST drivers/gpu/drm/xe/xe_uc.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_debugfs.h
  HDRTEST drivers/gpu/drm/i915/display/vlv_dsi_pll_regs.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_mst.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw_abi.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pci/gp100.o
  HDRTEST drivers/gpu/drm/xe/xe_uc_types.h
  HDRTEST drivers/gpu/drm/i915/display/intel_fdi_regs.h
  HDRTEST drivers/gpu/drm/i915/display/g4x_dp.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu8_smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_tc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/tonga_smumgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/memx.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_params.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gt215.o
  HDRTEST drivers/gpu/drm/xe/xe_vm.h
  HDRTEST drivers/gpu/drm/i915/display/intel_frontbuffer.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/fiji_smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dsi_vbt.h
  HDRTEST drivers/gpu/drm/i915/display/intel_psr.h
  HDRTEST drivers/gpu/drm/i915/display/intel_crt.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_doc.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_madvise.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/polaris10_smumgr.o
  HDRTEST drivers/gpu/drm/xe/xe_wa.h
  HDRTEST drivers/gpu/drm/i915/display/intel_opregion.h
  HDRTEST drivers/gpu/drm/i915/display/intel_snps_phy_regs.h
  HDRTEST drivers/gpu/drm/xe/xe_wait_user_fence.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gf100.o
  HDRTEST drivers/gpu/drm/i915/display/i9xx_wm.h
  HDRTEST drivers/gpu/drm/i915/display/intel_cx0_phy_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_global_state.h
  HDRTEST drivers/gpu/drm/xe/xe_wopcm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gf119.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gk104.o
  HDRTEST drivers/gpu/drm/xe/xe_wopcm_types.h
  HDRTEST drivers/gpu/drm/i915/display/intel_lpe_audio.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/iceland_smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_drrs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gk110.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu7_smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_rps.h
  LD [M]  drivers/gpu/drm/xe/xe.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gk208.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/vega10_smumgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gk20a.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu10_smumgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gm107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/ci_smumgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/vega12_smumgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_fbdev.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/vegam_smumgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gm20b.o
  HDRTEST drivers/gpu/drm/i915/display/intel_pps_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu9_smumgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gp102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/pmu/gp10b.o
  HDRTEST drivers/gpu/drm/i915/display/intel_hdmi.h
  HDRTEST drivers/gpu/drm/i915/display/intel_fdi.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/vega20_smumgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/hwmgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/processpptables.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gf100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gf117.o
  HDRTEST drivers/gpu/drm/i915/display/intel_fb.h
  HDRTEST drivers/gpu/drm/i915/display/intel_qp_tables.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dsb_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/hardwaremanager.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu8_hwmgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/pppcielanes.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/process_pptables_v1_0.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/ppatomctrl.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/ppatomfwctrl.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu7_hwmgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu7_powertune.o
  HDRTEST drivers/gpu/drm/i915/display/intel_vdsc.h
  HDRTEST drivers/gpu/drm/i915/display/intel_snps_phy.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu7_thermal.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_core.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gk20a.o
  HDRTEST drivers/gpu/drm/i915/display/vlv_dsi_pll.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu7_clockpowergating.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega10_processpptables.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega10_hwmgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/privring/gp10b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/base.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dvo_dev.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega10_powertune.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega10_thermal.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/fan.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/fannil.o
  HDRTEST drivers/gpu/drm/i915/display/intel_hdcp.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/fanpwm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu10_hwmgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_sdvo_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/fantog.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/ic.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/temp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/pp_psm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/nv40.o
  HDRTEST drivers/gpu/drm/i915/display/intel_pch_refclk.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega12_processpptables.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/nv50.o
  HDRTEST drivers/gpu/drm/i915/display/intel_modeset_lock.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/g84.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_trace.h
  HDRTEST drivers/gpu/drm/i915/display/intel_display_power.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega12_hwmgr.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_aux_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gt215.o
  HDRTEST drivers/gpu/drm/i915/display/i9xx_plane.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega12_thermal.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/pp_overdriver.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gf119.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu_helper.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega20_processpptables.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_aux_backlight.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega20_hwmgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega20_powertune.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega20_thermal.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gm107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/common_baco.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dpll_mgr.h
  HDRTEST drivers/gpu/drm/i915/display/vlv_dsi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega10_baco.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/therm/gp100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega20_baco.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/timer/base.o
  HDRTEST drivers/gpu/drm/i915/display/intel_plane_initial.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/timer/nv04.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/timer/nv40.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/timer/nv41.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/vega12_baco.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu9_baco.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/tonga_baco.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/timer/gk20a.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_device.h
  HDRTEST drivers/gpu/drm/i915/display/intel_fifo_underrun.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/polaris_baco.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/fiji_baco.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/top/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/ci_baco.o
  HDRTEST drivers/gpu/drm/i915/display/intel_cursor.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/hwmgr/smu7_baco.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/powerplay/amd_powerplay.o
  HDRTEST drivers/gpu/drm/i915/display/vlv_dsi_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/legacy-dpm/legacy_dpm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/top/gk104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/top/ga100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/vfn/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/vfn/uvfn.o
  HDRTEST drivers/gpu/drm/i915/display/intel_cx0_phy.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/legacy-dpm/kv_dpm.o
  HDRTEST drivers/gpu/drm/i915/display/skl_scaler.h
  HDRTEST drivers/gpu/drm/i915/display/intel_hti.h
  HDRTEST drivers/gpu/drm/i915/display/icl_dsi_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/vfn/gv100.o
  HDRTEST drivers/gpu/drm/i915/display/intel_atomic_plane.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/vfn/tu102.o
  HDRTEST drivers/gpu/drm/i915/display/skl_watermark.h
  HDRTEST drivers/gpu/drm/i915/display/intel_fbc.h
  HDRTEST drivers/gpu/drm/i915/display/intel_display_reg_defs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/vfn/ga100.o
  HDRTEST drivers/gpu/drm/i915/display/intel_acpi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gpio.o
  HDRTEST drivers/gpu/drm/i915/display/intel_connector.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/legacy-dpm/kv_smc.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/legacy-dpm/si_dpm.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dpt.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/nv40.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/legacy-dpm/si_smc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gf100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gf117.o
  HDRTEST drivers/gpu/drm/i915/display/intel_quirks.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_link_training.h
  HDRTEST drivers/gpu/drm/i915/display/intel_color.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/amdgpu_dpm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/amdgpu_pm.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../pm/amdgpu_dpm_internal.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gk104.o
  HDRTEST drivers/gpu/drm/i915/display/intel_crtc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/subdev/volt/gm20b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/falcon.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_plane.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_crtc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/xtensa.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_debugfs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_irq.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_mst_types.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_color.o
  HDRTEST drivers/gpu/drm/i915/display/intel_modeset_verify.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/dc_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/bsp/g84.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gt215.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_services.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gk104.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_power_well.h
  HDRTEST drivers/gpu/drm/i915/display/intel_psr_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_helpers.o
  HDRTEST drivers/gpu/drm/i915/display/intel_wm.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_pp_smu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gm107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_psr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gm200.o
  HDRTEST drivers/gpu/drm/i915/display/intel_pipe_crc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_replay.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_hdcp.o
  HDRTEST drivers/gpu/drm/i915/display/intel_audio_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_panel.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_crc.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gp100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gp102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/gv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm_debugfs.o
  HDRTEST drivers/gpu/drm/i915/display/intel_sprite.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/tu102.o
  HDRTEST drivers/gpu/drm/i915/display/intel_wm_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/ga100.o
  HDRTEST drivers/gpu/drm/i915/display/intel_tv.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/basics/conversion.o
  HDRTEST drivers/gpu/drm/i915/display/intel_hti_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_vrr.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/basics/fixpt31_32.o
  HDRTEST drivers/gpu/drm/i915/display/intel_load_detect.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/ce/ga102.o
  HDRTEST drivers/gpu/drm/i915/display/skl_universal_plane.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/cipher/g84.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/basics/vector.o
  HDRTEST drivers/gpu/drm/i915/display/intel_mg_phy_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/basics/dc_common.o
  HDRTEST drivers/gpu/drm/i915/display/intel_bw.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/bios_parser.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_irq.h
  HDRTEST drivers/gpu/drm/i915/display/intel_de.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/acpi.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/bios_parser_interface.o
  HDRTEST drivers/gpu/drm/i915/display/intel_lvds_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/bios_parser_helper.o
  HDRTEST drivers/gpu/drm/i915/display/intel_gmbus_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/command_table.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/ctrl.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dsi_dcs_backlight.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/pci.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dvo.h
  HDRTEST drivers/gpu/drm/i915/display/intel_sdvo.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/command_table_helper.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_aux.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/device/user.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/bios_parser_common.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/command_table2.o
  HDRTEST drivers/gpu/drm/i915/display/intel_vdsc_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/command_table_helper2.o
  HDRTEST drivers/gpu/drm/i915/display/intel_combo_phy.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/chan.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/conn.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dvo_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/bios_parser2.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/dce60/command_table_helper_dce60.o
  HDRTEST drivers/gpu/drm/i915/display/intel_gmbus.h
  HDRTEST drivers/gpu/drm/i915/display/intel_hdcp_gsc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/dp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/dce80/command_table_helper_dce80.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dsi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/hdmi.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dmc_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_ddi.h
  HDRTEST drivers/gpu/drm/i915/display/intel_hotplug_irq.h
  HDRTEST drivers/gpu/drm/i915/display/intel_tv_regs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/dce110/command_table_helper_dce110.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dsb.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/head.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/ior.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/dce112/command_table_helper_dce112.o
  HDRTEST drivers/gpu/drm/i915/display/intel_bios.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/outp.o
  HDRTEST drivers/gpu/drm/i915/display/intel_pch_display.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/bios/dce112/command_table_helper2_dce112.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/dce_calcs.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/custom_float.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/vga.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_types.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/bw_fixed.o
  HDRTEST drivers/gpu/drm/i915/display/intel_backlight.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/nv04.o
  HDRTEST drivers/gpu/drm/i915/display/intel_vblank.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/display_mode_lib.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/display_rq_dlg_helpers.o
  HDRTEST drivers/gpu/drm/i915/display/intel_dp.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dml1_display_rq_dlg_calc.o
  HDRTEST drivers/gpu/drm/i915/display/intel_pmdemand.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn10/dcn10_fpu.o
  HDRTEST drivers/gpu/drm/i915/display/intel_backlight_regs.h
  HDRTEST drivers/gpu/drm/i915/display/intel_combo_phy_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/nv50.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/dcn20_fpu.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_reset.h
  HDRTEST drivers/gpu/drm/i915/display/intel_display_power_map.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/g84.o
  HDRTEST drivers/gpu/drm/i915/display/intel_ddi_buf_trans.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/g94.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_wa.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gt200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/display_mode_vba.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_rq_dlg_calc_20.o
  HDRTEST drivers/gpu/drm/i915/display/icl_dsi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/mcp77.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gt215.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_rq_dlg_calc_20v2.o
  HDRTEST drivers/gpu/drm/i915/display/intel_lspcon.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dpio_phy.h
  HDRTEST drivers/gpu/drm/i915/display/intel_dp_hdcp.h
  HDRTEST drivers/gpu/drm/i915/display/intel_fb_pin.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/mcp89.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gf119.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gk104.o
  HDRTEST drivers/gpu/drm/i915/display/intel_display_debugfs_params.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gk110.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gm107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn20/display_mode_vba_20v2.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn21/display_rq_dlg_calc_21.o
  HDRTEST drivers/gpu/drm/i915/display/intel_link_bw.h
  HDRTEST drivers/gpu/drm/i915/display/intel_pps.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn21/display_mode_vba_21.o
  HDRTEST drivers/gpu/drm/i915/display/intel_sprite_uapi.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_ttm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn30/dcn30_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gp100.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_region.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn30/display_mode_vba_30.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn30/display_rq_dlg_calc_30.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gp102.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_context_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/gv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn31/display_mode_vba_31.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_lmem.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/tu102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/ga102.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn31/display_rq_dlg_calc_31.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/udisp.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_mman.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/uconn.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn314/display_mode_vba_314.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/uoutp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn314/display_rq_dlg_calc_314.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn32/display_mode_vba_32.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_object_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/disp/uhead.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn32/display_rq_dlg_calc_32.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/nv04.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn32/display_mode_vba_util_32.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn31/dcn31_fpu.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_context.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/gf100.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_clflush.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/gf119.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/gv100.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_tiling.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_stolen.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/user.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_ttm_pm.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_create.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/usernv04.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_ttm_move.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/usernv50.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_ioctls.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_domain.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/usergf100.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_internal.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_dmabuf.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/usergf119.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn32/dcn32_fpu.o
  HDRTEST drivers/gpu/drm/i915/gem/selftests/mock_context.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/dma/usergv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn321/dcn321_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/base.o
  HDRTEST drivers/gpu/drm/i915/gem/selftests/huge_gem_object.h
  HDRTEST drivers/gpu/drm/i915/gem/selftests/mock_gem_object.h
  HDRTEST drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/cgrp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn301/dcn301_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/chan.o
  HDRTEST drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/runl.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn302/dcn302_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/runq.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/nv04.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn303/dcn303_fpu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/nv10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/nv17.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/nv40.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_userptr.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/g84.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dcn314/dcn314_fpu.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_pm.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_shrinker.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/g98.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gemfs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/dsc/rc_calc_fpu.o
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_object.h
  HDRTEST drivers/gpu/drm/i915/gem/i915_gem_object_frontbuffer.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_timeline_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/dcn_calcs.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk110.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/dcn_calc_math.o
  HDRTEST drivers/gpu/drm/i915/gt/selftest_engine.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk208.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dml/calcs/dcn_calc_auto.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gk20a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gm107.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gp100.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_breadcrumbs.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_engine_heartbeat.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_context_types.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_execlists_submission.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/gv100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/tu102.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_pm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/ga100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/ga102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/ucgrp.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dce60/dce60_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/selftest_rc6.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/fifo/uchan.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dce100/dce_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dce110/dce110_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_llc_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv04.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv10.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv15.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv17.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dce112/dce112_clk_mgr.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dce120/dce120_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv20.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_region_lmem.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn10/rv1_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_requests.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv25.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_ggtt_gmch.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_print.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv2a.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv30.o
  HDRTEST drivers/gpu/drm/i915/gt/gen8_ppgtt.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv34.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn10/rv2_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv35.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv40.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_mcr.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_timeline.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv44.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn20/dcn20_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/gen6_engine_cs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/nv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/g84.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gt200.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/mcp79.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gt215.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_pm_debugfs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn201/dcn201_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/mcp89.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf100.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_workarounds_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf108.o
  HDRTEST drivers/gpu/drm/i915/gt/selftest_rps.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn21/rn_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_sa_media.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf110.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_debugfs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn30/dcn30_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_clock_utils.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_rps_types.h
  HDRTEST drivers/gpu/drm/i915/gt/selftest_engine_heartbeat.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf117.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn30/dcn30_clk_mgr_smu_msg.o
  HDRTEST drivers/gpu/drm/i915/gt/sysfs_engines.h
  HDRTEST drivers/gpu/drm/i915/gt/gen7_renderclear.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gf119.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_context.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn301/vg_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_wopcm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn301/dcn301_smu.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_mocs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_engine_pm.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gk110b.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn31/dcn31_smu.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_sysfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gk208.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_rc6.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn31/dcn31_clk_mgr.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_defines.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn314/dcn314_smu.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_ring_types.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_workarounds.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn314/dcn314_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gk20a.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_engine_regs.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_pm_irq.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gm107.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gm200.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gm20b.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp100.o
  HDRTEST drivers/gpu/drm/i915/gt/shmem_utils.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_engine.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn315/dcn315_smu.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_reset_types.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_regs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp107.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_reset.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp108.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn315/dcn315_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gp10b.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_fw.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/gv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn316/dcn316_smu.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn316/dcn316_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/tu102.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/guc_capture_fwif.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ga102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxnv40.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxnv50.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf100.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_uc.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_uc_fw_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn32/dcn32_clk_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf108.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_print.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_fw.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_debugfs.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_klvs_abi.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/clk_mgr/dcn32/dcn32_clk_mgr_smu_msg.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_audio.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_errors_abi.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_stream_encoder.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_actions_slpc_abi.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_communication_mmio_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf110.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_link_encoder.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_hwseq.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf117.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_actions_abi.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgf119.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_mem_input.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_communication_ctb_abi.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/abi/guc_messages_abi.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_clock_source.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_uc_debugfs.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgk104.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_scl_filters.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgk110.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_uc_heci_cmd_submit.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_transform.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgk110b.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_reg.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_uc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgk208.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_huc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgk20a.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_binary_headers.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgm107.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_opp.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_huc_print.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_huc_fw.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgm200.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_dmcu.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgm20b.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgp100.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgp102.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_capture.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_abm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgp104.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgp107.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_ipp.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_log_debugfs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_aux.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxgv100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_i2c.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_gsc_proxy.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_i2c_hw.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxtu102.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_i2c_sw.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/gr/ctxga102.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mpeg/nv31.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_submission.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mpeg/nv40.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mpeg/nv44.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_psr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mpeg/nv50.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_slpc_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mpeg/g84.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_log.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_ct.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mspdec/base.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_slpc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_abm.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mspdec/g98.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_uc_fw.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_ads.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_uc_debugfs.h
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_guc_rc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_abm_lcd.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mspdec/gt215.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dce_panel_cntl.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mspdec/gf100.o
  HDRTEST drivers/gpu/drm/i915/gt/uc/intel_huc_debugfs.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_hwconfig.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/mspdec/gk104.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_llc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_hw_lock_mgr.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msppp/base.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_outbox.o
  HDRTEST drivers/gpu/drm/i915/gt/gen8_engine_cs.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/dce/dmub_replay.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_sseu_debugfs.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_rc6_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msppp/g98.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_context_param.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/gpio_base.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gpu_commands.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msppp/gt215.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_engine_user.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_irq.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_gsc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/gpio_service.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_factory.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_rps.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_gpio.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_tlb.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msppp/gf100.o
  HDRTEST drivers/gpu/drm/i915/gt/selftest_llc.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_hpd.o
  HDRTEST drivers/gpu/drm/i915/gt/gen6_ppgtt.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_ggtt_fencing.h
  HDRTEST drivers/gpu/drm/i915/gt/intel_migrate_types.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/base.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/g98.o
  HDRTEST drivers/gpu/drm/i915/gt/selftests/mock_timeline.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/gt215.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_ddc.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_lrc.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/mcp89.o
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/gf100.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_generic.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_lrc_reg.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/hw_translate.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_migrate.h
  CC [M]  drivers/gpu/drm/nouveau/nvkm/engine/msvld/gk104.o
  HDRTEST drivers/gpu/drm/i915/gt/intel_gt_sysfs_pm.h
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/dce60/hw_translate_dce60.o
  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/dc/gpio/dce



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.Hooks: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
                   ` (4 preceding siblings ...)
  (?)
@ 2023-11-10 18:34 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:34 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

run-parts: executing /workspace/ci/hooks/00-showenv
+ pwd
+ ls -la
/workspace
total 916
drwxrwxr-x 12 1003 1003   4096 Nov 10 18:34 .
drwxr-xr-x  1 root root   4096 Nov 10 18:34 ..
-rw-rw-r--  1 1003 1003 797646 Nov 10 18:34 build.log
-rw-rw-r--  1 1003 1003   1660 Nov 10 18:25 checkpatch.log
drwxrwxr-x  5 1003 1003   4096 Nov 10 18:23 ci
drwxrwxr-x  9 1003 1003   4096 Nov 10 18:23 docker
drwxrwxr-x  8 1003 1003   4096 Nov 10 18:23 .git
-rw-rw-r--  1 1003 1003    231 Nov 10 18:25 git_apply.log
drwxrwxr-x  4 1003 1003   4096 Nov 10 18:23 .github
-rw-rw-r--  1 1003 1003    233 Nov 10 18:23 .groovylintrc.json
-rw-rw-r--  1 1003 1003     78 Nov 10 18:34 hooks.log
drwxrwxr-x 31 1003 1003   4096 Nov 10 18:34 kernel
-rw-rw-r--  1 1003 1003  30114 Nov 10 18:25 kernel.mbox
-rw-rw-r--  1 1003 1003  28749 Nov 10 18:26 kunit.log
-rw-rw-r--  1 1003 1003     48 Nov 10 18:25 parent.tag
drwxrwxr-x 44 1003 1003   4096 Nov 10 18:23 pipelines
-rw-rw-r--  1 1003 1003    793 Nov 10 18:23 README.adoc
drwxrwxr-x  3 1003 1003   4096 Nov 10 18:23 scripts
drwxrwxr-x  3 1003 1003   4096 Nov 10 18:23 src
drwxrwxr-x  2 1003 1003   4096 Nov 10 18:23 vars
drwxrwxr-x  2 1003 1003   4096 Nov 10 18:23 .vscode
+ uname -a
Linux 1da3befdaeae 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
+ export
+ grep -Ei '(^|\W)CI_'
declare -x CI_KERNEL_BUILD_DIR="/workspace/kernel/build64-default"
declare -x CI_KERNEL_SRC_DIR="/workspace/kernel"
declare -x CI_TOOLS_SRC_DIR="/workspace/ci"
declare -x CI_WORKSPACE_DIR="/workspace"
+ '[' -n /workspace ']'
+ git_args='-C /workspace/kernel'
+ git_log_args=
+ git --no-pager -C /workspace/kernel log --format=oneline --abbrev-commit
c5d52ad56 drm/doc/rfc: SR-IOV support on the new Xe driver
b3890fb43 drm/xe: Drop EXECLIST_CONTROL from error state dump
run-parts: executing /workspace/ci/hooks/10-build-W1
+ SRC_DIR=/workspace/kernel
+ RESTORE_DISPLAY_CONFIG=0
+ '[' -n /workspace/kernel/build64-default ']'
+ BUILD_DIR=/workspace/kernel/build64-default
+ cd /workspace/kernel
+ grep -q -e '^CONFIG_DRM_XE_DISPLAY=[yY]' /workspace/kernel/build64-default/.config
+ RESTORE_DISPLAY_CONFIG=1
+ trap cleanup EXIT
+ ./scripts/config --file /workspace/kernel/build64-default/.config --disable CONFIG_DRM_XE_DISPLAY
++ nproc
+ make -j48 O=/workspace/kernel/build64-default modules_prepare
make[1]: Entering directory '/workspace/kernel/build64-default'
  SYNC    include/config/auto.conf.cmd
  GEN     Makefile
  GEN     Makefile
  UPD     include/config/kernel.release
  UPD     include/generated/compile.h
  UPD     include/generated/utsrelease.h
  DESCEND objtool
  CALL    ../scripts/checksyscalls.sh
  HOSTCC  /workspace/kernel/build64-default/tools/objtool/fixdep.o
  HOSTLD  /workspace/kernel/build64-default/tools/objtool/fixdep-in.o
  LINK    /workspace/kernel/build64-default/tools/objtool/fixdep
  INSTALL libsubcmd_headers
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/exec-cmd.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/help.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/pager.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/parse-options.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/run-command.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/sigchain.o
  CC      /workspace/kernel/build64-default/tools/objtool/libsubcmd/subcmd-config.o
  LD      /workspace/kernel/build64-default/tools/objtool/libsubcmd/libsubcmd-in.o
  AR      /workspace/kernel/build64-default/tools/objtool/libsubcmd/libsubcmd.a
  CC      /workspace/kernel/build64-default/tools/objtool/weak.o
  CC      /workspace/kernel/build64-default/tools/objtool/check.o
  CC      /workspace/kernel/build64-default/tools/objtool/special.o
  CC      /workspace/kernel/build64-default/tools/objtool/builtin-check.o
  CC      /workspace/kernel/build64-default/tools/objtool/elf.o
  CC      /workspace/kernel/build64-default/tools/objtool/objtool.o
  CC      /workspace/kernel/build64-default/tools/objtool/orc_gen.o
  CC      /workspace/kernel/build64-default/tools/objtool/orc_dump.o
  CC      /workspace/kernel/build64-default/tools/objtool/libstring.o
  CC      /workspace/kernel/build64-default/tools/objtool/libctype.o
  CC      /workspace/kernel/build64-default/tools/objtool/str_error_r.o
  CC      /workspace/kernel/build64-default/tools/objtool/librbtree.o
  CC      /workspace/kernel/build64-default/tools/objtool/arch/x86/special.o
  CC      /workspace/kernel/build64-default/tools/objtool/arch/x86/decode.o
  LD      /workspace/kernel/build64-default/tools/objtool/arch/x86/objtool-in.o
  LD      /workspace/kernel/build64-default/tools/objtool/objtool-in.o
  LINK    /workspace/kernel/build64-default/tools/objtool/objtool
make[1]: Leaving directory '/workspace/kernel/build64-default'
++ nproc
+ make -j48 O=/workspace/kernel/build64-default M=drivers/gpu/drm/xe W=1
make[1]: Entering directory '/workspace/kernel/build64-default'
  CC [M]  drivers/gpu/drm/xe/xe_bb.o
  CC [M]  drivers/gpu/drm/xe/xe_bo.o
  CC [M]  drivers/gpu/drm/xe/xe_bo_evict.o
  CC [M]  drivers/gpu/drm/xe/xe_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_devcoredump.o
  CC [M]  drivers/gpu/drm/xe/xe_device.o
  CC [M]  drivers/gpu/drm/xe/xe_device_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_dma_buf.o
  CC [M]  drivers/gpu/drm/xe/xe_drm_client.o
  CC [M]  drivers/gpu/drm/xe/xe_exec.o
  CC [M]  drivers/gpu/drm/xe/xe_execlist.o
  CC [M]  drivers/gpu/drm/xe/xe_exec_queue.o
  CC [M]  drivers/gpu/drm/xe/xe_force_wake.o
  CC [M]  drivers/gpu/drm/xe/xe_ggtt.o
  CC [M]  drivers/gpu/drm/xe/xe_gt.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_clock.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_idle_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_mcr.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_pagefault.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_tlb_invalidation.o
  CC [M]  drivers/gpu/drm/xe/xe_gt_topology.o
  HOSTCC  drivers/gpu/drm/xe/xe_gen_wa_oob
  CC [M]  drivers/gpu/drm/xe/xe_guc_ads.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_ct.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_hwconfig.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_log.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_pc.o
  CC [M]  drivers/gpu/drm/xe/xe_guc_submit.o
  CC [M]  drivers/gpu/drm/xe/xe_heci_gsc.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_engine.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_engine_class_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_hw_fence.o
  CC [M]  drivers/gpu/drm/xe/xe_huc.o
  CC [M]  drivers/gpu/drm/xe/xe_huc_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_irq.o
  CC [M]  drivers/gpu/drm/xe/xe_lrc.o
  CC [M]  drivers/gpu/drm/xe/xe_mmio.o
  CC [M]  drivers/gpu/drm/xe/xe_mocs.o
  CC [M]  drivers/gpu/drm/xe/xe_module.o
  CC [M]  drivers/gpu/drm/xe/xe_pat.o
  CC [M]  drivers/gpu/drm/xe/xe_pci.o
  CC [M]  drivers/gpu/drm/xe/xe_pcode.o
  CC [M]  drivers/gpu/drm/xe/xe_pm.o
  CC [M]  drivers/gpu/drm/xe/xe_preempt_fence.o
  CC [M]  drivers/gpu/drm/xe/xe_pt.o
  CC [M]  drivers/gpu/drm/xe/xe_pt_walk.o
  CC [M]  drivers/gpu/drm/xe/xe_query.o
  CC [M]  drivers/gpu/drm/xe/xe_range_fence.o
  CC [M]  drivers/gpu/drm/xe/xe_reg_sr.o
  CC [M]  drivers/gpu/drm/xe/xe_reg_whitelist.o
  CC [M]  drivers/gpu/drm/xe/xe_rtp.o
  CC [M]  drivers/gpu/drm/xe/xe_sa.o
  CC [M]  drivers/gpu/drm/xe/xe_sched_job.o
  CC [M]  drivers/gpu/drm/xe/xe_step.o
  CC [M]  drivers/gpu/drm/xe/xe_sync.o
  CC [M]  drivers/gpu/drm/xe/xe_tile.o
  CC [M]  drivers/gpu/drm/xe/xe_tile_sysfs.o
  CC [M]  drivers/gpu/drm/xe/xe_trace.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_sys_mgr.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_vram_mgr.o
  CC [M]  drivers/gpu/drm/xe/xe_tuning.o
  CC [M]  drivers/gpu/drm/xe/xe_uc.o
  CC [M]  drivers/gpu/drm/xe/xe_uc_debugfs.o
  CC [M]  drivers/gpu/drm/xe/xe_uc_fw.o
  CC [M]  drivers/gpu/drm/xe/xe_vm_madvise.o
  CC [M]  drivers/gpu/drm/xe/xe_wait_user_fence.o
  CC [M]  drivers/gpu/drm/xe/xe_wopcm.o
  CC [M]  drivers/gpu/drm/xe/xe_hwmon.o
  CC [M]  drivers/gpu/drm/xe/xe_pmu.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_klvs_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_errors_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_actions_slpc_abi.h
  CC [M]  drivers/gpu/drm/xe/tests/xe_bo_test.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_communication_mmio_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_actions_abi.h
  HDRTEST drivers/gpu/drm/xe/abi/guc_communication_ctb_abi.h
  CC [M]  drivers/gpu/drm/xe/tests/xe_dma_buf_test.o
  HDRTEST drivers/gpu/drm/xe/abi/guc_messages_abi.h
  HDRTEST drivers/gpu/drm/xe/instructions/xe_instr_defs.h
  CC [M]  drivers/gpu/drm/xe/tests/xe_migrate_test.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_pci_test.o
  HDRTEST drivers/gpu/drm/xe/instructions/xe_gfxpipe_commands.h
  CC [M]  drivers/gpu/drm/xe/tests/xe_rtp_test.o
  HDRTEST drivers/gpu/drm/xe/instructions/xe_mi_commands.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_reg_defs.h
  CC [M]  drivers/gpu/drm/xe/tests/xe_wa_test.o
  HDRTEST drivers/gpu/drm/xe/regs/xe_guc_regs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_gt_regs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_regs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_gpu_commands.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_lrc_layout.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_mchbar_regs.h
  HDRTEST drivers/gpu/drm/xe/regs/xe_engine_regs.h
  HDRTEST drivers/gpu/drm/xe/tests/xe_test.h
  HDRTEST drivers/gpu/drm/xe/tests/xe_pci_test.h
  HDRTEST drivers/gpu/drm/xe/tests/xe_migrate_test.h
  HDRTEST drivers/gpu/drm/xe/tests/xe_dma_buf_test.h
  HDRTEST drivers/gpu/drm/xe/tests/xe_bo_test.h
  HDRTEST drivers/gpu/drm/xe/xe_assert.h
  HDRTEST drivers/gpu/drm/xe/xe_bb.h
  HDRTEST drivers/gpu/drm/xe/xe_bb_types.h
  HDRTEST drivers/gpu/drm/xe/xe_bo.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_doc.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_evict.h
  HDRTEST drivers/gpu/drm/xe/xe_bo_types.h
  HDRTEST drivers/gpu/drm/xe/xe_debugfs.h
  HDRTEST drivers/gpu/drm/xe/xe_devcoredump.h
  HDRTEST drivers/gpu/drm/xe/xe_devcoredump_types.h
  HDRTEST drivers/gpu/drm/xe/xe_device.h
  HDRTEST drivers/gpu/drm/xe/xe_device_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_device_types.h
  HDRTEST drivers/gpu/drm/xe/xe_dma_buf.h
  HDRTEST drivers/gpu/drm/xe/xe_drm_client.h
  HDRTEST drivers/gpu/drm/xe/xe_drv.h
  HDRTEST drivers/gpu/drm/xe/xe_exec.h
  HDRTEST drivers/gpu/drm/xe/xe_exec_queue.h
  HDRTEST drivers/gpu/drm/xe/xe_exec_queue_types.h
  HDRTEST drivers/gpu/drm/xe/xe_execlist.h
  HDRTEST drivers/gpu/drm/xe/xe_execlist_types.h
  HDRTEST drivers/gpu/drm/xe/xe_force_wake.h
  HDRTEST drivers/gpu/drm/xe/xe_force_wake_types.h
  HDRTEST drivers/gpu/drm/xe/xe_ggtt.h
  HDRTEST drivers/gpu/drm/xe/xe_ggtt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_gt.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_clock.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_debugfs.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_idle_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_idle_sysfs_types.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_mcr.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_pagefault.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_printk.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_sysfs_types.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_tlb_invalidation_types.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_topology.h
  HDRTEST drivers/gpu/drm/xe/xe_gt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_ads.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_ads_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_ct.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_ct_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_debugfs.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_fwif.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_hwconfig.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_log.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_log_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_pc.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_pc_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_submit.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_submit_types.h
  HDRTEST drivers/gpu/drm/xe/xe_guc_types.h
  HDRTEST drivers/gpu/drm/xe/xe_heci_gsc.h
  HDRTEST drivers/gpu/drm/xe/xe_huc.h
  HDRTEST drivers/gpu/drm/xe/xe_huc_debugfs.h
  HDRTEST drivers/gpu/drm/xe/xe_huc_types.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine_class_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_engine_types.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_fence.h
  HDRTEST drivers/gpu/drm/xe/xe_hw_fence_types.h
  HDRTEST drivers/gpu/drm/xe/xe_hwmon.h
  HDRTEST drivers/gpu/drm/xe/xe_irq.h
  HDRTEST drivers/gpu/drm/xe/xe_lrc.h
  HDRTEST drivers/gpu/drm/xe/xe_lrc_types.h
  HDRTEST drivers/gpu/drm/xe/xe_macros.h
  HDRTEST drivers/gpu/drm/xe/xe_map.h
  HDRTEST drivers/gpu/drm/xe/xe_migrate.h
  HDRTEST drivers/gpu/drm/xe/xe_migrate_doc.h
  HDRTEST drivers/gpu/drm/xe/xe_mmio.h
  HDRTEST drivers/gpu/drm/xe/xe_mocs.h
  HDRTEST drivers/gpu/drm/xe/xe_module.h
  HDRTEST drivers/gpu/drm/xe/xe_pat.h
  HDRTEST drivers/gpu/drm/xe/xe_pci.h
  HDRTEST drivers/gpu/drm/xe/xe_pci_types.h
  HDRTEST drivers/gpu/drm/xe/xe_pcode.h
  HDRTEST drivers/gpu/drm/xe/xe_pcode_api.h
  HDRTEST drivers/gpu/drm/xe/xe_platform_types.h
  HDRTEST drivers/gpu/drm/xe/xe_pm.h
  HDRTEST drivers/gpu/drm/xe/xe_pmu.h
  HDRTEST drivers/gpu/drm/xe/xe_pmu_types.h
  HDRTEST drivers/gpu/drm/xe/xe_preempt_fence.h
  HDRTEST drivers/gpu/drm/xe/xe_preempt_fence_types.h
  HDRTEST drivers/gpu/drm/xe/xe_pt.h
  HDRTEST drivers/gpu/drm/xe/xe_pt_types.h
  HDRTEST drivers/gpu/drm/xe/xe_pt_walk.h
  HDRTEST drivers/gpu/drm/xe/xe_query.h
  HDRTEST drivers/gpu/drm/xe/xe_range_fence.h
  HDRTEST drivers/gpu/drm/xe/xe_reg_sr.h
  HDRTEST drivers/gpu/drm/xe/xe_reg_sr_types.h
  HDRTEST drivers/gpu/drm/xe/xe_reg_whitelist.h
  HDRTEST drivers/gpu/drm/xe/xe_res_cursor.h
  HDRTEST drivers/gpu/drm/xe/xe_ring_ops.h
  HDRTEST drivers/gpu/drm/xe/xe_ring_ops_types.h
  HDRTEST drivers/gpu/drm/xe/xe_rtp.h
  HDRTEST drivers/gpu/drm/xe/xe_rtp_types.h
  HDRTEST drivers/gpu/drm/xe/xe_sa.h
  HDRTEST drivers/gpu/drm/xe/xe_sa_types.h
  HDRTEST drivers/gpu/drm/xe/xe_sched_job.h
  HDRTEST drivers/gpu/drm/xe/xe_sched_job_types.h
  HDRTEST drivers/gpu/drm/xe/xe_step.h
  HDRTEST drivers/gpu/drm/xe/xe_step_types.h
  HDRTEST drivers/gpu/drm/xe/xe_sync.h
  HDRTEST drivers/gpu/drm/xe/xe_sync_types.h
  HDRTEST drivers/gpu/drm/xe/xe_tile.h
  HDRTEST drivers/gpu/drm/xe/xe_tile_sysfs.h
  HDRTEST drivers/gpu/drm/xe/xe_tile_sysfs_types.h
  HDRTEST drivers/gpu/drm/xe/xe_trace.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_stolen_mgr.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_sys_mgr.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
  HDRTEST drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
  HDRTEST drivers/gpu/drm/xe/xe_tuning.h
  HDRTEST drivers/gpu/drm/xe/xe_uc.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_debugfs.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw_abi.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_fw_types.h
  HDRTEST drivers/gpu/drm/xe/xe_uc_types.h
  HDRTEST drivers/gpu/drm/xe/xe_vm.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_doc.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_madvise.h
  HDRTEST drivers/gpu/drm/xe/xe_vm_types.h
  HDRTEST drivers/gpu/drm/xe/xe_wa.h
  HDRTEST drivers/gpu/drm/xe/xe_wait_user_fence.h
  HDRTEST drivers/gpu/drm/xe/xe_wopcm.h
  HDRTEST drivers/gpu/drm/xe/xe_wopcm_types.h
  GEN     xe_wa_oob.c xe_wa_oob.h
  GEN     xe_wa_oob.c xe_wa_oob.h
  CC [M]  drivers/gpu/drm/xe/xe_guc.o
  CC [M]  drivers/gpu/drm/xe/xe_migrate.o
  CC [M]  drivers/gpu/drm/xe/xe_ring_ops.o
  CC [M]  drivers/gpu/drm/xe/xe_ttm_stolen_mgr.o
  CC [M]  drivers/gpu/drm/xe/xe_vm.o
  CC [M]  drivers/gpu/drm/xe/xe_wa.o
  LD [M]  drivers/gpu/drm/xe/xe.o
  MODPOST drivers/gpu/drm/xe/Module.symvers
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_bo_test.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_dma_buf_test.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_migrate_test.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_pci_test.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_rtp_test.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/xe/tests/xe_wa_test.o
  CC [M]  drivers/gpu/drm/xe/xe.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_bo_test.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_dma_buf_test.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_migrate_test.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_pci_test.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_rtp_test.mod.o
  CC [M]  drivers/gpu/drm/xe/tests/xe_wa_test.mod.o
  LD [M]  drivers/gpu/drm/xe/tests/xe_wa_test.ko
  LD [M]  drivers/gpu/drm/xe/tests/xe_dma_buf_test.ko
  LD [M]  drivers/gpu/drm/xe/tests/xe_migrate_test.ko
  LD [M]  drivers/gpu/drm/xe/tests/xe_pci_test.ko
  LD [M]  drivers/gpu/drm/xe/tests/xe_rtp_test.ko
  LD [M]  drivers/gpu/drm/xe/xe.ko
  LD [M]  drivers/gpu/drm/xe/tests/xe_bo_test.ko
make[1]: Leaving directory '/workspace/kernel/build64-default'
+ cleanup
+ '[' 1 -eq 1 ']'
+ ./scripts/config --file /workspace/kernel/build64-default/.config --enable CONFIG_DRM_XE_DISPLAY
run-parts: executing /workspace/ci/hooks/20-kernel-doc
+ SRC_DIR=/workspace/kernel
+ cd /workspace/kernel
+ find drivers/gpu/drm/xe/ -name '*.[ch]' -not -path 'drivers/gpu/drm/xe/display/*'
+ xargs ./scripts/kernel-doc -Werror -none include/uapi/drm/xe_drm.h
All hooks done



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.checksparse: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
                   ` (5 preceding siblings ...)
  (?)
@ 2023-11-10 18:36 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 18:36 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

+ trap cleanup EXIT
+ KERNEL=/kernel
+ MT=/root/linux/maintainer-tools
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools /root/linux/maintainer-tools
Cloning into '/root/linux/maintainer-tools'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ make -C /root/linux/maintainer-tools
make: Entering directory '/root/linux/maintainer-tools'
cc -O2 -g -Wextra -o remap-log remap-log.c
make: Leaving directory '/root/linux/maintainer-tools'
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ /root/linux/maintainer-tools/dim sparse --fast b3890fb4386dcef68a96888141c4cc773f6241ce
Sparse version: 0.6.1 (Ubuntu: 0.6.1-2build1)
Fast mode used, each commit won't be checked separately.
Okay!

+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel



^ permalink raw reply	[flat|nested] 30+ messages in thread

* [Intel-xe] ✓ CI.BAT: success for drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
                   ` (6 preceding siblings ...)
  (?)
@ 2023-11-10 19:04 ` Patchwork
  -1 siblings, 0 replies; 30+ messages in thread
From: Patchwork @ 2023-11-10 19:04 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: intel-xe

[-- Attachment #1: Type: text/plain, Size: 3386 bytes --]

== Series Details ==

Series: drm/doc/rfc: SR-IOV support on the new Xe driver
URL   : https://patchwork.freedesktop.org/series/126289/
State : success

== Summary ==

CI Bug Log - changes from xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce_BAT -> xe-pw-126289v1_BAT
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  

Participating hosts (3 -> 3)
------------------------------

  No changes in participating hosts

Known issues
------------

  Here are the changes found in xe-pw-126289v1_BAT that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@kms_flip@basic-flip-vs-wf_vblank:
    - bat-dg2-oem2:       [PASS][1] -> [FAIL][2] ([Intel XE#480]) +1 other test fail
   [1]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce/bat-dg2-oem2/igt@kms_flip@basic-flip-vs-wf_vblank.html
   [2]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-126289v1/bat-dg2-oem2/igt@kms_flip@basic-flip-vs-wf_vblank.html

  * igt@kms_flip@basic-flip-vs-wf_vblank@a-edp1:
    - bat-adlp-7:         [PASS][3] -> [FAIL][4] ([Intel XE#480]) +1 other test fail
   [3]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce/bat-adlp-7/igt@kms_flip@basic-flip-vs-wf_vblank@a-edp1.html
   [4]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-126289v1/bat-adlp-7/igt@kms_flip@basic-flip-vs-wf_vblank@a-edp1.html

  
#### Possible fixes ####

  * {igt@xe_create@create-execqueues-leak}:
    - bat-atsm-2:         [FAIL][5] ([Intel XE#524]) -> [PASS][6]
   [5]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce/bat-atsm-2/igt@xe_create@create-execqueues-leak.html
   [6]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-126289v1/bat-atsm-2/igt@xe_create@create-execqueues-leak.html

  
#### Warnings ####

  * igt@kms_frontbuffer_tracking@basic:
    - bat-adlp-7:         [FAIL][7] ([Intel XE#616] / [Intel XE#750]) -> [DMESG-FAIL][8] ([Intel XE#282] / [i915#2017])
   [7]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html
   [8]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-126289v1/bat-adlp-7/igt@kms_frontbuffer_tracking@basic.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [Intel XE#282]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/282
  [Intel XE#480]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/480
  [Intel XE#524]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/524
  [Intel XE#616]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/616
  [Intel XE#750]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/750
  [i915#2017]: https://gitlab.freedesktop.org/drm/intel/issues/2017


Build changes
-------------

  * IGT: IGT_7581 -> IGT_7582
  * Linux: xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce -> xe-pw-126289v1

  IGT_7581: 7581
  IGT_7582: 453b9df12fbc9fff561bdb4eb97992983e74c3d4 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  xe-485-b3890fb4386dcef68a96888141c4cc773f6241ce: b3890fb4386dcef68a96888141c4cc773f6241ce
  xe-pw-126289v1: 126289v1

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-126289v1/index.html

[-- Attachment #2: Type: text/html, Size: 4116 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
@ 2023-11-14 10:08   ` Francois Dugast
  -1 siblings, 0 replies; 30+ messages in thread
From: Francois Dugast @ 2023-11-14 10:08 UTC (permalink / raw)
  To: Michal Wajdeczko; +Cc: Tvrtko Ursulin, intel-xe, dri-devel, Rodrigo Vivi

On Fri, Nov 10, 2023 at 07:22:31PM +0100, Michal Wajdeczko wrote:
> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> Express (PCIe) specification suite is supported starting from 12th
> generation of Intel Graphics processors.
> 
> This RFC aims to explain how do we want to add support for SR-IOV
> to the new Xe driver and to propose related additions to the sysfs.
> 
> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Cc: Oded Gabbay <ogabbay@kernel.org>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> Cc: Daniel Vetter <daniel@ffwll.ch>
> ---
>  Documentation/gpu/rfc/index.rst             |   5 +
>  Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>  Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>  3 files changed, 698 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>  create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> 
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index e4f7b005138d..fc5bc447f30d 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -35,3 +35,8 @@ host such documentation:
>  .. toctree::
>  
>     xe.rst
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   xe_sriov.rst
> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> new file mode 100644
> index 000000000000..77748204dd83
> --- /dev/null
> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> @@ -0,0 +1,501 @@
> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> +..
> +.. Intel Xe driver ABI (SR-IOV extensions)
> +..
> +    The Single Root I/O Virtualization (SR-IOV) extension to
> +    the PCI Express (PCIe) specification suite is supported
> +    starting from 12th generation of Intel Graphics processors.
> +
> +    This document describes Xe driver specific additions.
> +
> +    For description of generic SR-IOV sysfs attributes see
> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_auto_provisioning
> +    │   ├── admin_mode
> +    │   ├── enabled
> +    │   ├── reset_defaults
> +    │   ├── resources
> +    │   │   ├── default_contexts_quota
> +    │   │   ├── default_doorbells_quota
> +    │   │   ├── default_ggtt_quota
> +    │   │   └── default_lmem_quota
> +    │   ├── scheduling
> +    │   │   ├── default_exec_quantum_ms
> +    │   │   └── default_preempt_timeout_us
> +    │   └── monitoring
> +    │       ├── default_cat_error_count
> +    │       ├── default_doorbell_time_us
> +    │       ├── default_engine_reset_count
> +    │       ├── default_h2g_time_us
> +    │       ├── default_irq_time_us
> +    │       └── default_page_fault_count
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_extensions
> +    │   ├── monitoring_period_ms
> +    │   ├── strict_scheduling_enabled
> +    │   ├── pf
> +    │   │   ├── device -> ../../../BDF
> +    │   │   ├── priority
> +    │   │   ├── tile0
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   ├── vf1
> +    │   │   ├── device -> ../../../BDF+1
> +    │   │   ├── stop
> +    │   │   ├── tile0
> +    │   │   │   ├── ggtt_quota
> +    │   │   │   ├── lmem_quota
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── contexts_quota
> +    │   │   │   │   ├── doorbells_quota
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   └── vfN
> +..
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on the device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - xe driver supports SR-IOV PF on given device, and
> +		 - xe driver supports automatic VFs provisioning.
> +
> +		This directory is used as a root for all attributes related to
> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
> +		Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		(un)provisioning that could be performed by the PF.
> +
> +		The default value is 1 (true).
> +
> +		This flag can be set to false, unless manual provisioning is not
> +		applicable for given platform or it is not supported by current
> +		PF implementation. In such cases -EPERM will be returned.
> +
> +		This flag will be automatically set to false when there will be
> +		other attempts to change any of VF's resource provisioning.
> +		See "sriov_extensions" section for details.
> +
> +		This flag can be set back to true if and only if all VFs are
> +		fully unprovisioned, otherwise -EEXIST error will be returned.
> +
> +		false = "disabled"
> +			When disabled, then PF will not attempt to do automatic
> +			VFs provisioning when VFs are being enabled and will not
> +			perform automatic unprovisioning of the VFs when VFs will
> +			be disabled.
> +
> +		true = "enabled"
> +			When enabled, then on VFs enabling PF will do automatic
> +			VFs provisioning based on the default settings described
> +			below.
> +
> +			If automatic VFs provisioning fails due to some reasons,
> +			then VFs will not be enabled.
> +
> +			If enabled, all resources allocated during VFs enabling
> +			will be released during VFs disabling (automatic unprovisioning).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		provisioning that could be performed by the PF.
> +
> +		The default value depends on the platform type.
> +
> +		This flag can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		If enabled (default on discrete platforms) then the PF will
> +		retain only minimum hardcoded resources for its own use when
> +		doing VFs automatic provisioning and will not use any default
> +		values described below for its own configuration.
> +
> +		If disabled (default on integrated platforms) then the PF will
> +		treat itself like yet another additional VF in all fair resource
> +		allocations and will also try to apply default provisioning
> +		values described below for its own configuration.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Writing to this file will reset all default provisioning parameters
> +		listed below to the default values.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent default provisioning that should be used
> +		for VFs automatic provisioning.
> +
> +		These values can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		default_contexts_quota: (RW) integer 0..U32_MAX
> +			The number of GuC context IDs to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> +
> +		default_doorbells_quota: (RW) integer 0..U32_MAX
> +			The number of GuC doorbells to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> +
> +		default_ggtt_quota: (RW) integer 0..U32_MAX
> +			The size of the GGTT address space (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
> +
> +		default_lmem_quota: (RW) integer 0..U32_MAX
> +			The size of the LMEM (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
> +
> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum (in millisecs) assigned to the function.
> +			The default value is 0 (infinify).
> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> +
> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout (in microsecs) assigned to the function.
> +			The default value is 0 (infinity).
> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
> +
> +		default_cat_error_count: (RW) integer 0..U32_MAX
> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
> +		default_engine_reset_count: (RW) integer 0..U32_MAX
> +		default_h2g_time_us: (RW) integer 0..U32_MAX
> +		default_irq_time_us: (RW) integer 0..U32_MAX
> +		default_page_fault_count: (RW) integer 0..U32_MAX
> +			The monitoring threshold to be set for the function.
> +			The default value is 0 (don't monitor).
> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on Xe device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - driver is enabled to support SR-IOV PF on given device.
> +
> +		This directory is used as a root for all attributes required to
> +		manage both Physical Function (PF) and Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool
> +
> +		This file represents a flag used to determine if scheduling
> +		parameters should be respected even if there is no active
> +		workloads submitted by the PF or VFs.
> +
> +		This flag is disabled by default, unless strict scheduling is
> +		not applicable on given platform. In such case this file will
> +		be read-only.
> +
> +		The change to this file may have no effect if VFs are not yet enabled.
> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) integer
> +
> +		This file represents the configuration knob used by adverse event
> +		monitoring. A value here is the period in millisecs during which
> +		events are counted and the total is checked against a threshold.
> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
> +
> +		Default is 0 (monitoring is disabled).
> +
> +		If monitoring capability is not available, then attempt to enable
> +		will fail with -EPERM error. If monitoring can't be enabled in
> +		GuC then write will fail with -EIO.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Physical Function (PF).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Virtual Function (VF).
> +
> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
> +		The Xe driver implementaton follows that naming schema.
> +
> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
> +		value of the PCI "sriov_totalvfs" attribute.
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device tile.
> +		The tile numbers (T) start from 0.
> +
> +		There is at least one "tile0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device GT.
> +		The GT numbers (X) start from 0.
> +
> +		There is at least one "gt0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(symbolic link)
> +
> +		Backlink to the PCI device entry representing given function.
> +		For PF this link is always present.
> +		For VF this link is present only for currently enabled VFs.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) string
> +
> +		This file represents a GuC Scheduler knob to override the default
> +		round-robin or FIFO scheduler policies implemented by the GuC.
> +
> +		The default value is "peer".
> +
> +		This flag can be changed, unless such change is not applicable
> +		for given platform or is not supported by current GuC firmware.
> +		In such case this file could be read-only or will return -EPERM
> +		on write attempt.
> +
> +		"immediate"
> +			GuC will Schedule PF workloads immediately and PF
> +			workloads only until the PF's work queues in GuC
> +			are empty.
> +
> +		"lazy"
> +			GuC will Schedule PF workloads at the next opportune
> +			moment and PF workloads only until the PF work queues
> +			in GuC are empty.
> +
> +		"peer"
> +			GuC Scheduler will treat PF and VFs with equal priority.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Write to this file will force GuC to stop handle any requests from
> +		this VF, but without triggering a FLR.
> +		To recover, the full FLR must be issued using generic "device/reset".
> +
> +		This file allows to implement custom policy mechanism when VF is
> +		misbehaving and triggering adverse events above defined thresholds.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent scheduling parameters of the functions.
> +
> +		These scheduling parameters can be changed even if VFs are enabled
> +		and running, unless such change is not applicable on given platform
> +		due to fixed hardware or firmware assignment.
> +
> +		exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum in [ms] assigned to the function.
> +			Requested quantum might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +		preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout in [us] assigned to the function.
> +			Requested timeout might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent shared resource assigned to the functions.
> +
> +		These resource parameters can be changed, unless VF is already running,
> +		or such change is not applicable on given platform due to fixed hardware
> +		or firmware assignment.
> +
> +		Writes to these attributes may fail with:
> +			-EPERM if change is not applicable on give HW/FW.
> +			-E2BIG if value larger that HW/FW limit.
> +			-EDQUOT if value is larger than maximum quota defined by the PF.
> +			-ENOSPC if PF can't allocate required quota.
> +			-EBUSY if the resource is currently in use by the VF.
> +			-EIO if GuC refuses to change provisioning.
> +
> +		ggtt_quota: (RW) integer 0..U64_MAX
> +			The size of the GGTT address space (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		lmem_quota: (RW) integer 0..U64_MAX
> +			The size of the Local Memory (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			This attribute is only available on discrete platforms.
> +
> +			Default is 0 (unprovisioned).
> +
> +		contexts_quota: (RW) 0..U16_MAX
> +			The number of GuC submission contexts assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		doorbells_quota: (RW) 0..U16_MAX
> +			The number of GuC doorbells assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent threshold values used by the GuC to trigger
> +		security events if adverse event monitoring is enabled.
> +
> +		These thresholds are checked every "monitoring_period_ms".
> +		Refer to GuC ABI for details about each threshold category.
> +
> +		Default value for all thresholds is 0 (disabled).
> +
> +		cat_error_count: (RW) integer
> +		doorbell_time_us: (RW) integer
> +		engine_reset_count: (RW) integer
> +		h2g_time_us: (RW) integer
> +		irq_time_us: (RW) integer
> +		page_fault_count: (RW) integer
> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
> new file mode 100644
> index 000000000000..574f6414eabb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/xe_sriov.rst
> @@ -0,0 +1,192 @@
> +.. SPDX-License-Identifier: MIT
> +
> +========================
> +Xe – SR-IOV Support Plan
> +========================
> +
> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
> +specification suite is supported starting from 12th generation of Intel Graphics
> +processors.
> +
> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
> +provide flexible configuration and management options related to the SR-IOV.
> +It will also highlight few most important changes to the Xe driver
> +implementation to deal with Intel GPU SR-IOV specific requirements.
> +
> +
> +SR-IOV Capability
> +=================
> +
> +Due to SR-IOV complexity and required co-operation between hardware, firmware
> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
> +or fully functional.
> +
> +To control at the driver level which platform will provide support for SR-IOV,
> +as we can't just rely on the PCI configuration data exposed by the hardware,
> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
> +a device capabilities that driver checks during the probe.
> +
> +Initially this flag will be set to disabled even on platforms that we plan to
> +support. We will enable this flag only once we finish merging all required
> +changes to the driver and related validated firmwares are also made available.
> +
> +
> +SR-IOV Platforms
> +================
> +
> +Initially we plan to add SR-IOV functionality to the following SDV platforms
> +already supported by the Xe driver:
> +
> + - TGL (up to 7 VFs)
> + - ADL (up to 7 VFs)
> + - MTL (up to 7 VFs)
> + - ATSM (up to 31 VFs)
> + - PVC (up to 63 VFs)
> +
> +Newer platforms will be supported later, but we hope that enabling will be
> +much faster, as majority of the driver changes are either platform agnostic
> +or are similar between earlier platforms (hence we start with SDVs).
> +
> +
> +PF Mode
> +=======
> +
> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
> +enabled by default.
> +
> +However, due to potentially conflicting requirements for SR-IOV and other mega
> +features, we might want to have an option to disable SR-IOV PF mode support at
> +the driver load time.

What about making SR-IOV support in Xe dependent on a separate build option, such
as CONFIG_DRM_XE_SRIOV? This would allow users to enable SR-IOV with CONFIG_PCI_IOV
to virtualize other devices, let's say a network adapter, but to keep this feature
compiled out of Xe.

Francois

> +
> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
> +0 will force the driver to operate in the native (non-virtualized) mode.
> +The same modparam could be used to limit number of supported Virtual Functions
> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
> +
> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
> +by default exposes hardware capability.
> +
> +The default value of this param will allow to support all possible VFs as
> +claimed by the hardware.
> +
> +This modparam will have no effect if driver is running on the VF device.
> +
> +
> +VFs Enabling
> +============
> +
> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
> +provision VFs in our custom implementation of the "sriov_configure" hook from
> +the struct pci_driver.
> +
> +If for some reason, including explicit request to disable SR-IOV PF mode using
> +modparam, we will not be able to correctly support any VFs, driver will change
> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
> +to 0, thus preventing configuration of the VFs.
> +
> +
> +VF Mode
> +=======
> +
> +When driver is running on the VF device, then due to hardware enforcements,
> +access to the privileged registers is not possible. To avoid relying on these
> +registers, we plan to perform early detection if we are running on the VF
> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
> +IS_SRIOV_VF(xe) to control the driver logic.
> +
> +To speed up merging of the required changes, we might first introduce dummy
> +macro that is always set to false, to prepare driver to avoid some code paths
> +before we finalize our VF mode detection and other VFs enabling changes.
> +
> +
> +Resources
> +=========
> +
> +Most of the hardware (or firmware) resources available on the Xe architecture,
> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
> +VFs and will require some provisioning steps to assign those resources for use
> +by the VF.
> +
> +Until VFs are provisioned with resources, the PF driver will be able to use all
> +resources, in the same way as it would be running in non-virtualized mode.
> +
> +If some resource (of part or region of it) is assigned to specific VF, then PF
> +is not allowed to use that part or region of the resource, but can continue to
> +use whatever is left available.
> +
> +Those resources are usually fully virtualized, so they will not require any
> +special handling when used by the VF driver, except that VF driver must know
> +the assigned quota.
> +
> +The most notable exception is the GGTT address space, as on some platforms,
> +the VF driver must additionally know the real range that it can access.
> +
> +Once the resources were assigned to the VF use and the VF driver has started,
> +then it is not allowed to change such provisioning, as that would break the
> +VF driver. To make changes the VF driver, which was using these resources,
> +must be unloaded (or the VM is terminated) and the VF device must be reset
> +using the FLR.
> +
> +
> +Scheduling
> +==========
> +
> +The workloads from PF driver and VF drivers must be submitted to the hardware
> +always by using the GuC submission mechanism. Unless VF has exclusive access
> +to the GT then submissions from different VFs are time-sliced and controlled
> +with additional "execution_quantum" and "preemption_timeout" parameters.
> +
> +In contrast to the resource provisioning, those scheduling parameters can be
> +changed even if VF drivers are already running and are active.
> +
> +
> +Automatic VFs Provisioning
> +==========================
> +
> +To provide out-of-the box experience when user will be enabling VFs using
> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
> +
> +By default, all VFs will be allocated with the fair amount of the mandatory
> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
> +Such provisioning should be sufficient for most of the normal usages, when
> +no strict SLA is required.
> +
> +The PF driver will also expose some additional sysfs files to allow adjusting
> +this automatic VFs provisioning, like default values for most of the
> +provisioning parameters that PF will then apply for each enabled VF.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +Manual VFs Provisioning
> +=======================
> +
> +If automatic VFs provisioning, which applies same configuration to every VF,
> +is not sufficient or there is a need for advanced customization of some VF,
> +the PF driver will also provide extended sysfs interface which will allow
> +control every provisioning attribute to the lowest feasible level.
> +
> +It is expected that these low-level attributes will be mostly used by the
> +advanced users or by the custom tools that will setup configurations that
> +meet predefined and validated SLA as required by the customers.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +VFs Monitoring
> +==============
> +
> +In addition to the resource provisioning or changing scheduling parameters,
> +the PF driver might also allow configure some monitoring parameters, like
> +thresholds of adverse events or sample period, to track undesired behavior
> +of the VFs that could impact the whole system.
> +
> +Once those thresholds are setup and sampling period is defined, the GuC will
> +notify the PF driver about which VF is excessing the threshold and then PF is
> +able to trigger the uevent to notify the administrator (or VMM) that could
> +take some action against the VF.
> -- 
> 2.25.1
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 10:08   ` Francois Dugast
  0 siblings, 0 replies; 30+ messages in thread
From: Francois Dugast @ 2023-11-14 10:08 UTC (permalink / raw)
  To: Michal Wajdeczko
  Cc: Daniel Vetter, Tvrtko Ursulin, intel-xe, dri-devel, Rodrigo Vivi

On Fri, Nov 10, 2023 at 07:22:31PM +0100, Michal Wajdeczko wrote:
> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> Express (PCIe) specification suite is supported starting from 12th
> generation of Intel Graphics processors.
> 
> This RFC aims to explain how do we want to add support for SR-IOV
> to the new Xe driver and to propose related additions to the sysfs.
> 
> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Cc: Oded Gabbay <ogabbay@kernel.org>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> Cc: Daniel Vetter <daniel@ffwll.ch>
> ---
>  Documentation/gpu/rfc/index.rst             |   5 +
>  Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>  Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>  3 files changed, 698 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>  create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> 
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index e4f7b005138d..fc5bc447f30d 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -35,3 +35,8 @@ host such documentation:
>  .. toctree::
>  
>     xe.rst
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   xe_sriov.rst
> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> new file mode 100644
> index 000000000000..77748204dd83
> --- /dev/null
> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> @@ -0,0 +1,501 @@
> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> +..
> +.. Intel Xe driver ABI (SR-IOV extensions)
> +..
> +    The Single Root I/O Virtualization (SR-IOV) extension to
> +    the PCI Express (PCIe) specification suite is supported
> +    starting from 12th generation of Intel Graphics processors.
> +
> +    This document describes Xe driver specific additions.
> +
> +    For description of generic SR-IOV sysfs attributes see
> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_auto_provisioning
> +    │   ├── admin_mode
> +    │   ├── enabled
> +    │   ├── reset_defaults
> +    │   ├── resources
> +    │   │   ├── default_contexts_quota
> +    │   │   ├── default_doorbells_quota
> +    │   │   ├── default_ggtt_quota
> +    │   │   └── default_lmem_quota
> +    │   ├── scheduling
> +    │   │   ├── default_exec_quantum_ms
> +    │   │   └── default_preempt_timeout_us
> +    │   └── monitoring
> +    │       ├── default_cat_error_count
> +    │       ├── default_doorbell_time_us
> +    │       ├── default_engine_reset_count
> +    │       ├── default_h2g_time_us
> +    │       ├── default_irq_time_us
> +    │       └── default_page_fault_count
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_extensions
> +    │   ├── monitoring_period_ms
> +    │   ├── strict_scheduling_enabled
> +    │   ├── pf
> +    │   │   ├── device -> ../../../BDF
> +    │   │   ├── priority
> +    │   │   ├── tile0
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   ├── vf1
> +    │   │   ├── device -> ../../../BDF+1
> +    │   │   ├── stop
> +    │   │   ├── tile0
> +    │   │   │   ├── ggtt_quota
> +    │   │   │   ├── lmem_quota
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── contexts_quota
> +    │   │   │   │   ├── doorbells_quota
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   └── vfN
> +..
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on the device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - xe driver supports SR-IOV PF on given device, and
> +		 - xe driver supports automatic VFs provisioning.
> +
> +		This directory is used as a root for all attributes related to
> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
> +		Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		(un)provisioning that could be performed by the PF.
> +
> +		The default value is 1 (true).
> +
> +		This flag can be set to false, unless manual provisioning is not
> +		applicable for given platform or it is not supported by current
> +		PF implementation. In such cases -EPERM will be returned.
> +
> +		This flag will be automatically set to false when there will be
> +		other attempts to change any of VF's resource provisioning.
> +		See "sriov_extensions" section for details.
> +
> +		This flag can be set back to true if and only if all VFs are
> +		fully unprovisioned, otherwise -EEXIST error will be returned.
> +
> +		false = "disabled"
> +			When disabled, then PF will not attempt to do automatic
> +			VFs provisioning when VFs are being enabled and will not
> +			perform automatic unprovisioning of the VFs when VFs will
> +			be disabled.
> +
> +		true = "enabled"
> +			When enabled, then on VFs enabling PF will do automatic
> +			VFs provisioning based on the default settings described
> +			below.
> +
> +			If automatic VFs provisioning fails due to some reasons,
> +			then VFs will not be enabled.
> +
> +			If enabled, all resources allocated during VFs enabling
> +			will be released during VFs disabling (automatic unprovisioning).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		provisioning that could be performed by the PF.
> +
> +		The default value depends on the platform type.
> +
> +		This flag can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		If enabled (default on discrete platforms) then the PF will
> +		retain only minimum hardcoded resources for its own use when
> +		doing VFs automatic provisioning and will not use any default
> +		values described below for its own configuration.
> +
> +		If disabled (default on integrated platforms) then the PF will
> +		treat itself like yet another additional VF in all fair resource
> +		allocations and will also try to apply default provisioning
> +		values described below for its own configuration.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Writing to this file will reset all default provisioning parameters
> +		listed below to the default values.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent default provisioning that should be used
> +		for VFs automatic provisioning.
> +
> +		These values can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		default_contexts_quota: (RW) integer 0..U32_MAX
> +			The number of GuC context IDs to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> +
> +		default_doorbells_quota: (RW) integer 0..U32_MAX
> +			The number of GuC doorbells to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> +
> +		default_ggtt_quota: (RW) integer 0..U32_MAX
> +			The size of the GGTT address space (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
> +
> +		default_lmem_quota: (RW) integer 0..U32_MAX
> +			The size of the LMEM (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
> +
> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum (in millisecs) assigned to the function.
> +			The default value is 0 (infinify).
> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> +
> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout (in microsecs) assigned to the function.
> +			The default value is 0 (infinity).
> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
> +
> +		default_cat_error_count: (RW) integer 0..U32_MAX
> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
> +		default_engine_reset_count: (RW) integer 0..U32_MAX
> +		default_h2g_time_us: (RW) integer 0..U32_MAX
> +		default_irq_time_us: (RW) integer 0..U32_MAX
> +		default_page_fault_count: (RW) integer 0..U32_MAX
> +			The monitoring threshold to be set for the function.
> +			The default value is 0 (don't monitor).
> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on Xe device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - driver is enabled to support SR-IOV PF on given device.
> +
> +		This directory is used as a root for all attributes required to
> +		manage both Physical Function (PF) and Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool
> +
> +		This file represents a flag used to determine if scheduling
> +		parameters should be respected even if there is no active
> +		workloads submitted by the PF or VFs.
> +
> +		This flag is disabled by default, unless strict scheduling is
> +		not applicable on given platform. In such case this file will
> +		be read-only.
> +
> +		The change to this file may have no effect if VFs are not yet enabled.
> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) integer
> +
> +		This file represents the configuration knob used by adverse event
> +		monitoring. A value here is the period in millisecs during which
> +		events are counted and the total is checked against a threshold.
> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
> +
> +		Default is 0 (monitoring is disabled).
> +
> +		If monitoring capability is not available, then attempt to enable
> +		will fail with -EPERM error. If monitoring can't be enabled in
> +		GuC then write will fail with -EIO.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Physical Function (PF).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Virtual Function (VF).
> +
> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
> +		The Xe driver implementaton follows that naming schema.
> +
> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
> +		value of the PCI "sriov_totalvfs" attribute.
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device tile.
> +		The tile numbers (T) start from 0.
> +
> +		There is at least one "tile0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device GT.
> +		The GT numbers (X) start from 0.
> +
> +		There is at least one "gt0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(symbolic link)
> +
> +		Backlink to the PCI device entry representing given function.
> +		For PF this link is always present.
> +		For VF this link is present only for currently enabled VFs.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) string
> +
> +		This file represents a GuC Scheduler knob to override the default
> +		round-robin or FIFO scheduler policies implemented by the GuC.
> +
> +		The default value is "peer".
> +
> +		This flag can be changed, unless such change is not applicable
> +		for given platform or is not supported by current GuC firmware.
> +		In such case this file could be read-only or will return -EPERM
> +		on write attempt.
> +
> +		"immediate"
> +			GuC will Schedule PF workloads immediately and PF
> +			workloads only until the PF's work queues in GuC
> +			are empty.
> +
> +		"lazy"
> +			GuC will Schedule PF workloads at the next opportune
> +			moment and PF workloads only until the PF work queues
> +			in GuC are empty.
> +
> +		"peer"
> +			GuC Scheduler will treat PF and VFs with equal priority.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Write to this file will force GuC to stop handle any requests from
> +		this VF, but without triggering a FLR.
> +		To recover, the full FLR must be issued using generic "device/reset".
> +
> +		This file allows to implement custom policy mechanism when VF is
> +		misbehaving and triggering adverse events above defined thresholds.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent scheduling parameters of the functions.
> +
> +		These scheduling parameters can be changed even if VFs are enabled
> +		and running, unless such change is not applicable on given platform
> +		due to fixed hardware or firmware assignment.
> +
> +		exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum in [ms] assigned to the function.
> +			Requested quantum might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +		preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout in [us] assigned to the function.
> +			Requested timeout might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent shared resource assigned to the functions.
> +
> +		These resource parameters can be changed, unless VF is already running,
> +		or such change is not applicable on given platform due to fixed hardware
> +		or firmware assignment.
> +
> +		Writes to these attributes may fail with:
> +			-EPERM if change is not applicable on give HW/FW.
> +			-E2BIG if value larger that HW/FW limit.
> +			-EDQUOT if value is larger than maximum quota defined by the PF.
> +			-ENOSPC if PF can't allocate required quota.
> +			-EBUSY if the resource is currently in use by the VF.
> +			-EIO if GuC refuses to change provisioning.
> +
> +		ggtt_quota: (RW) integer 0..U64_MAX
> +			The size of the GGTT address space (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		lmem_quota: (RW) integer 0..U64_MAX
> +			The size of the Local Memory (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			This attribute is only available on discrete platforms.
> +
> +			Default is 0 (unprovisioned).
> +
> +		contexts_quota: (RW) 0..U16_MAX
> +			The number of GuC submission contexts assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		doorbells_quota: (RW) 0..U16_MAX
> +			The number of GuC doorbells assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent threshold values used by the GuC to trigger
> +		security events if adverse event monitoring is enabled.
> +
> +		These thresholds are checked every "monitoring_period_ms".
> +		Refer to GuC ABI for details about each threshold category.
> +
> +		Default value for all thresholds is 0 (disabled).
> +
> +		cat_error_count: (RW) integer
> +		doorbell_time_us: (RW) integer
> +		engine_reset_count: (RW) integer
> +		h2g_time_us: (RW) integer
> +		irq_time_us: (RW) integer
> +		page_fault_count: (RW) integer
> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
> new file mode 100644
> index 000000000000..574f6414eabb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/xe_sriov.rst
> @@ -0,0 +1,192 @@
> +.. SPDX-License-Identifier: MIT
> +
> +========================
> +Xe – SR-IOV Support Plan
> +========================
> +
> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
> +specification suite is supported starting from 12th generation of Intel Graphics
> +processors.
> +
> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
> +provide flexible configuration and management options related to the SR-IOV.
> +It will also highlight few most important changes to the Xe driver
> +implementation to deal with Intel GPU SR-IOV specific requirements.
> +
> +
> +SR-IOV Capability
> +=================
> +
> +Due to SR-IOV complexity and required co-operation between hardware, firmware
> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
> +or fully functional.
> +
> +To control at the driver level which platform will provide support for SR-IOV,
> +as we can't just rely on the PCI configuration data exposed by the hardware,
> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
> +a device capabilities that driver checks during the probe.
> +
> +Initially this flag will be set to disabled even on platforms that we plan to
> +support. We will enable this flag only once we finish merging all required
> +changes to the driver and related validated firmwares are also made available.
> +
> +
> +SR-IOV Platforms
> +================
> +
> +Initially we plan to add SR-IOV functionality to the following SDV platforms
> +already supported by the Xe driver:
> +
> + - TGL (up to 7 VFs)
> + - ADL (up to 7 VFs)
> + - MTL (up to 7 VFs)
> + - ATSM (up to 31 VFs)
> + - PVC (up to 63 VFs)
> +
> +Newer platforms will be supported later, but we hope that enabling will be
> +much faster, as majority of the driver changes are either platform agnostic
> +or are similar between earlier platforms (hence we start with SDVs).
> +
> +
> +PF Mode
> +=======
> +
> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
> +enabled by default.
> +
> +However, due to potentially conflicting requirements for SR-IOV and other mega
> +features, we might want to have an option to disable SR-IOV PF mode support at
> +the driver load time.

What about making SR-IOV support in Xe dependent on a separate build option, such
as CONFIG_DRM_XE_SRIOV? This would allow users to enable SR-IOV with CONFIG_PCI_IOV
to virtualize other devices, let's say a network adapter, but to keep this feature
compiled out of Xe.

Francois

> +
> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
> +0 will force the driver to operate in the native (non-virtualized) mode.
> +The same modparam could be used to limit number of supported Virtual Functions
> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
> +
> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
> +by default exposes hardware capability.
> +
> +The default value of this param will allow to support all possible VFs as
> +claimed by the hardware.
> +
> +This modparam will have no effect if driver is running on the VF device.
> +
> +
> +VFs Enabling
> +============
> +
> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
> +provision VFs in our custom implementation of the "sriov_configure" hook from
> +the struct pci_driver.
> +
> +If for some reason, including explicit request to disable SR-IOV PF mode using
> +modparam, we will not be able to correctly support any VFs, driver will change
> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
> +to 0, thus preventing configuration of the VFs.
> +
> +
> +VF Mode
> +=======
> +
> +When driver is running on the VF device, then due to hardware enforcements,
> +access to the privileged registers is not possible. To avoid relying on these
> +registers, we plan to perform early detection if we are running on the VF
> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
> +IS_SRIOV_VF(xe) to control the driver logic.
> +
> +To speed up merging of the required changes, we might first introduce dummy
> +macro that is always set to false, to prepare driver to avoid some code paths
> +before we finalize our VF mode detection and other VFs enabling changes.
> +
> +
> +Resources
> +=========
> +
> +Most of the hardware (or firmware) resources available on the Xe architecture,
> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
> +VFs and will require some provisioning steps to assign those resources for use
> +by the VF.
> +
> +Until VFs are provisioned with resources, the PF driver will be able to use all
> +resources, in the same way as it would be running in non-virtualized mode.
> +
> +If some resource (of part or region of it) is assigned to specific VF, then PF
> +is not allowed to use that part or region of the resource, but can continue to
> +use whatever is left available.
> +
> +Those resources are usually fully virtualized, so they will not require any
> +special handling when used by the VF driver, except that VF driver must know
> +the assigned quota.
> +
> +The most notable exception is the GGTT address space, as on some platforms,
> +the VF driver must additionally know the real range that it can access.
> +
> +Once the resources were assigned to the VF use and the VF driver has started,
> +then it is not allowed to change such provisioning, as that would break the
> +VF driver. To make changes the VF driver, which was using these resources,
> +must be unloaded (or the VM is terminated) and the VF device must be reset
> +using the FLR.
> +
> +
> +Scheduling
> +==========
> +
> +The workloads from PF driver and VF drivers must be submitted to the hardware
> +always by using the GuC submission mechanism. Unless VF has exclusive access
> +to the GT then submissions from different VFs are time-sliced and controlled
> +with additional "execution_quantum" and "preemption_timeout" parameters.
> +
> +In contrast to the resource provisioning, those scheduling parameters can be
> +changed even if VF drivers are already running and are active.
> +
> +
> +Automatic VFs Provisioning
> +==========================
> +
> +To provide out-of-the box experience when user will be enabling VFs using
> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
> +
> +By default, all VFs will be allocated with the fair amount of the mandatory
> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
> +Such provisioning should be sufficient for most of the normal usages, when
> +no strict SLA is required.
> +
> +The PF driver will also expose some additional sysfs files to allow adjusting
> +this automatic VFs provisioning, like default values for most of the
> +provisioning parameters that PF will then apply for each enabled VF.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +Manual VFs Provisioning
> +=======================
> +
> +If automatic VFs provisioning, which applies same configuration to every VF,
> +is not sufficient or there is a need for advanced customization of some VF,
> +the PF driver will also provide extended sysfs interface which will allow
> +control every provisioning attribute to the lowest feasible level.
> +
> +It is expected that these low-level attributes will be mostly used by the
> +advanced users or by the custom tools that will setup configurations that
> +meet predefined and validated SLA as required by the customers.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +VFs Monitoring
> +==============
> +
> +In addition to the resource provisioning or changing scheduling parameters,
> +the PF driver might also allow configure some monitoring parameters, like
> +thresholds of adverse events or sample period, to track undesired behavior
> +of the VFs that could impact the whole system.
> +
> +Once those thresholds are setup and sampling period is defined, the GuC will
> +notify the PF driver about which VF is excessing the threshold and then PF is
> +able to trigger the uevent to notify the administrator (or VMM) that could
> +take some action against the VF.
> -- 
> 2.25.1
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-14 10:08   ` Francois Dugast
@ 2023-11-14 10:29     ` Michal Wajdeczko
  -1 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 10:29 UTC (permalink / raw)
  To: Francois Dugast; +Cc: Tvrtko Ursulin, intel-xe, dri-devel, Rodrigo Vivi



On 14.11.2023 11:08, Francois Dugast wrote:
> On Fri, Nov 10, 2023 at 07:22:31PM +0100, Michal Wajdeczko wrote:
>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe) specification suite is supported starting from 12th
>> generation of Intel Graphics processors.
>>
>> This RFC aims to explain how do we want to add support for SR-IOV
>> to the new Xe driver and to propose related additions to the sysfs.
>>
>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>> Cc: Oded Gabbay <ogabbay@kernel.org>
>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>> Cc: Daniel Vetter <daniel@ffwll.ch>
>> ---
>>  Documentation/gpu/rfc/index.rst             |   5 +
>>  Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>>  Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>  3 files changed, 698 insertions(+)
>>  create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>  create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>
>> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
>> index e4f7b005138d..fc5bc447f30d 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -35,3 +35,8 @@ host such documentation:
>>  .. toctree::
>>  
>>     xe.rst
>> +
>> +.. toctree::
>> +   :maxdepth: 1
>> +
>> +   xe_sriov.rst
>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> new file mode 100644
>> index 000000000000..77748204dd83
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> @@ -0,0 +1,501 @@
>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>> +..
>> +.. Intel Xe driver ABI (SR-IOV extensions)
>> +..
>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>> +    the PCI Express (PCIe) specification suite is supported
>> +    starting from 12th generation of Intel Graphics processors.
>> +
>> +    This document describes Xe driver specific additions.
>> +
>> +    For description of generic SR-IOV sysfs attributes see
>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_auto_provisioning
>> +    │   ├── admin_mode
>> +    │   ├── enabled
>> +    │   ├── reset_defaults
>> +    │   ├── resources
>> +    │   │   ├── default_contexts_quota
>> +    │   │   ├── default_doorbells_quota
>> +    │   │   ├── default_ggtt_quota
>> +    │   │   └── default_lmem_quota
>> +    │   ├── scheduling
>> +    │   │   ├── default_exec_quantum_ms
>> +    │   │   └── default_preempt_timeout_us
>> +    │   └── monitoring
>> +    │       ├── default_cat_error_count
>> +    │       ├── default_doorbell_time_us
>> +    │       ├── default_engine_reset_count
>> +    │       ├── default_h2g_time_us
>> +    │       ├── default_irq_time_us
>> +    │       └── default_page_fault_count
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_extensions
>> +    │   ├── monitoring_period_ms
>> +    │   ├── strict_scheduling_enabled
>> +    │   ├── pf
>> +    │   │   ├── device -> ../../../BDF
>> +    │   │   ├── priority
>> +    │   │   ├── tile0
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   ├── vf1
>> +    │   │   ├── device -> ../../../BDF+1
>> +    │   │   ├── stop
>> +    │   │   ├── tile0
>> +    │   │   │   ├── ggtt_quota
>> +    │   │   │   ├── lmem_quota
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── contexts_quota
>> +    │   │   │   │   ├── doorbells_quota
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   └── vfN
>> +..
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory appears on the device when:
>> +
>> +		 - device supports SR-IOV, and
>> +		 - device is a Physical Function (PF), and
>> +		 - xe driver supports SR-IOV PF on given device, and
>> +		 - xe driver supports automatic VFs provisioning.
>> +
>> +		This directory is used as a root for all attributes related to
>> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
>> +		Virtual Functions (VFs).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool (0, 1)
>> +
>> +		This file represents configuration flag for the automatic VFs
>> +		(un)provisioning that could be performed by the PF.
>> +
>> +		The default value is 1 (true).
>> +
>> +		This flag can be set to false, unless manual provisioning is not
>> +		applicable for given platform or it is not supported by current
>> +		PF implementation. In such cases -EPERM will be returned.
>> +
>> +		This flag will be automatically set to false when there will be
>> +		other attempts to change any of VF's resource provisioning.
>> +		See "sriov_extensions" section for details.
>> +
>> +		This flag can be set back to true if and only if all VFs are
>> +		fully unprovisioned, otherwise -EEXIST error will be returned.
>> +
>> +		false = "disabled"
>> +			When disabled, then PF will not attempt to do automatic
>> +			VFs provisioning when VFs are being enabled and will not
>> +			perform automatic unprovisioning of the VFs when VFs will
>> +			be disabled.
>> +
>> +		true = "enabled"
>> +			When enabled, then on VFs enabling PF will do automatic
>> +			VFs provisioning based on the default settings described
>> +			below.
>> +
>> +			If automatic VFs provisioning fails due to some reasons,
>> +			then VFs will not be enabled.
>> +
>> +			If enabled, all resources allocated during VFs enabling
>> +			will be released during VFs disabling (automatic unprovisioning).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool (0, 1)
>> +
>> +		This file represents configuration flag for the automatic VFs
>> +		provisioning that could be performed by the PF.
>> +
>> +		The default value depends on the platform type.
>> +
>> +		This flag can be changed any time, but will have no effect if
>> +		VFs are already provisioned.
>> +
>> +		If enabled (default on discrete platforms) then the PF will
>> +		retain only minimum hardcoded resources for its own use when
>> +		doing VFs automatic provisioning and will not use any default
>> +		values described below for its own configuration.
>> +
>> +		If disabled (default on integrated platforms) then the PF will
>> +		treat itself like yet another additional VF in all fair resource
>> +		allocations and will also try to apply default provisioning
>> +		values described below for its own configuration.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(WO) bool (1)
>> +
>> +		Writing to this file will reset all default provisioning parameters
>> +		listed below to the default values.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent default provisioning that should be used
>> +		for VFs automatic provisioning.
>> +
>> +		These values can be changed any time, but will have no effect if
>> +		VFs are already provisioned.
>> +
>> +		default_contexts_quota: (RW) integer 0..U32_MAX
>> +			The number of GuC context IDs to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
>> +
>> +		default_doorbells_quota: (RW) integer 0..U32_MAX
>> +			The number of GuC doorbells to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
>> +
>> +		default_ggtt_quota: (RW) integer 0..U32_MAX
>> +			The size of the GGTT address space (in bytes) to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
>> +
>> +		default_lmem_quota: (RW) integer 0..U32_MAX
>> +			The size of the LMEM (in bytes) to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
>> +
>> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
>> +			The GT execution quantum (in millisecs) assigned to the function.
>> +			The default value is 0 (infinify).
>> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
>> +
>> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
>> +			The GT preemption timeout (in microsecs) assigned to the function.
>> +			The default value is 0 (infinity).
>> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
>> +
>> +		default_cat_error_count: (RW) integer 0..U32_MAX
>> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
>> +		default_engine_reset_count: (RW) integer 0..U32_MAX
>> +		default_h2g_time_us: (RW) integer 0..U32_MAX
>> +		default_irq_time_us: (RW) integer 0..U32_MAX
>> +		default_page_fault_count: (RW) integer 0..U32_MAX
>> +			The monitoring threshold to be set for the function.
>> +			The default value is 0 (don't monitor).
>> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory appears on Xe device when:
>> +
>> +		 - device supports SR-IOV, and
>> +		 - device is a Physical Function (PF), and
>> +		 - driver is enabled to support SR-IOV PF on given device.
>> +
>> +		This directory is used as a root for all attributes required to
>> +		manage both Physical Function (PF) and Virtual Functions (VFs).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool
>> +
>> +		This file represents a flag used to determine if scheduling
>> +		parameters should be respected even if there is no active
>> +		workloads submitted by the PF or VFs.
>> +
>> +		This flag is disabled by default, unless strict scheduling is
>> +		not applicable on given platform. In such case this file will
>> +		be read-only.
>> +
>> +		The change to this file may have no effect if VFs are not yet enabled.
>> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) integer
>> +
>> +		This file represents the configuration knob used by adverse event
>> +		monitoring. A value here is the period in millisecs during which
>> +		events are counted and the total is checked against a threshold.
>> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
>> +
>> +		Default is 0 (monitoring is disabled).
>> +
>> +		If monitoring capability is not available, then attempt to enable
>> +		will fail with -EPERM error. If monitoring can't be enabled in
>> +		GuC then write will fail with -EIO.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all attributes related to the SR-IOV
>> +		Physical Function (PF).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all attributes related to the SR-IOV
>> +		Virtual Function (VF).
>> +
>> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
>> +		The Xe driver implementaton follows that naming schema.
>> +
>> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
>> +		value of the PCI "sriov_totalvfs" attribute.
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all SR-IOV attributes related to the device tile.
>> +		The tile numbers (T) start from 0.
>> +
>> +		There is at least one "tile0/" directory present.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all SR-IOV attributes related to the device GT.
>> +		The GT numbers (X) start from 0.
>> +
>> +		There is at least one "gt0/" directory present.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(symbolic link)
>> +
>> +		Backlink to the PCI device entry representing given function.
>> +		For PF this link is always present.
>> +		For VF this link is present only for currently enabled VFs.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) string
>> +
>> +		This file represents a GuC Scheduler knob to override the default
>> +		round-robin or FIFO scheduler policies implemented by the GuC.
>> +
>> +		The default value is "peer".
>> +
>> +		This flag can be changed, unless such change is not applicable
>> +		for given platform or is not supported by current GuC firmware.
>> +		In such case this file could be read-only or will return -EPERM
>> +		on write attempt.
>> +
>> +		"immediate"
>> +			GuC will Schedule PF workloads immediately and PF
>> +			workloads only until the PF's work queues in GuC
>> +			are empty.
>> +
>> +		"lazy"
>> +			GuC will Schedule PF workloads at the next opportune
>> +			moment and PF workloads only until the PF work queues
>> +			in GuC are empty.
>> +
>> +		"peer"
>> +			GuC Scheduler will treat PF and VFs with equal priority.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(WO) bool (1)
>> +
>> +		Write to this file will force GuC to stop handle any requests from
>> +		this VF, but without triggering a FLR.
>> +		To recover, the full FLR must be issued using generic "device/reset".
>> +
>> +		This file allows to implement custom policy mechanism when VF is
>> +		misbehaving and triggering adverse events above defined thresholds.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent scheduling parameters of the functions.
>> +
>> +		These scheduling parameters can be changed even if VFs are enabled
>> +		and running, unless such change is not applicable on given platform
>> +		due to fixed hardware or firmware assignment.
>> +
>> +		exec_quantum_ms: (RW) integer 0..U32_MAX
>> +			The GT execution quantum in [ms] assigned to the function.
>> +			Requested quantum might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unlimited).
>> +
>> +		preempt_timeout_us: (RW) integer 0..U32_MAX
>> +			The GT preemption timeout in [us] assigned to the function.
>> +			Requested timeout might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unlimited).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent shared resource assigned to the functions.
>> +
>> +		These resource parameters can be changed, unless VF is already running,
>> +		or such change is not applicable on given platform due to fixed hardware
>> +		or firmware assignment.
>> +
>> +		Writes to these attributes may fail with:
>> +			-EPERM if change is not applicable on give HW/FW.
>> +			-E2BIG if value larger that HW/FW limit.
>> +			-EDQUOT if value is larger than maximum quota defined by the PF.
>> +			-ENOSPC if PF can't allocate required quota.
>> +			-EBUSY if the resource is currently in use by the VF.
>> +			-EIO if GuC refuses to change provisioning.
>> +
>> +		ggtt_quota: (RW) integer 0..U64_MAX
>> +			The size of the GGTT address space (in bytes) assigned to the VF.
>> +			The value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		lmem_quota: (RW) integer 0..U64_MAX
>> +			The size of the Local Memory (in bytes) assigned to the VF.
>> +			The value might be aligned per HW/FW requirements.
>> +
>> +			This attribute is only available on discrete platforms.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		contexts_quota: (RW) 0..U16_MAX
>> +			The number of GuC submission contexts assigned to the VF.
>> +			This value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		doorbells_quota: (RW) 0..U16_MAX
>> +			The number of GuC doorbells assigned to the VF.
>> +			This value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent threshold values used by the GuC to trigger
>> +		security events if adverse event monitoring is enabled.
>> +
>> +		These thresholds are checked every "monitoring_period_ms".
>> +		Refer to GuC ABI for details about each threshold category.
>> +
>> +		Default value for all thresholds is 0 (disabled).
>> +
>> +		cat_error_count: (RW) integer
>> +		doorbell_time_us: (RW) integer
>> +		engine_reset_count: (RW) integer
>> +		h2g_time_us: (RW) integer
>> +		irq_time_us: (RW) integer
>> +		page_fault_count: (RW) integer
>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
>> new file mode 100644
>> index 000000000000..574f6414eabb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>> @@ -0,0 +1,192 @@
>> +.. SPDX-License-Identifier: MIT
>> +
>> +========================
>> +Xe – SR-IOV Support Plan
>> +========================
>> +
>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
>> +specification suite is supported starting from 12th generation of Intel Graphics
>> +processors.
>> +
>> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
>> +provide flexible configuration and management options related to the SR-IOV.
>> +It will also highlight few most important changes to the Xe driver
>> +implementation to deal with Intel GPU SR-IOV specific requirements.
>> +
>> +
>> +SR-IOV Capability
>> +=================
>> +
>> +Due to SR-IOV complexity and required co-operation between hardware, firmware
>> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
>> +or fully functional.
>> +
>> +To control at the driver level which platform will provide support for SR-IOV,
>> +as we can't just rely on the PCI configuration data exposed by the hardware,
>> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
>> +a device capabilities that driver checks during the probe.
>> +
>> +Initially this flag will be set to disabled even on platforms that we plan to
>> +support. We will enable this flag only once we finish merging all required
>> +changes to the driver and related validated firmwares are also made available.
>> +
>> +
>> +SR-IOV Platforms
>> +================
>> +
>> +Initially we plan to add SR-IOV functionality to the following SDV platforms
>> +already supported by the Xe driver:
>> +
>> + - TGL (up to 7 VFs)
>> + - ADL (up to 7 VFs)
>> + - MTL (up to 7 VFs)
>> + - ATSM (up to 31 VFs)
>> + - PVC (up to 63 VFs)
>> +
>> +Newer platforms will be supported later, but we hope that enabling will be
>> +much faster, as majority of the driver changes are either platform agnostic
>> +or are similar between earlier platforms (hence we start with SDVs).
>> +
>> +
>> +PF Mode
>> +=======
>> +
>> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
>> +enabled by default.
>> +
>> +However, due to potentially conflicting requirements for SR-IOV and other mega
>> +features, we might want to have an option to disable SR-IOV PF mode support at
>> +the driver load time.
> 
> What about making SR-IOV support in Xe dependent on a separate build option, such
> as CONFIG_DRM_XE_SRIOV? This would allow users to enable SR-IOV with CONFIG_PCI_IOV
> to virtualize other devices, let's say a network adapter, but to keep this feature
> compiled out of Xe.

We can try to hide all SR-IOV specific code behind another configs:
	XE_SRIOV_PF bool depends on PCI_IOV default PCI_IOV
	XE_SRIOV_VF bool default n
but that's not so flexible and may still not work for all scenarios
where we have to manage multiple devices on one system (like use SRIOV
PF on one dgfx device, but not on other dgfx)

Michal

> 
> Francois
> 
>> +
>> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
>> +0 will force the driver to operate in the native (non-virtualized) mode.
>> +The same modparam could be used to limit number of supported Virtual Functions
>> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
>> +
>> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
>> +by default exposes hardware capability.
>> +
>> +The default value of this param will allow to support all possible VFs as
>> +claimed by the hardware.
>> +
>> +This modparam will have no effect if driver is running on the VF device.
>> +
>> +
>> +VFs Enabling
>> +============
>> +
>> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
>> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
>> +provision VFs in our custom implementation of the "sriov_configure" hook from
>> +the struct pci_driver.
>> +
>> +If for some reason, including explicit request to disable SR-IOV PF mode using
>> +modparam, we will not be able to correctly support any VFs, driver will change
>> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
>> +to 0, thus preventing configuration of the VFs.
>> +
>> +
>> +VF Mode
>> +=======
>> +
>> +When driver is running on the VF device, then due to hardware enforcements,
>> +access to the privileged registers is not possible. To avoid relying on these
>> +registers, we plan to perform early detection if we are running on the VF
>> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
>> +IS_SRIOV_VF(xe) to control the driver logic.
>> +
>> +To speed up merging of the required changes, we might first introduce dummy
>> +macro that is always set to false, to prepare driver to avoid some code paths
>> +before we finalize our VF mode detection and other VFs enabling changes.
>> +
>> +
>> +Resources
>> +=========
>> +
>> +Most of the hardware (or firmware) resources available on the Xe architecture,
>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
>> +VFs and will require some provisioning steps to assign those resources for use
>> +by the VF.
>> +
>> +Until VFs are provisioned with resources, the PF driver will be able to use all
>> +resources, in the same way as it would be running in non-virtualized mode.
>> +
>> +If some resource (of part or region of it) is assigned to specific VF, then PF
>> +is not allowed to use that part or region of the resource, but can continue to
>> +use whatever is left available.
>> +
>> +Those resources are usually fully virtualized, so they will not require any
>> +special handling when used by the VF driver, except that VF driver must know
>> +the assigned quota.
>> +
>> +The most notable exception is the GGTT address space, as on some platforms,
>> +the VF driver must additionally know the real range that it can access.
>> +
>> +Once the resources were assigned to the VF use and the VF driver has started,
>> +then it is not allowed to change such provisioning, as that would break the
>> +VF driver. To make changes the VF driver, which was using these resources,
>> +must be unloaded (or the VM is terminated) and the VF device must be reset
>> +using the FLR.
>> +
>> +
>> +Scheduling
>> +==========
>> +
>> +The workloads from PF driver and VF drivers must be submitted to the hardware
>> +always by using the GuC submission mechanism. Unless VF has exclusive access
>> +to the GT then submissions from different VFs are time-sliced and controlled
>> +with additional "execution_quantum" and "preemption_timeout" parameters.
>> +
>> +In contrast to the resource provisioning, those scheduling parameters can be
>> +changed even if VF drivers are already running and are active.
>> +
>> +
>> +Automatic VFs Provisioning
>> +==========================
>> +
>> +To provide out-of-the box experience when user will be enabling VFs using
>> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
>> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
>> +
>> +By default, all VFs will be allocated with the fair amount of the mandatory
>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
>> +Such provisioning should be sufficient for most of the normal usages, when
>> +no strict SLA is required.
>> +
>> +The PF driver will also expose some additional sysfs files to allow adjusting
>> +this automatic VFs provisioning, like default values for most of the
>> +provisioning parameters that PF will then apply for each enabled VF.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +Manual VFs Provisioning
>> +=======================
>> +
>> +If automatic VFs provisioning, which applies same configuration to every VF,
>> +is not sufficient or there is a need for advanced customization of some VF,
>> +the PF driver will also provide extended sysfs interface which will allow
>> +control every provisioning attribute to the lowest feasible level.
>> +
>> +It is expected that these low-level attributes will be mostly used by the
>> +advanced users or by the custom tools that will setup configurations that
>> +meet predefined and validated SLA as required by the customers.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +VFs Monitoring
>> +==============
>> +
>> +In addition to the resource provisioning or changing scheduling parameters,
>> +the PF driver might also allow configure some monitoring parameters, like
>> +thresholds of adverse events or sample period, to track undesired behavior
>> +of the VFs that could impact the whole system.
>> +
>> +Once those thresholds are setup and sampling period is defined, the GuC will
>> +notify the PF driver about which VF is excessing the threshold and then PF is
>> +able to trigger the uevent to notify the administrator (or VMM) that could
>> +take some action against the VF.
>> -- 
>> 2.25.1
>>

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 10:29     ` Michal Wajdeczko
  0 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 10:29 UTC (permalink / raw)
  To: Francois Dugast
  Cc: Daniel Vetter, Tvrtko Ursulin, intel-xe, dri-devel, Rodrigo Vivi



On 14.11.2023 11:08, Francois Dugast wrote:
> On Fri, Nov 10, 2023 at 07:22:31PM +0100, Michal Wajdeczko wrote:
>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe) specification suite is supported starting from 12th
>> generation of Intel Graphics processors.
>>
>> This RFC aims to explain how do we want to add support for SR-IOV
>> to the new Xe driver and to propose related additions to the sysfs.
>>
>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>> Cc: Oded Gabbay <ogabbay@kernel.org>
>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>> Cc: Daniel Vetter <daniel@ffwll.ch>
>> ---
>>  Documentation/gpu/rfc/index.rst             |   5 +
>>  Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>>  Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>  3 files changed, 698 insertions(+)
>>  create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>  create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>
>> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
>> index e4f7b005138d..fc5bc447f30d 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -35,3 +35,8 @@ host such documentation:
>>  .. toctree::
>>  
>>     xe.rst
>> +
>> +.. toctree::
>> +   :maxdepth: 1
>> +
>> +   xe_sriov.rst
>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> new file mode 100644
>> index 000000000000..77748204dd83
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> @@ -0,0 +1,501 @@
>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>> +..
>> +.. Intel Xe driver ABI (SR-IOV extensions)
>> +..
>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>> +    the PCI Express (PCIe) specification suite is supported
>> +    starting from 12th generation of Intel Graphics processors.
>> +
>> +    This document describes Xe driver specific additions.
>> +
>> +    For description of generic SR-IOV sysfs attributes see
>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_auto_provisioning
>> +    │   ├── admin_mode
>> +    │   ├── enabled
>> +    │   ├── reset_defaults
>> +    │   ├── resources
>> +    │   │   ├── default_contexts_quota
>> +    │   │   ├── default_doorbells_quota
>> +    │   │   ├── default_ggtt_quota
>> +    │   │   └── default_lmem_quota
>> +    │   ├── scheduling
>> +    │   │   ├── default_exec_quantum_ms
>> +    │   │   └── default_preempt_timeout_us
>> +    │   └── monitoring
>> +    │       ├── default_cat_error_count
>> +    │       ├── default_doorbell_time_us
>> +    │       ├── default_engine_reset_count
>> +    │       ├── default_h2g_time_us
>> +    │       ├── default_irq_time_us
>> +    │       └── default_page_fault_count
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_extensions
>> +    │   ├── monitoring_period_ms
>> +    │   ├── strict_scheduling_enabled
>> +    │   ├── pf
>> +    │   │   ├── device -> ../../../BDF
>> +    │   │   ├── priority
>> +    │   │   ├── tile0
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   ├── vf1
>> +    │   │   ├── device -> ../../../BDF+1
>> +    │   │   ├── stop
>> +    │   │   ├── tile0
>> +    │   │   │   ├── ggtt_quota
>> +    │   │   │   ├── lmem_quota
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── contexts_quota
>> +    │   │   │   │   ├── doorbells_quota
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   └── vfN
>> +..
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory appears on the device when:
>> +
>> +		 - device supports SR-IOV, and
>> +		 - device is a Physical Function (PF), and
>> +		 - xe driver supports SR-IOV PF on given device, and
>> +		 - xe driver supports automatic VFs provisioning.
>> +
>> +		This directory is used as a root for all attributes related to
>> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
>> +		Virtual Functions (VFs).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool (0, 1)
>> +
>> +		This file represents configuration flag for the automatic VFs
>> +		(un)provisioning that could be performed by the PF.
>> +
>> +		The default value is 1 (true).
>> +
>> +		This flag can be set to false, unless manual provisioning is not
>> +		applicable for given platform or it is not supported by current
>> +		PF implementation. In such cases -EPERM will be returned.
>> +
>> +		This flag will be automatically set to false when there will be
>> +		other attempts to change any of VF's resource provisioning.
>> +		See "sriov_extensions" section for details.
>> +
>> +		This flag can be set back to true if and only if all VFs are
>> +		fully unprovisioned, otherwise -EEXIST error will be returned.
>> +
>> +		false = "disabled"
>> +			When disabled, then PF will not attempt to do automatic
>> +			VFs provisioning when VFs are being enabled and will not
>> +			perform automatic unprovisioning of the VFs when VFs will
>> +			be disabled.
>> +
>> +		true = "enabled"
>> +			When enabled, then on VFs enabling PF will do automatic
>> +			VFs provisioning based on the default settings described
>> +			below.
>> +
>> +			If automatic VFs provisioning fails due to some reasons,
>> +			then VFs will not be enabled.
>> +
>> +			If enabled, all resources allocated during VFs enabling
>> +			will be released during VFs disabling (automatic unprovisioning).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool (0, 1)
>> +
>> +		This file represents configuration flag for the automatic VFs
>> +		provisioning that could be performed by the PF.
>> +
>> +		The default value depends on the platform type.
>> +
>> +		This flag can be changed any time, but will have no effect if
>> +		VFs are already provisioned.
>> +
>> +		If enabled (default on discrete platforms) then the PF will
>> +		retain only minimum hardcoded resources for its own use when
>> +		doing VFs automatic provisioning and will not use any default
>> +		values described below for its own configuration.
>> +
>> +		If disabled (default on integrated platforms) then the PF will
>> +		treat itself like yet another additional VF in all fair resource
>> +		allocations and will also try to apply default provisioning
>> +		values described below for its own configuration.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(WO) bool (1)
>> +
>> +		Writing to this file will reset all default provisioning parameters
>> +		listed below to the default values.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent default provisioning that should be used
>> +		for VFs automatic provisioning.
>> +
>> +		These values can be changed any time, but will have no effect if
>> +		VFs are already provisioned.
>> +
>> +		default_contexts_quota: (RW) integer 0..U32_MAX
>> +			The number of GuC context IDs to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
>> +
>> +		default_doorbells_quota: (RW) integer 0..U32_MAX
>> +			The number of GuC doorbells to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
>> +
>> +		default_ggtt_quota: (RW) integer 0..U32_MAX
>> +			The size of the GGTT address space (in bytes) to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
>> +
>> +		default_lmem_quota: (RW) integer 0..U32_MAX
>> +			The size of the LMEM (in bytes) to provide to the VF.
>> +			The default value is 0 (use fair allocations).
>> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
>> +
>> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
>> +			The GT execution quantum (in millisecs) assigned to the function.
>> +			The default value is 0 (infinify).
>> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
>> +
>> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
>> +			The GT preemption timeout (in microsecs) assigned to the function.
>> +			The default value is 0 (infinity).
>> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
>> +
>> +		default_cat_error_count: (RW) integer 0..U32_MAX
>> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
>> +		default_engine_reset_count: (RW) integer 0..U32_MAX
>> +		default_h2g_time_us: (RW) integer 0..U32_MAX
>> +		default_irq_time_us: (RW) integer 0..U32_MAX
>> +		default_page_fault_count: (RW) integer 0..U32_MAX
>> +			The monitoring threshold to be set for the function.
>> +			The default value is 0 (don't monitor).
>> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory appears on Xe device when:
>> +
>> +		 - device supports SR-IOV, and
>> +		 - device is a Physical Function (PF), and
>> +		 - driver is enabled to support SR-IOV PF on given device.
>> +
>> +		This directory is used as a root for all attributes required to
>> +		manage both Physical Function (PF) and Virtual Functions (VFs).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) bool
>> +
>> +		This file represents a flag used to determine if scheduling
>> +		parameters should be respected even if there is no active
>> +		workloads submitted by the PF or VFs.
>> +
>> +		This flag is disabled by default, unless strict scheduling is
>> +		not applicable on given platform. In such case this file will
>> +		be read-only.
>> +
>> +		The change to this file may have no effect if VFs are not yet enabled.
>> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) integer
>> +
>> +		This file represents the configuration knob used by adverse event
>> +		monitoring. A value here is the period in millisecs during which
>> +		events are counted and the total is checked against a threshold.
>> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
>> +
>> +		Default is 0 (monitoring is disabled).
>> +
>> +		If monitoring capability is not available, then attempt to enable
>> +		will fail with -EPERM error. If monitoring can't be enabled in
>> +		GuC then write will fail with -EIO.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all attributes related to the SR-IOV
>> +		Physical Function (PF).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all attributes related to the SR-IOV
>> +		Virtual Function (VF).
>> +
>> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
>> +		The Xe driver implementaton follows that naming schema.
>> +
>> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
>> +		value of the PCI "sriov_totalvfs" attribute.
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all SR-IOV attributes related to the device tile.
>> +		The tile numbers (T) start from 0.
>> +
>> +		There is at least one "tile0/" directory present.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		This directory holds all SR-IOV attributes related to the device GT.
>> +		The GT numbers (X) start from 0.
>> +
>> +		There is at least one "gt0/" directory present.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(symbolic link)
>> +
>> +		Backlink to the PCI device entry representing given function.
>> +		For PF this link is always present.
>> +		For VF this link is present only for currently enabled VFs.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(RW) string
>> +
>> +		This file represents a GuC Scheduler knob to override the default
>> +		round-robin or FIFO scheduler policies implemented by the GuC.
>> +
>> +		The default value is "peer".
>> +
>> +		This flag can be changed, unless such change is not applicable
>> +		for given platform or is not supported by current GuC firmware.
>> +		In such case this file could be read-only or will return -EPERM
>> +		on write attempt.
>> +
>> +		"immediate"
>> +			GuC will Schedule PF workloads immediately and PF
>> +			workloads only until the PF's work queues in GuC
>> +			are empty.
>> +
>> +		"lazy"
>> +			GuC will Schedule PF workloads at the next opportune
>> +			moment and PF workloads only until the PF work queues
>> +			in GuC are empty.
>> +
>> +		"peer"
>> +			GuC Scheduler will treat PF and VFs with equal priority.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		(WO) bool (1)
>> +
>> +		Write to this file will force GuC to stop handle any requests from
>> +		this VF, but without triggering a FLR.
>> +		To recover, the full FLR must be issued using generic "device/reset".
>> +
>> +		This file allows to implement custom policy mechanism when VF is
>> +		misbehaving and triggering adverse events above defined thresholds.
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent scheduling parameters of the functions.
>> +
>> +		These scheduling parameters can be changed even if VFs are enabled
>> +		and running, unless such change is not applicable on given platform
>> +		due to fixed hardware or firmware assignment.
>> +
>> +		exec_quantum_ms: (RW) integer 0..U32_MAX
>> +			The GT execution quantum in [ms] assigned to the function.
>> +			Requested quantum might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unlimited).
>> +
>> +		preempt_timeout_us: (RW) integer 0..U32_MAX
>> +			The GT preemption timeout in [us] assigned to the function.
>> +			Requested timeout might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unlimited).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent shared resource assigned to the functions.
>> +
>> +		These resource parameters can be changed, unless VF is already running,
>> +		or such change is not applicable on given platform due to fixed hardware
>> +		or firmware assignment.
>> +
>> +		Writes to these attributes may fail with:
>> +			-EPERM if change is not applicable on give HW/FW.
>> +			-E2BIG if value larger that HW/FW limit.
>> +			-EDQUOT if value is larger than maximum quota defined by the PF.
>> +			-ENOSPC if PF can't allocate required quota.
>> +			-EBUSY if the resource is currently in use by the VF.
>> +			-EIO if GuC refuses to change provisioning.
>> +
>> +		ggtt_quota: (RW) integer 0..U64_MAX
>> +			The size of the GGTT address space (in bytes) assigned to the VF.
>> +			The value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		lmem_quota: (RW) integer 0..U64_MAX
>> +			The size of the Local Memory (in bytes) assigned to the VF.
>> +			The value might be aligned per HW/FW requirements.
>> +
>> +			This attribute is only available on discrete platforms.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		contexts_quota: (RW) 0..U16_MAX
>> +			The number of GuC submission contexts assigned to the VF.
>> +			This value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +		doorbells_quota: (RW) 0..U16_MAX
>> +			The number of GuC doorbells assigned to the VF.
>> +			This value might be aligned per HW/FW requirements.
>> +
>> +			Default is 0 (unprovisioned).
>> +
>> +
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
>> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
>> +Date:		2024
>> +KernelVersion:	TBD
>> +Contact:	intel-xe@lists.freedesktop.org
>> +Description:
>> +		These files represent threshold values used by the GuC to trigger
>> +		security events if adverse event monitoring is enabled.
>> +
>> +		These thresholds are checked every "monitoring_period_ms".
>> +		Refer to GuC ABI for details about each threshold category.
>> +
>> +		Default value for all thresholds is 0 (disabled).
>> +
>> +		cat_error_count: (RW) integer
>> +		doorbell_time_us: (RW) integer
>> +		engine_reset_count: (RW) integer
>> +		h2g_time_us: (RW) integer
>> +		irq_time_us: (RW) integer
>> +		page_fault_count: (RW) integer
>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
>> new file mode 100644
>> index 000000000000..574f6414eabb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>> @@ -0,0 +1,192 @@
>> +.. SPDX-License-Identifier: MIT
>> +
>> +========================
>> +Xe – SR-IOV Support Plan
>> +========================
>> +
>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
>> +specification suite is supported starting from 12th generation of Intel Graphics
>> +processors.
>> +
>> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
>> +provide flexible configuration and management options related to the SR-IOV.
>> +It will also highlight few most important changes to the Xe driver
>> +implementation to deal with Intel GPU SR-IOV specific requirements.
>> +
>> +
>> +SR-IOV Capability
>> +=================
>> +
>> +Due to SR-IOV complexity and required co-operation between hardware, firmware
>> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
>> +or fully functional.
>> +
>> +To control at the driver level which platform will provide support for SR-IOV,
>> +as we can't just rely on the PCI configuration data exposed by the hardware,
>> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
>> +a device capabilities that driver checks during the probe.
>> +
>> +Initially this flag will be set to disabled even on platforms that we plan to
>> +support. We will enable this flag only once we finish merging all required
>> +changes to the driver and related validated firmwares are also made available.
>> +
>> +
>> +SR-IOV Platforms
>> +================
>> +
>> +Initially we plan to add SR-IOV functionality to the following SDV platforms
>> +already supported by the Xe driver:
>> +
>> + - TGL (up to 7 VFs)
>> + - ADL (up to 7 VFs)
>> + - MTL (up to 7 VFs)
>> + - ATSM (up to 31 VFs)
>> + - PVC (up to 63 VFs)
>> +
>> +Newer platforms will be supported later, but we hope that enabling will be
>> +much faster, as majority of the driver changes are either platform agnostic
>> +or are similar between earlier platforms (hence we start with SDVs).
>> +
>> +
>> +PF Mode
>> +=======
>> +
>> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
>> +enabled by default.
>> +
>> +However, due to potentially conflicting requirements for SR-IOV and other mega
>> +features, we might want to have an option to disable SR-IOV PF mode support at
>> +the driver load time.
> 
> What about making SR-IOV support in Xe dependent on a separate build option, such
> as CONFIG_DRM_XE_SRIOV? This would allow users to enable SR-IOV with CONFIG_PCI_IOV
> to virtualize other devices, let's say a network adapter, but to keep this feature
> compiled out of Xe.

We can try to hide all SR-IOV specific code behind another configs:
	XE_SRIOV_PF bool depends on PCI_IOV default PCI_IOV
	XE_SRIOV_VF bool default n
but that's not so flexible and may still not work for all scenarios
where we have to manage multiple devices on one system (like use SRIOV
PF on one dgfx device, but not on other dgfx)

Michal

> 
> Francois
> 
>> +
>> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
>> +0 will force the driver to operate in the native (non-virtualized) mode.
>> +The same modparam could be used to limit number of supported Virtual Functions
>> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
>> +
>> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
>> +by default exposes hardware capability.
>> +
>> +The default value of this param will allow to support all possible VFs as
>> +claimed by the hardware.
>> +
>> +This modparam will have no effect if driver is running on the VF device.
>> +
>> +
>> +VFs Enabling
>> +============
>> +
>> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
>> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
>> +provision VFs in our custom implementation of the "sriov_configure" hook from
>> +the struct pci_driver.
>> +
>> +If for some reason, including explicit request to disable SR-IOV PF mode using
>> +modparam, we will not be able to correctly support any VFs, driver will change
>> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
>> +to 0, thus preventing configuration of the VFs.
>> +
>> +
>> +VF Mode
>> +=======
>> +
>> +When driver is running on the VF device, then due to hardware enforcements,
>> +access to the privileged registers is not possible. To avoid relying on these
>> +registers, we plan to perform early detection if we are running on the VF
>> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
>> +IS_SRIOV_VF(xe) to control the driver logic.
>> +
>> +To speed up merging of the required changes, we might first introduce dummy
>> +macro that is always set to false, to prepare driver to avoid some code paths
>> +before we finalize our VF mode detection and other VFs enabling changes.
>> +
>> +
>> +Resources
>> +=========
>> +
>> +Most of the hardware (or firmware) resources available on the Xe architecture,
>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
>> +VFs and will require some provisioning steps to assign those resources for use
>> +by the VF.
>> +
>> +Until VFs are provisioned with resources, the PF driver will be able to use all
>> +resources, in the same way as it would be running in non-virtualized mode.
>> +
>> +If some resource (of part or region of it) is assigned to specific VF, then PF
>> +is not allowed to use that part or region of the resource, but can continue to
>> +use whatever is left available.
>> +
>> +Those resources are usually fully virtualized, so they will not require any
>> +special handling when used by the VF driver, except that VF driver must know
>> +the assigned quota.
>> +
>> +The most notable exception is the GGTT address space, as on some platforms,
>> +the VF driver must additionally know the real range that it can access.
>> +
>> +Once the resources were assigned to the VF use and the VF driver has started,
>> +then it is not allowed to change such provisioning, as that would break the
>> +VF driver. To make changes the VF driver, which was using these resources,
>> +must be unloaded (or the VM is terminated) and the VF device must be reset
>> +using the FLR.
>> +
>> +
>> +Scheduling
>> +==========
>> +
>> +The workloads from PF driver and VF drivers must be submitted to the hardware
>> +always by using the GuC submission mechanism. Unless VF has exclusive access
>> +to the GT then submissions from different VFs are time-sliced and controlled
>> +with additional "execution_quantum" and "preemption_timeout" parameters.
>> +
>> +In contrast to the resource provisioning, those scheduling parameters can be
>> +changed even if VF drivers are already running and are active.
>> +
>> +
>> +Automatic VFs Provisioning
>> +==========================
>> +
>> +To provide out-of-the box experience when user will be enabling VFs using
>> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
>> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
>> +
>> +By default, all VFs will be allocated with the fair amount of the mandatory
>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
>> +Such provisioning should be sufficient for most of the normal usages, when
>> +no strict SLA is required.
>> +
>> +The PF driver will also expose some additional sysfs files to allow adjusting
>> +this automatic VFs provisioning, like default values for most of the
>> +provisioning parameters that PF will then apply for each enabled VF.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +Manual VFs Provisioning
>> +=======================
>> +
>> +If automatic VFs provisioning, which applies same configuration to every VF,
>> +is not sufficient or there is a need for advanced customization of some VF,
>> +the PF driver will also provide extended sysfs interface which will allow
>> +control every provisioning attribute to the lowest feasible level.
>> +
>> +It is expected that these low-level attributes will be mostly used by the
>> +advanced users or by the custom tools that will setup configurations that
>> +meet predefined and validated SLA as required by the customers.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +VFs Monitoring
>> +==============
>> +
>> +In addition to the resource provisioning or changing scheduling parameters,
>> +the PF driver might also allow configure some monitoring parameters, like
>> +thresholds of adverse events or sample period, to track undesired behavior
>> +of the VFs that could impact the whole system.
>> +
>> +Once those thresholds are setup and sampling period is defined, the GuC will
>> +notify the PF driver about which VF is excessing the threshold and then PF is
>> +able to trigger the uevent to notify the administrator (or VMM) that could
>> +take some action against the VF.
>> -- 
>> 2.25.1
>>

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-10 18:22 ` [Intel-xe] " Michal Wajdeczko
@ 2023-11-14 12:37   ` Tvrtko Ursulin
  -1 siblings, 0 replies; 30+ messages in thread
From: Tvrtko Ursulin @ 2023-11-14 12:37 UTC (permalink / raw)
  To: Michal Wajdeczko, intel-xe, dri-devel; +Cc: Oded Gabbay, Rodrigo Vivi


On 10/11/2023 18:22, Michal Wajdeczko wrote:
> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> Express (PCIe) specification suite is supported starting from 12th
> generation of Intel Graphics processors.
> 
> This RFC aims to explain how do we want to add support for SR-IOV
> to the new Xe driver and to propose related additions to the sysfs.
> 
> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Cc: Oded Gabbay <ogabbay@kernel.org>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> Cc: Daniel Vetter <daniel@ffwll.ch>
> ---
>   Documentation/gpu/rfc/index.rst             |   5 +
>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>   3 files changed, 698 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> 
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index e4f7b005138d..fc5bc447f30d 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -35,3 +35,8 @@ host such documentation:
>   .. toctree::
>   
>      xe.rst
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   xe_sriov.rst
> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> new file mode 100644
> index 000000000000..77748204dd83
> --- /dev/null
> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> @@ -0,0 +1,501 @@
> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> +..
> +.. Intel Xe driver ABI (SR-IOV extensions)
> +..
> +    The Single Root I/O Virtualization (SR-IOV) extension to
> +    the PCI Express (PCIe) specification suite is supported
> +    starting from 12th generation of Intel Graphics processors.
> +
> +    This document describes Xe driver specific additions.
> +
> +    For description of generic SR-IOV sysfs attributes see
> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_auto_provisioning
> +    │   ├── admin_mode
> +    │   ├── enabled
> +    │   ├── reset_defaults
> +    │   ├── resources
> +    │   │   ├── default_contexts_quota
> +    │   │   ├── default_doorbells_quota
> +    │   │   ├── default_ggtt_quota
> +    │   │   └── default_lmem_quota
> +    │   ├── scheduling
> +    │   │   ├── default_exec_quantum_ms
> +    │   │   └── default_preempt_timeout_us
> +    │   └── monitoring
> +    │       ├── default_cat_error_count
> +    │       ├── default_doorbell_time_us
> +    │       ├── default_engine_reset_count
> +    │       ├── default_h2g_time_us
> +    │       ├── default_irq_time_us
> +    │       └── default_page_fault_count

 From the department of bike-shedding, one alternative could be to have 
a directory called defaults which avoids having to have the default_ 
prefix on everything under it.

> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_extensions

Should this be xe_sriov_extensions or if not doesn't it need agreement 
to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
Sriov_auto_provisioning too I guess.

> +    │   ├── monitoring_period_ms
> +    │   ├── strict_scheduling_enabled
> +    │   ├── pf
> +    │   │   ├── device -> ../../../BDF
> +    │   │   ├── priority
> +    │   │   ├── tile0
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   ├── vf1
> +    │   │   ├── device -> ../../../BDF+1
> +    │   │   ├── stop
> +    │   │   ├── tile0
> +    │   │   │   ├── ggtt_quota
> +    │   │   │   ├── lmem_quota
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── contexts_quota
> +    │   │   │   │   ├── doorbells_quota
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   └── vfN
> +..
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on the device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - xe driver supports SR-IOV PF on given device, and
> +		 - xe driver supports automatic VFs provisioning.
> +
> +		This directory is used as a root for all attributes related to
> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
> +		Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		(un)provisioning that could be performed by the PF.
> +
> +		The default value is 1 (true).
> +
> +		This flag can be set to false, unless manual provisioning is not
> +		applicable for given platform or it is not supported by current
> +		PF implementation. In such cases -EPERM will be returned.
> +
> +		This flag will be automatically set to false when there will be
> +		other attempts to change any of VF's resource provisioning.
> +		See "sriov_extensions" section for details.
> +
> +		This flag can be set back to true if and only if all VFs are
> +		fully unprovisioned, otherwise -EEXIST error will be returned.
> +
> +		false = "disabled"
> +			When disabled, then PF will not attempt to do automatic
> +			VFs provisioning when VFs are being enabled and will not
> +			perform automatic unprovisioning of the VFs when VFs will
> +			be disabled.
> +
> +		true = "enabled"
> +			When enabled, then on VFs enabling PF will do automatic
> +			VFs provisioning based on the default settings described
> +			below.
> +
> +			If automatic VFs provisioning fails due to some reasons,
> +			then VFs will not be enabled.
> +
> +			If enabled, all resources allocated during VFs enabling
> +			will be released during VFs disabling (automatic unprovisioning).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		provisioning that could be performed by the PF.
> +
> +		The default value depends on the platform type.
> +
> +		This flag can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		If enabled (default on discrete platforms) then the PF will
> +		retain only minimum hardcoded resources for its own use when
> +		doing VFs automatic provisioning and will not use any default
> +		values described below for its own configuration.
> +
> +		If disabled (default on integrated platforms) then the PF will
> +		treat itself like yet another additional VF in all fair resource
> +		allocations and will also try to apply default provisioning
> +		values described below for its own configuration.
> +

One alternative could be to expose two sets of defaults, the PF and VF 
ones. With the advantage of allowing the "admin mode" / "minimal PF" to 
be explicitly configurable instead of hardcoded. Should be more flexible.

If the discrete vs integrated distinction is wanted it could simply be 
made by initialy populating (driver init) the respective defaults based 
on the platform type.

> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Writing to this file will reset all default provisioning parameters
> +		listed below to the default values.

Maybe this isn't required if you can say it is the responsibility of 
whoever changes the defaults to either know what they are doing, or to 
save and restore themselves if they. It is not a major concern but if 
writing kernel code can be avoided perhaps it can be considered.

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent default provisioning that should be used
> +		for VFs automatic provisioning.
> +
> +		These values can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		default_contexts_quota: (RW) integer 0..U32_MAX
> +			The number of GuC context IDs to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> +
> +		default_doorbells_quota: (RW) integer 0..U32_MAX
> +			The number of GuC doorbells to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> +
> +		default_ggtt_quota: (RW) integer 0..U32_MAX
> +			The size of the GGTT address space (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
> +
> +		default_lmem_quota: (RW) integer 0..U32_MAX
> +			The size of the LMEM (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
> +
> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum (in millisecs) assigned to the function.
> +			The default value is 0 (infinify).
> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> +
> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout (in microsecs) assigned to the function.
> +			The default value is 0 (infinity).
> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.

I have a slight concern here on the usability of GuC specific tunables.

Whereas one can imagine an external entity (some admin, somewhere) to 
probably pretty much understand what it means to partition the local 
memory, address space, and set the scheduling timeouts (all are intuitve 
and obvious concepts), how are they suppose to approach the GuC 
doorbells and contexts?

It could be a matter of adding more documentation for those two, or it 
even could make sense to shove them under a guc prefix (or 
subdirectory?) to signify the fact they are implementation details and 
not a fundamental concept.

> +
> +		default_cat_error_count: (RW) integer 0..U32_MAX
> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
> +		default_engine_reset_count: (RW) integer 0..U32_MAX
> +		default_h2g_time_us: (RW) integer 0..U32_MAX
> +		default_irq_time_us: (RW) integer 0..U32_MAX
> +		default_page_fault_count: (RW) integer 0..U32_MAX
> +			The monitoring threshold to be set for the function.
> +			The default value is 0 (don't monitor).
> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on Xe device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - driver is enabled to support SR-IOV PF on given device.
> +
> +		This directory is used as a root for all attributes required to
> +		manage both Physical Function (PF) and Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool
> +
> +		This file represents a flag used to determine if scheduling
> +		parameters should be respected even if there is no active
> +		workloads submitted by the PF or VFs.
> +
> +		This flag is disabled by default, unless strict scheduling is
> +		not applicable on given platform. In such case this file will
> +		be read-only.
> +
> +		The change to this file may have no effect if VFs are not yet enabled.
> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.

I think the semantics of this need to be documented ie. how it interacts 
with exec_quantum_ms. If it does? I am guessing that it has to otherwise 
I don't know what it would mean - presumably unused timeslices are not 
given to other entities but time just goes wasted? But it is also a 
question on over what time interval. Or that too is purely defined by 
the number of PF+VFs and their respective allocated quanta.

Also, would there be benefit, assuming it is possible with GuC, to allow 
configuring it per PF/VF?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) integer
> +
> +		This file represents the configuration knob used by adverse event
> +		monitoring. A value here is the period in millisecs during which
> +		events are counted and the total is checked against a threshold.
> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
> +
> +		Default is 0 (monitoring is disabled).
> +
> +		If monitoring capability is not available, then attempt to enable
> +		will fail with -EPERM error. If monitoring can't be enabled in
> +		GuC then write will fail with -EIO.

Could the docs explain if there is a downside to enabling it, which is 
probably why it isn't enabled by default? Because it does sound natural 
that adverse events should be noticed.

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Physical Function (PF).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Virtual Function (VF).
> +
> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
> +		The Xe driver implementaton follows that naming schema.
> +
> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
> +		value of the PCI "sriov_totalvfs" attribute.
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device tile.
> +		The tile numbers (T) start from 0.
> +
> +		There is at least one "tile0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device GT.
> +		The GT numbers (X) start from 0.
> +
> +		There is at least one "gt0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(symbolic link)
> +
> +		Backlink to the PCI device entry representing given function.
> +		For PF this link is always present.
> +		For VF this link is present only for currently enabled VFs.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) string
> +
> +		This file represents a GuC Scheduler knob to override the default
> +		round-robin or FIFO scheduler policies implemented by the GuC.
> +
> +		The default value is "peer".
> +
> +		This flag can be changed, unless such change is not applicable
> +		for given platform or is not supported by current GuC firmware.
> +		In such case this file could be read-only or will return -EPERM
> +		on write attempt.
> +
> +		"immediate"
> +			GuC will Schedule PF workloads immediately and PF
> +			workloads only until the PF's work queues in GuC
> +			are empty.
> +
> +		"lazy"
> +			GuC will Schedule PF workloads at the next opportune
> +			moment and PF workloads only until the PF work queues
> +			in GuC are empty.
> +
> +		"peer"
> +			GuC Scheduler will treat PF and VFs with equal priority.

Hmmm this is too very GuC specific and I wonder what is the usecase for 
lazy? Lazy = "don't care when it runs, but when it runs it will run 
everything queued so far", right? Feels a bit odd on first.

"Immediate" may also not be depending on preemption granularity and 
workloads, right?

Are there any ideas to express the knobs in a more generic fashion?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Write to this file will force GuC to stop handle any requests from
> +		this VF, but without triggering a FLR.
> +		To recover, the full FLR must be issued using generic "device/reset".
> +
> +		This file allows to implement custom policy mechanism when VF is
> +		misbehaving and triggering adverse events above defined thresholds.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent scheduling parameters of the functions.
> +
> +		These scheduling parameters can be changed even if VFs are enabled
> +		and running, unless such change is not applicable on given platform
> +		due to fixed hardware or firmware assignment.
> +
> +		exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum in [ms] assigned to the function.
> +			Requested quantum might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +		preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout in [us] assigned to the function.
> +			Requested timeout might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).

Alignment for the above two will be visible after read-back?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent shared resource assigned to the functions.
> +
> +		These resource parameters can be changed, unless VF is already running,
> +		or such change is not applicable on given platform due to fixed hardware
> +		or firmware assignment.
> +
> +		Writes to these attributes may fail with:
> +			-EPERM if change is not applicable on give HW/FW.
> +			-E2BIG if value larger that HW/FW limit.
> +			-EDQUOT if value is larger than maximum quota defined by the PF.
> +			-ENOSPC if PF can't allocate required quota.
> +			-EBUSY if the resource is currently in use by the VF.
> +			-EIO if GuC refuses to change provisioning.

Why it would refuse if input is valid? In other words, what is the 
user/admin supposed to do on -EIO?

> +
> +		ggtt_quota: (RW) integer 0..U64_MAX
> +			The size of the GGTT address space (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		lmem_quota: (RW) integer 0..U64_MAX
> +			The size of the Local Memory (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			This attribute is only available on discrete platforms.
> +
> +			Default is 0 (unprovisioned).
> +
> +		contexts_quota: (RW) 0..U16_MAX
> +			The number of GuC submission contexts assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		doorbells_quota: (RW) 0..U16_MAX
> +			The number of GuC doorbells assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent threshold values used by the GuC to trigger
> +		security events if adverse event monitoring is enabled.

How are the security events delivered? There is mention of uevents in a 
later paragraph - are they already defined or should be together with 
this so the link can be place here?

> +
> +		These thresholds are checked every "monitoring_period_ms".
> +		Refer to GuC ABI for details about each threshold category.

Is it possible to have a link here to GuC ABI?

Regards,

Tvrtko

> +
> +		Default value for all thresholds is 0 (disabled).
> +
> +		cat_error_count: (RW) integer
> +		doorbell_time_us: (RW) integer
> +		engine_reset_count: (RW) integer
> +		h2g_time_us: (RW) integer
> +		irq_time_us: (RW) integer
> +		page_fault_count: (RW) integer
> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
> new file mode 100644
> index 000000000000..574f6414eabb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/xe_sriov.rst
> @@ -0,0 +1,192 @@
> +.. SPDX-License-Identifier: MIT
> +
> +========================
> +Xe – SR-IOV Support Plan
> +========================
> +
> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
> +specification suite is supported starting from 12th generation of Intel Graphics
> +processors.
> +
> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
> +provide flexible configuration and management options related to the SR-IOV.
> +It will also highlight few most important changes to the Xe driver
> +implementation to deal with Intel GPU SR-IOV specific requirements.
> +
> +
> +SR-IOV Capability
> +=================
> +
> +Due to SR-IOV complexity and required co-operation between hardware, firmware
> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
> +or fully functional.
> +
> +To control at the driver level which platform will provide support for SR-IOV,
> +as we can't just rely on the PCI configuration data exposed by the hardware,
> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
> +a device capabilities that driver checks during the probe.
> +
> +Initially this flag will be set to disabled even on platforms that we plan to
> +support. We will enable this flag only once we finish merging all required
> +changes to the driver and related validated firmwares are also made available.
> +
> +
> +SR-IOV Platforms
> +================
> +
> +Initially we plan to add SR-IOV functionality to the following SDV platforms
> +already supported by the Xe driver:
> +
> + - TGL (up to 7 VFs)
> + - ADL (up to 7 VFs)
> + - MTL (up to 7 VFs)
> + - ATSM (up to 31 VFs)
> + - PVC (up to 63 VFs)
> +
> +Newer platforms will be supported later, but we hope that enabling will be
> +much faster, as majority of the driver changes are either platform agnostic
> +or are similar between earlier platforms (hence we start with SDVs).
> +
> +
> +PF Mode
> +=======
> +
> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
> +enabled by default.
> +
> +However, due to potentially conflicting requirements for SR-IOV and other mega
> +features, we might want to have an option to disable SR-IOV PF mode support at
> +the driver load time.
> +
> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
> +0 will force the driver to operate in the native (non-virtualized) mode.
> +The same modparam could be used to limit number of supported Virtual Functions
> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
> +
> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
> +by default exposes hardware capability.
> +
> +The default value of this param will allow to support all possible VFs as
> +claimed by the hardware.
> +
> +This modparam will have no effect if driver is running on the VF device.
> +
> +
> +VFs Enabling
> +============
> +
> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
> +provision VFs in our custom implementation of the "sriov_configure" hook from
> +the struct pci_driver.
> +
> +If for some reason, including explicit request to disable SR-IOV PF mode using
> +modparam, we will not be able to correctly support any VFs, driver will change
> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
> +to 0, thus preventing configuration of the VFs.
> +
> +
> +VF Mode
> +=======
> +
> +When driver is running on the VF device, then due to hardware enforcements,
> +access to the privileged registers is not possible. To avoid relying on these
> +registers, we plan to perform early detection if we are running on the VF
> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
> +IS_SRIOV_VF(xe) to control the driver logic.
> +
> +To speed up merging of the required changes, we might first introduce dummy
> +macro that is always set to false, to prepare driver to avoid some code paths
> +before we finalize our VF mode detection and other VFs enabling changes.
> +
> +
> +Resources
> +=========
> +
> +Most of the hardware (or firmware) resources available on the Xe architecture,
> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
> +VFs and will require some provisioning steps to assign those resources for use
> +by the VF.
> +
> +Until VFs are provisioned with resources, the PF driver will be able to use all
> +resources, in the same way as it would be running in non-virtualized mode.
> +
> +If some resource (of part or region of it) is assigned to specific VF, then PF
> +is not allowed to use that part or region of the resource, but can continue to
> +use whatever is left available.
> +
> +Those resources are usually fully virtualized, so they will not require any
> +special handling when used by the VF driver, except that VF driver must know
> +the assigned quota.
> +
> +The most notable exception is the GGTT address space, as on some platforms,
> +the VF driver must additionally know the real range that it can access.
> +
> +Once the resources were assigned to the VF use and the VF driver has started,
> +then it is not allowed to change such provisioning, as that would break the
> +VF driver. To make changes the VF driver, which was using these resources,
> +must be unloaded (or the VM is terminated) and the VF device must be reset
> +using the FLR.
> +
> +
> +Scheduling
> +==========
> +
> +The workloads from PF driver and VF drivers must be submitted to the hardware
> +always by using the GuC submission mechanism. Unless VF has exclusive access
> +to the GT then submissions from different VFs are time-sliced and controlled
> +with additional "execution_quantum" and "preemption_timeout" parameters.
> +
> +In contrast to the resource provisioning, those scheduling parameters can be
> +changed even if VF drivers are already running and are active.
> +
> +
> +Automatic VFs Provisioning
> +==========================
> +
> +To provide out-of-the box experience when user will be enabling VFs using
> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
> +
> +By default, all VFs will be allocated with the fair amount of the mandatory
> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
> +Such provisioning should be sufficient for most of the normal usages, when
> +no strict SLA is required.
> +
> +The PF driver will also expose some additional sysfs files to allow adjusting
> +this automatic VFs provisioning, like default values for most of the
> +provisioning parameters that PF will then apply for each enabled VF.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +Manual VFs Provisioning
> +=======================
> +
> +If automatic VFs provisioning, which applies same configuration to every VF,
> +is not sufficient or there is a need for advanced customization of some VF,
> +the PF driver will also provide extended sysfs interface which will allow
> +control every provisioning attribute to the lowest feasible level.
> +
> +It is expected that these low-level attributes will be mostly used by the
> +advanced users or by the custom tools that will setup configurations that
> +meet predefined and validated SLA as required by the customers.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +VFs Monitoring
> +==============
> +
> +In addition to the resource provisioning or changing scheduling parameters,
> +the PF driver might also allow configure some monitoring parameters, like
> +thresholds of adverse events or sample period, to track undesired behavior
> +of the VFs that could impact the whole system.
> +
> +Once those thresholds are setup and sampling period is defined, the GuC will
> +notify the PF driver about which VF is excessing the threshold and then PF is
> +able to trigger the uevent to notify the administrator (or VMM) that could
> +take some action against the VF.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 12:37   ` Tvrtko Ursulin
  0 siblings, 0 replies; 30+ messages in thread
From: Tvrtko Ursulin @ 2023-11-14 12:37 UTC (permalink / raw)
  To: Michal Wajdeczko, intel-xe, dri-devel; +Cc: Daniel Vetter, Rodrigo Vivi


On 10/11/2023 18:22, Michal Wajdeczko wrote:
> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> Express (PCIe) specification suite is supported starting from 12th
> generation of Intel Graphics processors.
> 
> This RFC aims to explain how do we want to add support for SR-IOV
> to the new Xe driver and to propose related additions to the sysfs.
> 
> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> Cc: Oded Gabbay <ogabbay@kernel.org>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> Cc: Daniel Vetter <daniel@ffwll.ch>
> ---
>   Documentation/gpu/rfc/index.rst             |   5 +
>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>   3 files changed, 698 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> 
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index e4f7b005138d..fc5bc447f30d 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -35,3 +35,8 @@ host such documentation:
>   .. toctree::
>   
>      xe.rst
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   xe_sriov.rst
> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> new file mode 100644
> index 000000000000..77748204dd83
> --- /dev/null
> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> @@ -0,0 +1,501 @@
> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> +..
> +.. Intel Xe driver ABI (SR-IOV extensions)
> +..
> +    The Single Root I/O Virtualization (SR-IOV) extension to
> +    the PCI Express (PCIe) specification suite is supported
> +    starting from 12th generation of Intel Graphics processors.
> +
> +    This document describes Xe driver specific additions.
> +
> +    For description of generic SR-IOV sysfs attributes see
> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_auto_provisioning
> +    │   ├── admin_mode
> +    │   ├── enabled
> +    │   ├── reset_defaults
> +    │   ├── resources
> +    │   │   ├── default_contexts_quota
> +    │   │   ├── default_doorbells_quota
> +    │   │   ├── default_ggtt_quota
> +    │   │   └── default_lmem_quota
> +    │   ├── scheduling
> +    │   │   ├── default_exec_quantum_ms
> +    │   │   └── default_preempt_timeout_us
> +    │   └── monitoring
> +    │       ├── default_cat_error_count
> +    │       ├── default_doorbell_time_us
> +    │       ├── default_engine_reset_count
> +    │       ├── default_h2g_time_us
> +    │       ├── default_irq_time_us
> +    │       └── default_page_fault_count

 From the department of bike-shedding, one alternative could be to have 
a directory called defaults which avoids having to have the default_ 
prefix on everything under it.

> +
> +    /sys/bus/pci/drivers/xe/BDF/
> +    ├── sriov_extensions

Should this be xe_sriov_extensions or if not doesn't it need agreement 
to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
Sriov_auto_provisioning too I guess.

> +    │   ├── monitoring_period_ms
> +    │   ├── strict_scheduling_enabled
> +    │   ├── pf
> +    │   │   ├── device -> ../../../BDF
> +    │   │   ├── priority
> +    │   │   ├── tile0
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   ├── vf1
> +    │   │   ├── device -> ../../../BDF+1
> +    │   │   ├── stop
> +    │   │   ├── tile0
> +    │   │   │   ├── ggtt_quota
> +    │   │   │   ├── lmem_quota
> +    │   │   │   ├── gt0
> +    │   │   │   │   ├── contexts_quota
> +    │   │   │   │   ├── doorbells_quota
> +    │   │   │   │   ├── exec_quantum_ms
> +    │   │   │   │   ├── preempt_timeout_us
> +    │   │   │   │   └── thresholds
> +    │   │   │   │       ├── cat_error_count
> +    │   │   │   │       ├── doorbell_time_us
> +    │   │   │   │       ├── engine_reset_count
> +    │   │   │   │       ├── h2g_time_us
> +    │   │   │   │       ├── irq_time_us
> +    │   │   │   │       └── page_fault_count
> +    │   │   │   └── gtX
> +    │   │   └── tileT
> +    │   └── vfN
> +..
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on the device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - xe driver supports SR-IOV PF on given device, and
> +		 - xe driver supports automatic VFs provisioning.
> +
> +		This directory is used as a root for all attributes related to
> +		automatic provisioning of SR-IOV Physical Function (PF) and/or
> +		Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		(un)provisioning that could be performed by the PF.
> +
> +		The default value is 1 (true).
> +
> +		This flag can be set to false, unless manual provisioning is not
> +		applicable for given platform or it is not supported by current
> +		PF implementation. In such cases -EPERM will be returned.
> +
> +		This flag will be automatically set to false when there will be
> +		other attempts to change any of VF's resource provisioning.
> +		See "sriov_extensions" section for details.
> +
> +		This flag can be set back to true if and only if all VFs are
> +		fully unprovisioned, otherwise -EEXIST error will be returned.
> +
> +		false = "disabled"
> +			When disabled, then PF will not attempt to do automatic
> +			VFs provisioning when VFs are being enabled and will not
> +			perform automatic unprovisioning of the VFs when VFs will
> +			be disabled.
> +
> +		true = "enabled"
> +			When enabled, then on VFs enabling PF will do automatic
> +			VFs provisioning based on the default settings described
> +			below.
> +
> +			If automatic VFs provisioning fails due to some reasons,
> +			then VFs will not be enabled.
> +
> +			If enabled, all resources allocated during VFs enabling
> +			will be released during VFs disabling (automatic unprovisioning).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool (0, 1)
> +
> +		This file represents configuration flag for the automatic VFs
> +		provisioning that could be performed by the PF.
> +
> +		The default value depends on the platform type.
> +
> +		This flag can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		If enabled (default on discrete platforms) then the PF will
> +		retain only minimum hardcoded resources for its own use when
> +		doing VFs automatic provisioning and will not use any default
> +		values described below for its own configuration.
> +
> +		If disabled (default on integrated platforms) then the PF will
> +		treat itself like yet another additional VF in all fair resource
> +		allocations and will also try to apply default provisioning
> +		values described below for its own configuration.
> +

One alternative could be to expose two sets of defaults, the PF and VF 
ones. With the advantage of allowing the "admin mode" / "minimal PF" to 
be explicitly configurable instead of hardcoded. Should be more flexible.

If the discrete vs integrated distinction is wanted it could simply be 
made by initialy populating (driver init) the respective defaults based 
on the platform type.

> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Writing to this file will reset all default provisioning parameters
> +		listed below to the default values.

Maybe this isn't required if you can say it is the responsibility of 
whoever changes the defaults to either know what they are doing, or to 
save and restore themselves if they. It is not a major concern but if 
writing kernel code can be avoided perhaps it can be considered.

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent default provisioning that should be used
> +		for VFs automatic provisioning.
> +
> +		These values can be changed any time, but will have no effect if
> +		VFs are already provisioned.
> +
> +		default_contexts_quota: (RW) integer 0..U32_MAX
> +			The number of GuC context IDs to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> +
> +		default_doorbells_quota: (RW) integer 0..U32_MAX
> +			The number of GuC doorbells to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> +
> +		default_ggtt_quota: (RW) integer 0..U32_MAX
> +			The size of the GGTT address space (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
> +
> +		default_lmem_quota: (RW) integer 0..U32_MAX
> +			The size of the LMEM (in bytes) to provide to the VF.
> +			The default value is 0 (use fair allocations).
> +			See "sriov_extensions/vfN/tileT/lmem_quota" for details.
> +
> +		default_exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum (in millisecs) assigned to the function.
> +			The default value is 0 (infinify).
> +			See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> +
> +		default_preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout (in microsecs) assigned to the function.
> +			The default value is 0 (infinity).
> +			See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.

I have a slight concern here on the usability of GuC specific tunables.

Whereas one can imagine an external entity (some admin, somewhere) to 
probably pretty much understand what it means to partition the local 
memory, address space, and set the scheduling timeouts (all are intuitve 
and obvious concepts), how are they suppose to approach the GuC 
doorbells and contexts?

It could be a matter of adding more documentation for those two, or it 
even could make sense to shove them under a guc prefix (or 
subdirectory?) to signify the fact they are implementation details and 
not a fundamental concept.

> +
> +		default_cat_error_count: (RW) integer 0..U32_MAX
> +		default_doorbell_time_us: (RW) integer 0..U32_MAX
> +		default_engine_reset_count: (RW) integer 0..U32_MAX
> +		default_h2g_time_us: (RW) integer 0..U32_MAX
> +		default_irq_time_us: (RW) integer 0..U32_MAX
> +		default_page_fault_count: (RW) integer 0..U32_MAX
> +			The monitoring threshold to be set for the function.
> +			The default value is 0 (don't monitor).
> +			See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory appears on Xe device when:
> +
> +		 - device supports SR-IOV, and
> +		 - device is a Physical Function (PF), and
> +		 - driver is enabled to support SR-IOV PF on given device.
> +
> +		This directory is used as a root for all attributes required to
> +		manage both Physical Function (PF) and Virtual Functions (VFs).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) bool
> +
> +		This file represents a flag used to determine if scheduling
> +		parameters should be respected even if there is no active
> +		workloads submitted by the PF or VFs.
> +
> +		This flag is disabled by default, unless strict scheduling is
> +		not applicable on given platform. In such case this file will
> +		be read-only.
> +
> +		The change to this file may have no effect if VFs are not yet enabled.
> +		If strict scheduling can't be enabled in GuC then write will fail with -EIO.

I think the semantics of this need to be documented ie. how it interacts 
with exec_quantum_ms. If it does? I am guessing that it has to otherwise 
I don't know what it would mean - presumably unused timeslices are not 
given to other entities but time just goes wasted? But it is also a 
question on over what time interval. Or that too is purely defined by 
the number of PF+VFs and their respective allocated quanta.

Also, would there be benefit, assuming it is possible with GuC, to allow 
configuring it per PF/VF?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) integer
> +
> +		This file represents the configuration knob used by adverse event
> +		monitoring. A value here is the period in millisecs during which
> +		events are counted and the total is checked against a threshold.
> +		See "sriov_extensions/vfN/tileT/gtX/thresholds" for more details.
> +
> +		Default is 0 (monitoring is disabled).
> +
> +		If monitoring capability is not available, then attempt to enable
> +		will fail with -EPERM error. If monitoring can't be enabled in
> +		GuC then write will fail with -EIO.

Could the docs explain if there is a downside to enabling it, which is 
probably why it isn't enabled by default? Because it does sound natural 
that adverse events should be noticed.

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Physical Function (PF).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all attributes related to the SR-IOV
> +		Virtual Function (VF).
> +
> +		Note that VF numbers (N) are 1-based as described in PCI SR-IOV specification.
> +		The Xe driver implementaton follows that naming schema.
> +
> +		There will be "vf1", "vf2" up to "vfN" directories, where N matches
> +		value of the PCI "sriov_totalvfs" attribute.
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device tile.
> +		The tile numbers (T) start from 0.
> +
> +		There is at least one "tile0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		This directory holds all SR-IOV attributes related to the device GT.
> +		The GT numbers (X) start from 0.
> +
> +		There is at least one "gt0/" directory present.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(symbolic link)
> +
> +		Backlink to the PCI device entry representing given function.
> +		For PF this link is always present.
> +		For VF this link is present only for currently enabled VFs.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(RW) string
> +
> +		This file represents a GuC Scheduler knob to override the default
> +		round-robin or FIFO scheduler policies implemented by the GuC.
> +
> +		The default value is "peer".
> +
> +		This flag can be changed, unless such change is not applicable
> +		for given platform or is not supported by current GuC firmware.
> +		In such case this file could be read-only or will return -EPERM
> +		on write attempt.
> +
> +		"immediate"
> +			GuC will Schedule PF workloads immediately and PF
> +			workloads only until the PF's work queues in GuC
> +			are empty.
> +
> +		"lazy"
> +			GuC will Schedule PF workloads at the next opportune
> +			moment and PF workloads only until the PF work queues
> +			in GuC are empty.
> +
> +		"peer"
> +			GuC Scheduler will treat PF and VFs with equal priority.

Hmmm this is too very GuC specific and I wonder what is the usecase for 
lazy? Lazy = "don't care when it runs, but when it runs it will run 
everything queued so far", right? Feels a bit odd on first.

"Immediate" may also not be depending on preemption granularity and 
workloads, right?

Are there any ideas to express the knobs in a more generic fashion?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		(WO) bool (1)
> +
> +		Write to this file will force GuC to stop handle any requests from
> +		this VF, but without triggering a FLR.
> +		To recover, the full FLR must be issued using generic "device/reset".
> +
> +		This file allows to implement custom policy mechanism when VF is
> +		misbehaving and triggering adverse events above defined thresholds.
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent scheduling parameters of the functions.
> +
> +		These scheduling parameters can be changed even if VFs are enabled
> +		and running, unless such change is not applicable on given platform
> +		due to fixed hardware or firmware assignment.
> +
> +		exec_quantum_ms: (RW) integer 0..U32_MAX
> +			The GT execution quantum in [ms] assigned to the function.
> +			Requested quantum might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).
> +
> +		preempt_timeout_us: (RW) integer 0..U32_MAX
> +			The GT preemption timeout in [us] assigned to the function.
> +			Requested timeout might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unlimited).

Alignment for the above two will be visible after read-back?

> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent shared resource assigned to the functions.
> +
> +		These resource parameters can be changed, unless VF is already running,
> +		or such change is not applicable on given platform due to fixed hardware
> +		or firmware assignment.
> +
> +		Writes to these attributes may fail with:
> +			-EPERM if change is not applicable on give HW/FW.
> +			-E2BIG if value larger that HW/FW limit.
> +			-EDQUOT if value is larger than maximum quota defined by the PF.
> +			-ENOSPC if PF can't allocate required quota.
> +			-EBUSY if the resource is currently in use by the VF.
> +			-EIO if GuC refuses to change provisioning.

Why it would refuse if input is valid? In other words, what is the 
user/admin supposed to do on -EIO?

> +
> +		ggtt_quota: (RW) integer 0..U64_MAX
> +			The size of the GGTT address space (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		lmem_quota: (RW) integer 0..U64_MAX
> +			The size of the Local Memory (in bytes) assigned to the VF.
> +			The value might be aligned per HW/FW requirements.
> +
> +			This attribute is only available on discrete platforms.
> +
> +			Default is 0 (unprovisioned).
> +
> +		contexts_quota: (RW) 0..U16_MAX
> +			The number of GuC submission contexts assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +		doorbells_quota: (RW) 0..U16_MAX
> +			The number of GuC doorbells assigned to the VF.
> +			This value might be aligned per HW/FW requirements.
> +
> +			Default is 0 (unprovisioned).
> +
> +
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
> +What:		/sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
> +Date:		2024
> +KernelVersion:	TBD
> +Contact:	intel-xe@lists.freedesktop.org
> +Description:
> +		These files represent threshold values used by the GuC to trigger
> +		security events if adverse event monitoring is enabled.

How are the security events delivered? There is mention of uevents in a 
later paragraph - are they already defined or should be together with 
this so the link can be place here?

> +
> +		These thresholds are checked every "monitoring_period_ms".
> +		Refer to GuC ABI for details about each threshold category.

Is it possible to have a link here to GuC ABI?

Regards,

Tvrtko

> +
> +		Default value for all thresholds is 0 (disabled).
> +
> +		cat_error_count: (RW) integer
> +		doorbell_time_us: (RW) integer
> +		engine_reset_count: (RW) integer
> +		h2g_time_us: (RW) integer
> +		irq_time_us: (RW) integer
> +		page_fault_count: (RW) integer
> diff --git a/Documentation/gpu/rfc/xe_sriov.rst b/Documentation/gpu/rfc/xe_sriov.rst
> new file mode 100644
> index 000000000000..574f6414eabb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/xe_sriov.rst
> @@ -0,0 +1,192 @@
> +.. SPDX-License-Identifier: MIT
> +
> +========================
> +Xe – SR-IOV Support Plan
> +========================
> +
> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI Express (PCIe)
> +specification suite is supported starting from 12th generation of Intel Graphics
> +processors.
> +
> +This document describes planned ABI of the new Xe driver (see xe.rst) that will
> +provide flexible configuration and management options related to the SR-IOV.
> +It will also highlight few most important changes to the Xe driver
> +implementation to deal with Intel GPU SR-IOV specific requirements.
> +
> +
> +SR-IOV Capability
> +=================
> +
> +Due to SR-IOV complexity and required co-operation between hardware, firmware
> +and kernel drivers, not all Xe architecture platforms might have SR-IOV enabled
> +or fully functional.
> +
> +To control at the driver level which platform will provide support for SR-IOV,
> +as we can't just rely on the PCI configuration data exposed by the hardware,
> +we will introduce "has_sriov" flag to the struct xe_device_desc that describes
> +a device capabilities that driver checks during the probe.
> +
> +Initially this flag will be set to disabled even on platforms that we plan to
> +support. We will enable this flag only once we finish merging all required
> +changes to the driver and related validated firmwares are also made available.
> +
> +
> +SR-IOV Platforms
> +================
> +
> +Initially we plan to add SR-IOV functionality to the following SDV platforms
> +already supported by the Xe driver:
> +
> + - TGL (up to 7 VFs)
> + - ADL (up to 7 VFs)
> + - MTL (up to 7 VFs)
> + - ATSM (up to 31 VFs)
> + - PVC (up to 63 VFs)
> +
> +Newer platforms will be supported later, but we hope that enabling will be
> +much faster, as majority of the driver changes are either platform agnostic
> +or are similar between earlier platforms (hence we start with SDVs).
> +
> +
> +PF Mode
> +=======
> +
> +Support in the driver for acting in Physical Function (PF) mode, i.e. mode
> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and will be
> +enabled by default.
> +
> +However, due to potentially conflicting requirements for SR-IOV and other mega
> +features, we might want to have an option to disable SR-IOV PF mode support at
> +the driver load time.
> +
> +Thus, we plan to use additional modparam named "sriov_totalvfs" which if set to
> +0 will force the driver to operate in the native (non-virtualized) mode.
> +The same modparam could be used to limit number of supported Virtual Functions
> +(VFs) by the driver compared to the hardware limit exposed in PCI configuration.
> +
> +The name of this modparam corresponds to the existing PCI sysfs attribute, that
> +by default exposes hardware capability.
> +
> +The default value of this param will allow to support all possible VFs as
> +claimed by the hardware.
> +
> +This modparam will have no effect if driver is running on the VF device.
> +
> +
> +VFs Enabling
> +============
> +
> +To enable or disable VFs we plan to rely on existing sysfs attribute exposed by
> +the PCI subsystem named "sriov_numvfs". We will provide all necessary tweaks to
> +provision VFs in our custom implementation of the "sriov_configure" hook from
> +the struct pci_driver.
> +
> +If for some reason, including explicit request to disable SR-IOV PF mode using
> +modparam, we will not be able to correctly support any VFs, driver will change
> +number of supported VFs, exposed to the userspace by "sriov_totalvfs" attribute,
> +to 0, thus preventing configuration of the VFs.
> +
> +
> +VF Mode
> +=======
> +
> +When driver is running on the VF device, then due to hardware enforcements,
> +access to the privileged registers is not possible. To avoid relying on these
> +registers, we plan to perform early detection if we are running on the VF
> +device using dedicated VF_CAP(0x1901f8) register and then use global macro
> +IS_SRIOV_VF(xe) to control the driver logic.
> +
> +To speed up merging of the required changes, we might first introduce dummy
> +macro that is always set to false, to prepare driver to avoid some code paths
> +before we finalize our VF mode detection and other VFs enabling changes.
> +
> +
> +Resources
> +=========
> +
> +Most of the hardware (or firmware) resources available on the Xe architecture,
> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared between PF and
> +VFs and will require some provisioning steps to assign those resources for use
> +by the VF.
> +
> +Until VFs are provisioned with resources, the PF driver will be able to use all
> +resources, in the same way as it would be running in non-virtualized mode.
> +
> +If some resource (of part or region of it) is assigned to specific VF, then PF
> +is not allowed to use that part or region of the resource, but can continue to
> +use whatever is left available.
> +
> +Those resources are usually fully virtualized, so they will not require any
> +special handling when used by the VF driver, except that VF driver must know
> +the assigned quota.
> +
> +The most notable exception is the GGTT address space, as on some platforms,
> +the VF driver must additionally know the real range that it can access.
> +
> +Once the resources were assigned to the VF use and the VF driver has started,
> +then it is not allowed to change such provisioning, as that would break the
> +VF driver. To make changes the VF driver, which was using these resources,
> +must be unloaded (or the VM is terminated) and the VF device must be reset
> +using the FLR.
> +
> +
> +Scheduling
> +==========
> +
> +The workloads from PF driver and VF drivers must be submitted to the hardware
> +always by using the GuC submission mechanism. Unless VF has exclusive access
> +to the GT then submissions from different VFs are time-sliced and controlled
> +with additional "execution_quantum" and "preemption_timeout" parameters.
> +
> +In contrast to the resource provisioning, those scheduling parameters can be
> +changed even if VF drivers are already running and are active.
> +
> +
> +Automatic VFs Provisioning
> +==========================
> +
> +To provide out-of-the box experience when user will be enabling VFs using
> +generic "sriov_numvfs" attribute without requiring complex provisioning steps,
> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
> +
> +By default, all VFs will be allocated with the fair amount of the mandatory
> +resources (like GGTT, GuC IDs) and with unrestricted scheduling parameters.
> +Such provisioning should be sufficient for most of the normal usages, when
> +no strict SLA is required.
> +
> +The PF driver will also expose some additional sysfs files to allow adjusting
> +this automatic VFs provisioning, like default values for most of the
> +provisioning parameters that PF will then apply for each enabled VF.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +Manual VFs Provisioning
> +=======================
> +
> +If automatic VFs provisioning, which applies same configuration to every VF,
> +is not sufficient or there is a need for advanced customization of some VF,
> +the PF driver will also provide extended sysfs interface which will allow
> +control every provisioning attribute to the lowest feasible level.
> +
> +It is expected that these low-level attributes will be mostly used by the
> +advanced users or by the custom tools that will setup configurations that
> +meet predefined and validated SLA as required by the customers.
> +
> +    Details about those extension can be found in
> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> +
> +
> +VFs Monitoring
> +==============
> +
> +In addition to the resource provisioning or changing scheduling parameters,
> +the PF driver might also allow configure some monitoring parameters, like
> +thresholds of adverse events or sample period, to track undesired behavior
> +of the VFs that could impact the whole system.
> +
> +Once those thresholds are setup and sampling period is defined, the GuC will
> +notify the PF driver about which VF is excessing the threshold and then PF is
> +able to trigger the uevent to notify the administrator (or VMM) that could
> +take some action against the VF.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-14 12:37   ` [Intel-xe] " Tvrtko Ursulin
@ 2023-11-14 13:22     ` Vivi, Rodrigo
  -1 siblings, 0 replies; 30+ messages in thread
From: Vivi, Rodrigo @ 2023-11-14 13:22 UTC (permalink / raw)
  To: intel-xe, dri-devel, tvrtko.ursulin, Wajdeczko, Michal; +Cc: ogabbay

On Tue, 2023-11-14 at 12:37 +0000, Tvrtko Ursulin wrote:
> 
> On 10/11/2023 18:22, Michal Wajdeczko wrote:
> > The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> > Express (PCIe) specification suite is supported starting from 12th
> > generation of Intel Graphics processors.
> > 
> > This RFC aims to explain how do we want to add support for SR-IOV
> > to the new Xe driver and to propose related additions to the sysfs.
> > 
> > Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> > Cc: Oded Gabbay <ogabbay@kernel.org>
> > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> > Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> > Cc: Daniel Vetter <daniel@ffwll.ch>
> > ---
> >   Documentation/gpu/rfc/index.rst             |   5 +
> >   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501
> > ++++++++++++++++++++
> >   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
> >   3 files changed, 698 insertions(+)
> >   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
> >   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> > 
> > diff --git a/Documentation/gpu/rfc/index.rst
> > b/Documentation/gpu/rfc/index.rst
> > index e4f7b005138d..fc5bc447f30d 100644
> > --- a/Documentation/gpu/rfc/index.rst
> > +++ b/Documentation/gpu/rfc/index.rst
> > @@ -35,3 +35,8 @@ host such documentation:
> >   .. toctree::
> >   
> >      xe.rst
> > +
> > +.. toctree::
> > +   :maxdepth: 1
> > +
> > +   xe_sriov.rst
> > diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > new file mode 100644
> > index 000000000000..77748204dd83
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > @@ -0,0 +1,501 @@
> > +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> > +..
> > +.. Intel Xe driver ABI (SR-IOV extensions)
> > +..
> > +    The Single Root I/O Virtualization (SR-IOV) extension to
> > +    the PCI Express (PCIe) specification suite is supported
> > +    starting from 12th generation of Intel Graphics processors.
> > +
> > +    This document describes Xe driver specific additions.
> > +
> > +    For description of generic SR-IOV sysfs attributes see
> > +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> > +
> > +    /sys/bus/pci/drivers/xe/BDF/
> > +    ├── sriov_auto_provisioning
> > +    │   ├── admin_mode
> > +    │   ├── enabled
> > +    │   ├── reset_defaults
> > +    │   ├── resources
> > +    │   │   ├── default_contexts_quota
> > +    │   │   ├── default_doorbells_quota
> > +    │   │   ├── default_ggtt_quota
> > +    │   │   └── default_lmem_quota
> > +    │   ├── scheduling
> > +    │   │   ├── default_exec_quantum_ms
> > +    │   │   └── default_preempt_timeout_us
> > +    │   └── monitoring
> > +    │       ├── default_cat_error_count
> > +    │       ├── default_doorbell_time_us
> > +    │       ├── default_engine_reset_count
> > +    │       ├── default_h2g_time_us
> > +    │       ├── default_irq_time_us
> > +    │       └── default_page_fault_count
> 
>  From the department of bike-shedding, one alternative could be to
> have 
> a directory called defaults which avoids having to have the default_ 
> prefix on everything under it.

good idea. probably with a '.' prefix to make it hidden like we have
for other stuff already.

'.defaults'

> 
> > +
> > +    /sys/bus/pci/drivers/xe/BDF/
> > +    ├── sriov_extensions
> 
> Should this be xe_sriov_extensions or if not doesn't it need
> agreement 
> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
> Sriov_auto_provisioning too I guess.
> 
> > +    │   ├── monitoring_period_ms
> > +    │   ├── strict_scheduling_enabled
> > +    │   ├── pf
> > +    │   │   ├── device -> ../../../BDF
> > +    │   │   ├── priority
> > +    │   │   ├── tile0
> > +    │   │   │   ├── gt0
> > +    │   │   │   │   ├── exec_quantum_ms
> > +    │   │   │   │   ├── preempt_timeout_us
> > +    │   │   │   │   └── thresholds
> > +    │   │   │   │       ├── cat_error_count
> > +    │   │   │   │       ├── doorbell_time_us
> > +    │   │   │   │       ├── engine_reset_count
> > +    │   │   │   │       ├── h2g_time_us
> > +    │   │   │   │       ├── irq_time_us
> > +    │   │   │   │       └── page_fault_count
> > +    │   │   │   └── gtX
> > +    │   │   └── tileT
> > +    │   ├── vf1
> > +    │   │   ├── device -> ../../../BDF+1
> > +    │   │   ├── stop
> > +    │   │   ├── tile0
> > +    │   │   │   ├── ggtt_quota
> > +    │   │   │   ├── lmem_quota
> > +    │   │   │   ├── gt0
> > +    │   │   │   │   ├── contexts_quota
> > +    │   │   │   │   ├── doorbells_quota
> > +    │   │   │   │   ├── exec_quantum_ms
> > +    │   │   │   │   ├── preempt_timeout_us
> > +    │   │   │   │   └── thresholds
> > +    │   │   │   │       ├── cat_error_count
> > +    │   │   │   │       ├── doorbell_time_us
> > +    │   │   │   │       ├── engine_reset_count
> > +    │   │   │   │       ├── h2g_time_us
> > +    │   │   │   │       ├── irq_time_us
> > +    │   │   │   │       └── page_fault_count
> > +    │   │   │   └── gtX
> > +    │   │   └── tileT
> > +    │   └── vfN
> > +..
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory appears on the device when:
> > +
> > +                - device supports SR-IOV, and
> > +                - device is a Physical Function (PF), and
> > +                - xe driver supports SR-IOV PF on given device,
> > and
> > +                - xe driver supports automatic VFs provisioning.
> > +
> > +               This directory is used as a root for all attributes
> > related to
> > +               automatic provisioning of SR-IOV Physical Function
> > (PF) and/or
> > +               Virtual Functions (VFs).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /enabled
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool (0, 1)
> > +
> > +               This file represents configuration flag for the
> > automatic VFs
> > +               (un)provisioning that could be performed by the PF.
> > +
> > +               The default value is 1 (true).
> > +
> > +               This flag can be set to false, unless manual
> > provisioning is not
> > +               applicable for given platform or it is not
> > supported by current
> > +               PF implementation. In such cases -EPERM will be
> > returned.
> > +
> > +               This flag will be automatically set to false when
> > there will be
> > +               other attempts to change any of VF's resource
> > provisioning.
> > +               See "sriov_extensions" section for details.
> > +
> > +               This flag can be set back to true if and only if
> > all VFs are
> > +               fully unprovisioned, otherwise -EEXIST error will
> > be returned.
> > +
> > +               false = "disabled"
> > +                       When disabled, then PF will not attempt to
> > do automatic
> > +                       VFs provisioning when VFs are being enabled
> > and will not
> > +                       perform automatic unprovisioning of the VFs
> > when VFs will
> > +                       be disabled.
> > +
> > +               true = "enabled"
> > +                       When enabled, then on VFs enabling PF will
> > do automatic
> > +                       VFs provisioning based on the default
> > settings described
> > +                       below.
> > +
> > +                       If automatic VFs provisioning fails due to
> > some reasons,
> > +                       then VFs will not be enabled.
> > +
> > +                       If enabled, all resources allocated during
> > VFs enabling
> > +                       will be released during VFs disabling
> > (automatic unprovisioning).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /admin_mode
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool (0, 1)
> > +
> > +               This file represents configuration flag for the
> > automatic VFs
> > +               provisioning that could be performed by the PF.
> > +
> > +               The default value depends on the platform type.
> > +
> > +               This flag can be changed any time, but will have no
> > effect if
> > +               VFs are already provisioned.
> > +
> > +               If enabled (default on discrete platforms) then the
> > PF will
> > +               retain only minimum hardcoded resources for its own
> > use when
> > +               doing VFs automatic provisioning and will not use
> > any default
> > +               values described below for its own configuration.
> > +
> > +               If disabled (default on integrated platforms) then
> > the PF will
> > +               treat itself like yet another additional VF in all
> > fair resource
> > +               allocations and will also try to apply default
> > provisioning
> > +               values described below for its own configuration.
> > +
> 
> One alternative could be to expose two sets of defaults, the PF and
> VF 
> ones. With the advantage of allowing the "admin mode" / "minimal PF"
> to 
> be explicitly configurable instead of hardcoded. Should be more
> flexible.
> 
> If the discrete vs integrated distinction is wanted it could simply
> be 
> made by initialy populating (driver init) the respective defaults
> based 
> on the platform type.
> 
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /reset_defaults
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (WO) bool (1)
> > +
> > +               Writing to this file will reset all default
> > provisioning parameters
> > +               listed below to the default values.
> 
> Maybe this isn't required if you can say it is the responsibility of 
> whoever changes the defaults to either know what they are doing, or
> to 
> save and restore themselves if they. It is not a major concern but if
> writing kernel code can be avoided perhaps it can be considered.
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_contexts_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_doorbells_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_ggtt_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_lmem_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /scheduling/default_exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /scheduling/default_preempt_timeout_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_page_fault_count
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent default provisioning that
> > should be used
> > +               for VFs automatic provisioning.
> > +
> > +               These values can be changed any time, but will have
> > no effect if
> > +               VFs are already provisioned.
> > +
> > +               default_contexts_quota: (RW) integer 0..U32_MAX
> > +                       The number of GuC context IDs to provide to
> > the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> > +
> > +               default_doorbells_quota: (RW) integer 0..U32_MAX
> > +                       The number of GuC doorbells to provide to
> > the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> > +
> > +               default_ggtt_quota: (RW) integer 0..U32_MAX
> > +                       The size of the GGTT address space (in
> > bytes) to provide to the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See "sriov_extensions/vfN/tileT/ggtt_quota"
> > for details.
> > +
> > +               default_lmem_quota: (RW) integer 0..U32_MAX
> > +                       The size of the LMEM (in bytes) to provide
> > to the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See "sriov_extensions/vfN/tileT/lmem_quota"
> > for details.
> > +
> > +               default_exec_quantum_ms: (RW) integer 0..U32_MAX
> > +                       The GT execution quantum (in millisecs)
> > assigned to the function.
> > +                       The default value is 0 (infinify).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> > +
> > +               default_preempt_timeout_us: (RW) integer 0..U32_MAX
> > +                       The GT preemption timeout (in microsecs)
> > assigned to the function.
> > +                       The default value is 0 (infinity).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
> 
> I have a slight concern here on the usability of GuC specific
> tunables.
> 
> Whereas one can imagine an external entity (some admin, somewhere) to
> probably pretty much understand what it means to partition the local 
> memory, address space, and set the scheduling timeouts (all are
> intuitve 
> and obvious concepts), how are they suppose to approach the GuC 
> doorbells and contexts?
> 
> It could be a matter of adding more documentation for those two, or
> it 
> even could make sense to shove them under a guc prefix (or 
> subdirectory?) to signify the fact they are implementation details
> and 
> not a fundamental concept.
> 
> > +
> > +               default_cat_error_count: (RW) integer 0..U32_MAX
> > +               default_doorbell_time_us: (RW) integer 0..U32_MAX
> > +               default_engine_reset_count: (RW) integer 0..U32_MAX
> > +               default_h2g_time_us: (RW) integer 0..U32_MAX
> > +               default_irq_time_us: (RW) integer 0..U32_MAX
> > +               default_page_fault_count: (RW) integer 0..U32_MAX
> > +                       The monitoring threshold to be set for the
> > function.
> > +                       The default value is 0 (don't monitor).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory appears on Xe device when:
> > +
> > +                - device supports SR-IOV, and
> > +                - device is a Physical Function (PF), and
> > +                - driver is enabled to support SR-IOV PF on given
> > device.
> > +
> > +               This directory is used as a root for all attributes
> > required to
> > +               manage both Physical Function (PF) and Virtual
> > Functions (VFs).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/strict
> > _scheduling_enabled
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool
> > +
> > +               This file represents a flag used to determine if
> > scheduling
> > +               parameters should be respected even if there is no
> > active
> > +               workloads submitted by the PF or VFs.
> > +
> > +               This flag is disabled by default, unless strict
> > scheduling is
> > +               not applicable on given platform. In such case this
> > file will
> > +               be read-only.
> > +
> > +               The change to this file may have no effect if VFs
> > are not yet enabled.
> > +               If strict scheduling can't be enabled in GuC then
> > write will fail with -EIO.
> 
> I think the semantics of this need to be documented ie. how it
> interacts 
> with exec_quantum_ms. If it does? I am guessing that it has to
> otherwise 
> I don't know what it would mean - presumably unused timeslices are
> not 
> given to other entities but time just goes wasted? But it is also a 
> question on over what time interval. Or that too is purely defined by
> the number of PF+VFs and their respective allocated quanta.
> 
> Also, would there be benefit, assuming it is possible with GuC, to
> allow 
> configuring it per PF/VF?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/monito
> > ring_period_ms
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) integer
> > +
> > +               This file represents the configuration knob used by
> > adverse event
> > +               monitoring. A value here is the period in millisecs
> > during which
> > +               events are counted and the total is checked against
> > a threshold.
> > +               See "sriov_extensions/vfN/tileT/gtX/thresholds" for
> > more details.
> > +
> > +               Default is 0 (monitoring is disabled).
> > +
> > +               If monitoring capability is not available, then
> > attempt to enable
> > +               will fail with -EPERM error. If monitoring can't be
> > enabled in
> > +               GuC then write will fail with -EIO.
> 
> Could the docs explain if there is a downside to enabling it, which
> is 
> probably why it isn't enabled by default? Because it does sound
> natural 
> that adverse events should be noticed.
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all attributes related to the
> > SR-IOV
> > +               Physical Function (PF).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all attributes related to the
> > SR-IOV
> > +               Virtual Function (VF).
> > +
> > +               Note that VF numbers (N) are 1-based as described
> > in PCI SR-IOV specification.
> > +               The Xe driver implementaton follows that naming
> > schema.
> > +
> > +               There will be "vf1", "vf2" up to "vfN" directories,
> > where N matches
> > +               value of the PCI "sriov_totalvfs" attribute.
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all SR-IOV attributes related
> > to the device tile.
> > +               The tile numbers (T) start from 0.
> > +
> > +               There is at least one "tile0/" directory present.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all SR-IOV attributes related
> > to the device GT.
> > +               The GT numbers (X) start from 0.
> > +
> > +               There is at least one "gt0/" directory present.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/dev
> > ice
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/de
> > vice
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (symbolic link)
> > +
> > +               Backlink to the PCI device entry representing given
> > function.
> > +               For PF this link is always present.
> > +               For VF this link is present only for currently
> > enabled VFs.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/pri
> > ority
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) string
> > +
> > +               This file represents a GuC Scheduler knob to
> > override the default
> > +               round-robin or FIFO scheduler policies implemented
> > by the GuC.
> > +
> > +               The default value is "peer".
> > +
> > +               This flag can be changed, unless such change is not
> > applicable
> > +               for given platform or is not supported by current
> > GuC firmware.
> > +               In such case this file could be read-only or will
> > return -EPERM
> > +               on write attempt.
> > +
> > +               "immediate"
> > +                       GuC will Schedule PF workloads immediately
> > and PF
> > +                       workloads only until the PF's work queues
> > in GuC
> > +                       are empty.
> > +
> > +               "lazy"
> > +                       GuC will Schedule PF workloads at the next
> > opportune
> > +                       moment and PF workloads only until the PF
> > work queues
> > +                       in GuC are empty.
> > +
> > +               "peer"
> > +                       GuC Scheduler will treat PF and VFs with
> > equal priority.
> 
> Hmmm this is too very GuC specific and I wonder what is the usecase
> for 
> lazy? Lazy = "don't care when it runs, but when it runs it will run 
> everything queued so far", right? Feels a bit odd on first.
> 
> "Immediate" may also not be depending on preemption granularity and 
> workloads, right?
> 
> Are there any ideas to express the knobs in a more generic fashion?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/st
> > op
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (WO) bool (1)
> > +
> > +               Write to this file will force GuC to stop handle
> > any requests from
> > +               this VF, but without triggering a FLR.
> > +               To recover, the full FLR must be issued using
> > generic "device/reset".
> > +
> > +               This file allows to implement custom policy
> > mechanism when VF is
> > +               misbehaving and triggering adverse events above
> > defined thresholds.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/preempt_timeout_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/preempt_timeout_us
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent scheduling parameters of the
> > functions.
> > +
> > +               These scheduling parameters can be changed even if
> > VFs are enabled
> > +               and running, unless such change is not applicable
> > on given platform
> > +               due to fixed hardware or firmware assignment.
> > +
> > +               exec_quantum_ms: (RW) integer 0..U32_MAX
> > +                       The GT execution quantum in [ms] assigned
> > to the function.
> > +                       Requested quantum might be aligned per
> > HW/FW requirements.
> > +
> > +                       Default is 0 (unlimited).
> > +
> > +               preempt_timeout_us: (RW) integer 0..U32_MAX
> > +                       The GT preemption timeout in [us] assigned
> > to the function.
> > +                       Requested timeout might be aligned per
> > HW/FW requirements.
> > +
> > +                       Default is 0 (unlimited).
> 
> Alignment for the above two will be visible after read-back?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/ggtt_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/lmem_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/contexts_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/doorbells_quota
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent shared resource assigned to
> > the functions.
> > +
> > +               These resource parameters can be changed, unless VF
> > is already running,
> > +               or such change is not applicable on given platform
> > due to fixed hardware
> > +               or firmware assignment.
> > +
> > +               Writes to these attributes may fail with:
> > +                       -EPERM if change is not applicable on give
> > HW/FW.
> > +                       -E2BIG if value larger that HW/FW limit.
> > +                       -EDQUOT if value is larger than maximum
> > quota defined by the PF.
> > +                       -ENOSPC if PF can't allocate required
> > quota.
> > +                       -EBUSY if the resource is currently in use
> > by the VF.
> > +                       -EIO if GuC refuses to change provisioning.
> 
> Why it would refuse if input is valid? In other words, what is the 
> user/admin supposed to do on -EIO?
> 
> > +
> > +               ggtt_quota: (RW) integer 0..U64_MAX
> > +                       The size of the GGTT address space (in
> > bytes) assigned to the VF.
> > +                       The value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               lmem_quota: (RW) integer 0..U64_MAX
> > +                       The size of the Local Memory (in bytes)
> > assigned to the VF.
> > +                       The value might be aligned per HW/FW
> > requirements.
> > +
> > +                       This attribute is only available on
> > discrete platforms.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               contexts_quota: (RW) 0..U16_MAX
> > +                       The number of GuC submission contexts
> > assigned to the VF.
> > +                       This value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               doorbells_quota: (RW) 0..U16_MAX
> > +                       The number of GuC doorbells assigned to the
> > VF.
> > +                       This value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/page_fault_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/page_fault_count
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent threshold values used by the
> > GuC to trigger
> > +               security events if adverse event monitoring is
> > enabled.
> 
> How are the security events delivered? There is mention of uevents in
> a 
> later paragraph - are they already defined or should be together with
> this so the link can be place here?
> 
> > +
> > +               These thresholds are checked every
> > "monitoring_period_ms".
> > +               Refer to GuC ABI for details about each threshold
> > category.
> 
> Is it possible to have a link here to GuC ABI?
> 
> Regards,
> 
> Tvrtko
> 
> > +
> > +               Default value for all thresholds is 0 (disabled).
> > +
> > +               cat_error_count: (RW) integer
> > +               doorbell_time_us: (RW) integer
> > +               engine_reset_count: (RW) integer
> > +               h2g_time_us: (RW) integer
> > +               irq_time_us: (RW) integer
> > +               page_fault_count: (RW) integer
> > diff --git a/Documentation/gpu/rfc/xe_sriov.rst
> > b/Documentation/gpu/rfc/xe_sriov.rst
> > new file mode 100644
> > index 000000000000..574f6414eabb
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/xe_sriov.rst
> > @@ -0,0 +1,192 @@
> > +.. SPDX-License-Identifier: MIT
> > +
> > +========================
> > +Xe – SR-IOV Support Plan
> > +========================
> > +
> > +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> > Express (PCIe)
> > +specification suite is supported starting from 12th generation of
> > Intel Graphics
> > +processors.
> > +
> > +This document describes planned ABI of the new Xe driver (see
> > xe.rst) that will
> > +provide flexible configuration and management options related to
> > the SR-IOV.
> > +It will also highlight few most important changes to the Xe driver
> > +implementation to deal with Intel GPU SR-IOV specific
> > requirements.
> > +
> > +
> > +SR-IOV Capability
> > +=================
> > +
> > +Due to SR-IOV complexity and required co-operation between
> > hardware, firmware
> > +and kernel drivers, not all Xe architecture platforms might have
> > SR-IOV enabled
> > +or fully functional.
> > +
> > +To control at the driver level which platform will provide support
> > for SR-IOV,
> > +as we can't just rely on the PCI configuration data exposed by the
> > hardware,
> > +we will introduce "has_sriov" flag to the struct xe_device_desc
> > that describes
> > +a device capabilities that driver checks during the probe.
> > +
> > +Initially this flag will be set to disabled even on platforms that
> > we plan to
> > +support. We will enable this flag only once we finish merging all
> > required
> > +changes to the driver and related validated firmwares are also
> > made available.
> > +
> > +
> > +SR-IOV Platforms
> > +================
> > +
> > +Initially we plan to add SR-IOV functionality to the following SDV
> > platforms
> > +already supported by the Xe driver:
> > +
> > + - TGL (up to 7 VFs)
> > + - ADL (up to 7 VFs)
> > + - MTL (up to 7 VFs)
> > + - ATSM (up to 31 VFs)
> > + - PVC (up to 63 VFs)
> > +
> > +Newer platforms will be supported later, but we hope that enabling
> > will be
> > +much faster, as majority of the driver changes are either platform
> > agnostic
> > +or are similar between earlier platforms (hence we start with
> > SDVs).
> > +
> > +
> > +PF Mode
> > +=======
> > +
> > +Support in the driver for acting in Physical Function (PF) mode,
> > i.e. mode
> > +that allows configuration of VFs, depends on the CONFIG_PCI_IOV
> > and will be
> > +enabled by default.
> > +
> > +However, due to potentially conflicting requirements for SR-IOV
> > and other mega
> > +features, we might want to have an option to disable SR-IOV PF
> > mode support at
> > +the driver load time.
> > +
> > +Thus, we plan to use additional modparam named "sriov_totalvfs"
> > which if set to
> > +0 will force the driver to operate in the native (non-virtualized)
> > mode.
> > +The same modparam could be used to limit number of supported
> > Virtual Functions
> > +(VFs) by the driver compared to the hardware limit exposed in PCI
> > configuration.
> > +
> > +The name of this modparam corresponds to the existing PCI sysfs
> > attribute, that
> > +by default exposes hardware capability.
> > +
> > +The default value of this param will allow to support all possible
> > VFs as
> > +claimed by the hardware.
> > +
> > +This modparam will have no effect if driver is running on the VF
> > device.
> > +
> > +
> > +VFs Enabling
> > +============
> > +
> > +To enable or disable VFs we plan to rely on existing sysfs
> > attribute exposed by
> > +the PCI subsystem named "sriov_numvfs". We will provide all
> > necessary tweaks to
> > +provision VFs in our custom implementation of the
> > "sriov_configure" hook from
> > +the struct pci_driver.
> > +
> > +If for some reason, including explicit request to disable SR-IOV
> > PF mode using
> > +modparam, we will not be able to correctly support any VFs, driver
> > will change
> > +number of supported VFs, exposed to the userspace by
> > "sriov_totalvfs" attribute,
> > +to 0, thus preventing configuration of the VFs.
> > +
> > +
> > +VF Mode
> > +=======
> > +
> > +When driver is running on the VF device, then due to hardware
> > enforcements,
> > +access to the privileged registers is not possible. To avoid
> > relying on these
> > +registers, we plan to perform early detection if we are running on
> > the VF
> > +device using dedicated VF_CAP(0x1901f8) register and then use
> > global macro
> > +IS_SRIOV_VF(xe) to control the driver logic.
> > +
> > +To speed up merging of the required changes, we might first
> > introduce dummy
> > +macro that is always set to false, to prepare driver to avoid some
> > code paths
> > +before we finalize our VF mode detection and other VFs enabling
> > changes.
> > +
> > +
> > +Resources
> > +=========
> > +
> > +Most of the hardware (or firmware) resources available on the Xe
> > architecture,
> > +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
> > between PF and
> > +VFs and will require some provisioning steps to assign those
> > resources for use
> > +by the VF.
> > +
> > +Until VFs are provisioned with resources, the PF driver will be
> > able to use all
> > +resources, in the same way as it would be running in non-
> > virtualized mode.
> > +
> > +If some resource (of part or region of it) is assigned to specific
> > VF, then PF
> > +is not allowed to use that part or region of the resource, but can
> > continue to
> > +use whatever is left available.
> > +
> > +Those resources are usually fully virtualized, so they will not
> > require any
> > +special handling when used by the VF driver, except that VF driver
> > must know
> > +the assigned quota.
> > +
> > +The most notable exception is the GGTT address space, as on some
> > platforms,
> > +the VF driver must additionally know the real range that it can
> > access.
> > +
> > +Once the resources were assigned to the VF use and the VF driver
> > has started,
> > +then it is not allowed to change such provisioning, as that would
> > break the
> > +VF driver. To make changes the VF driver, which was using these
> > resources,
> > +must be unloaded (or the VM is terminated) and the VF device must
> > be reset
> > +using the FLR.
> > +
> > +
> > +Scheduling
> > +==========
> > +
> > +The workloads from PF driver and VF drivers must be submitted to
> > the hardware
> > +always by using the GuC submission mechanism. Unless VF has
> > exclusive access
> > +to the GT then submissions from different VFs are time-sliced and
> > controlled
> > +with additional "execution_quantum" and "preemption_timeout"
> > parameters.
> > +
> > +In contrast to the resource provisioning, those scheduling
> > parameters can be
> > +changed even if VF drivers are already running and are active.
> > +
> > +
> > +Automatic VFs Provisioning
> > +==========================
> > +
> > +To provide out-of-the box experience when user will be enabling
> > VFs using
> > +generic "sriov_numvfs" attribute without requiring complex
> > provisioning steps,
> > +the SR-IOV PF driver will implement automatic VFs resource
> > provisioning.
> > +
> > +By default, all VFs will be allocated with the fair amount of the
> > mandatory
> > +resources (like GGTT, GuC IDs) and with unrestricted scheduling
> > parameters.
> > +Such provisioning should be sufficient for most of the normal
> > usages, when
> > +no strict SLA is required.
> > +
> > +The PF driver will also expose some additional sysfs files to
> > allow adjusting
> > +this automatic VFs provisioning, like default values for most of
> > the
> > +provisioning parameters that PF will then apply for each enabled
> > VF.
> > +
> > +    Details about those extension can be found in
> > +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> > +
> > +
> > +Manual VFs Provisioning
> > +=======================
> > +
> > +If automatic VFs provisioning, which applies same configuration to
> > every VF,
> > +is not sufficient or there is a need for advanced customization of
> > some VF,
> > +the PF driver will also provide extended sysfs interface which
> > will allow
> > +control every provisioning attribute to the lowest feasible level.
> > +
> > +It is expected that these low-level attributes will be mostly used
> > by the
> > +advanced users or by the custom tools that will setup
> > configurations that
> > +meet predefined and validated SLA as required by the customers.
> > +
> > +    Details about those extension can be found in
> > +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> > +
> > +
> > +VFs Monitoring
> > +==============
> > +
> > +In addition to the resource provisioning or changing scheduling
> > parameters,
> > +the PF driver might also allow configure some monitoring
> > parameters, like
> > +thresholds of adverse events or sample period, to track undesired
> > behavior
> > +of the VFs that could impact the whole system.
> > +
> > +Once those thresholds are setup and sampling period is defined,
> > the GuC will
> > +notify the PF driver about which VF is excessing the threshold and
> > then PF is
> > +able to trigger the uevent to notify the administrator (or VMM)
> > that could
> > +take some action against the VF.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 13:22     ` Vivi, Rodrigo
  0 siblings, 0 replies; 30+ messages in thread
From: Vivi, Rodrigo @ 2023-11-14 13:22 UTC (permalink / raw)
  To: intel-xe, dri-devel, tvrtko.ursulin, Wajdeczko, Michal; +Cc: daniel

On Tue, 2023-11-14 at 12:37 +0000, Tvrtko Ursulin wrote:
> 
> On 10/11/2023 18:22, Michal Wajdeczko wrote:
> > The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> > Express (PCIe) specification suite is supported starting from 12th
> > generation of Intel Graphics processors.
> > 
> > This RFC aims to explain how do we want to add support for SR-IOV
> > to the new Xe driver and to propose related additions to the sysfs.
> > 
> > Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
> > Cc: Oded Gabbay <ogabbay@kernel.org>
> > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> > Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> > Cc: Daniel Vetter <daniel@ffwll.ch>
> > ---
> >   Documentation/gpu/rfc/index.rst             |   5 +
> >   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501
> > ++++++++++++++++++++
> >   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
> >   3 files changed, 698 insertions(+)
> >   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
> >   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
> > 
> > diff --git a/Documentation/gpu/rfc/index.rst
> > b/Documentation/gpu/rfc/index.rst
> > index e4f7b005138d..fc5bc447f30d 100644
> > --- a/Documentation/gpu/rfc/index.rst
> > +++ b/Documentation/gpu/rfc/index.rst
> > @@ -35,3 +35,8 @@ host such documentation:
> >   .. toctree::
> >   
> >      xe.rst
> > +
> > +.. toctree::
> > +   :maxdepth: 1
> > +
> > +   xe_sriov.rst
> > diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > new file mode 100644
> > index 000000000000..77748204dd83
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
> > @@ -0,0 +1,501 @@
> > +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
> > +..
> > +.. Intel Xe driver ABI (SR-IOV extensions)
> > +..
> > +    The Single Root I/O Virtualization (SR-IOV) extension to
> > +    the PCI Express (PCIe) specification suite is supported
> > +    starting from 12th generation of Intel Graphics processors.
> > +
> > +    This document describes Xe driver specific additions.
> > +
> > +    For description of generic SR-IOV sysfs attributes see
> > +    "Documentation/ABI/testing/sysfs-bus-pci" document.
> > +
> > +    /sys/bus/pci/drivers/xe/BDF/
> > +    ├── sriov_auto_provisioning
> > +    │   ├── admin_mode
> > +    │   ├── enabled
> > +    │   ├── reset_defaults
> > +    │   ├── resources
> > +    │   │   ├── default_contexts_quota
> > +    │   │   ├── default_doorbells_quota
> > +    │   │   ├── default_ggtt_quota
> > +    │   │   └── default_lmem_quota
> > +    │   ├── scheduling
> > +    │   │   ├── default_exec_quantum_ms
> > +    │   │   └── default_preempt_timeout_us
> > +    │   └── monitoring
> > +    │       ├── default_cat_error_count
> > +    │       ├── default_doorbell_time_us
> > +    │       ├── default_engine_reset_count
> > +    │       ├── default_h2g_time_us
> > +    │       ├── default_irq_time_us
> > +    │       └── default_page_fault_count
> 
>  From the department of bike-shedding, one alternative could be to
> have 
> a directory called defaults which avoids having to have the default_ 
> prefix on everything under it.

good idea. probably with a '.' prefix to make it hidden like we have
for other stuff already.

'.defaults'

> 
> > +
> > +    /sys/bus/pci/drivers/xe/BDF/
> > +    ├── sriov_extensions
> 
> Should this be xe_sriov_extensions or if not doesn't it need
> agreement 
> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
> Sriov_auto_provisioning too I guess.
> 
> > +    │   ├── monitoring_period_ms
> > +    │   ├── strict_scheduling_enabled
> > +    │   ├── pf
> > +    │   │   ├── device -> ../../../BDF
> > +    │   │   ├── priority
> > +    │   │   ├── tile0
> > +    │   │   │   ├── gt0
> > +    │   │   │   │   ├── exec_quantum_ms
> > +    │   │   │   │   ├── preempt_timeout_us
> > +    │   │   │   │   └── thresholds
> > +    │   │   │   │       ├── cat_error_count
> > +    │   │   │   │       ├── doorbell_time_us
> > +    │   │   │   │       ├── engine_reset_count
> > +    │   │   │   │       ├── h2g_time_us
> > +    │   │   │   │       ├── irq_time_us
> > +    │   │   │   │       └── page_fault_count
> > +    │   │   │   └── gtX
> > +    │   │   └── tileT
> > +    │   ├── vf1
> > +    │   │   ├── device -> ../../../BDF+1
> > +    │   │   ├── stop
> > +    │   │   ├── tile0
> > +    │   │   │   ├── ggtt_quota
> > +    │   │   │   ├── lmem_quota
> > +    │   │   │   ├── gt0
> > +    │   │   │   │   ├── contexts_quota
> > +    │   │   │   │   ├── doorbells_quota
> > +    │   │   │   │   ├── exec_quantum_ms
> > +    │   │   │   │   ├── preempt_timeout_us
> > +    │   │   │   │   └── thresholds
> > +    │   │   │   │       ├── cat_error_count
> > +    │   │   │   │       ├── doorbell_time_us
> > +    │   │   │   │       ├── engine_reset_count
> > +    │   │   │   │       ├── h2g_time_us
> > +    │   │   │   │       ├── irq_time_us
> > +    │   │   │   │       └── page_fault_count
> > +    │   │   │   └── gtX
> > +    │   │   └── tileT
> > +    │   └── vfN
> > +..
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory appears on the device when:
> > +
> > +                - device supports SR-IOV, and
> > +                - device is a Physical Function (PF), and
> > +                - xe driver supports SR-IOV PF on given device,
> > and
> > +                - xe driver supports automatic VFs provisioning.
> > +
> > +               This directory is used as a root for all attributes
> > related to
> > +               automatic provisioning of SR-IOV Physical Function
> > (PF) and/or
> > +               Virtual Functions (VFs).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /enabled
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool (0, 1)
> > +
> > +               This file represents configuration flag for the
> > automatic VFs
> > +               (un)provisioning that could be performed by the PF.
> > +
> > +               The default value is 1 (true).
> > +
> > +               This flag can be set to false, unless manual
> > provisioning is not
> > +               applicable for given platform or it is not
> > supported by current
> > +               PF implementation. In such cases -EPERM will be
> > returned.
> > +
> > +               This flag will be automatically set to false when
> > there will be
> > +               other attempts to change any of VF's resource
> > provisioning.
> > +               See "sriov_extensions" section for details.
> > +
> > +               This flag can be set back to true if and only if
> > all VFs are
> > +               fully unprovisioned, otherwise -EEXIST error will
> > be returned.
> > +
> > +               false = "disabled"
> > +                       When disabled, then PF will not attempt to
> > do automatic
> > +                       VFs provisioning when VFs are being enabled
> > and will not
> > +                       perform automatic unprovisioning of the VFs
> > when VFs will
> > +                       be disabled.
> > +
> > +               true = "enabled"
> > +                       When enabled, then on VFs enabling PF will
> > do automatic
> > +                       VFs provisioning based on the default
> > settings described
> > +                       below.
> > +
> > +                       If automatic VFs provisioning fails due to
> > some reasons,
> > +                       then VFs will not be enabled.
> > +
> > +                       If enabled, all resources allocated during
> > VFs enabling
> > +                       will be released during VFs disabling
> > (automatic unprovisioning).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /admin_mode
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool (0, 1)
> > +
> > +               This file represents configuration flag for the
> > automatic VFs
> > +               provisioning that could be performed by the PF.
> > +
> > +               The default value depends on the platform type.
> > +
> > +               This flag can be changed any time, but will have no
> > effect if
> > +               VFs are already provisioned.
> > +
> > +               If enabled (default on discrete platforms) then the
> > PF will
> > +               retain only minimum hardcoded resources for its own
> > use when
> > +               doing VFs automatic provisioning and will not use
> > any default
> > +               values described below for its own configuration.
> > +
> > +               If disabled (default on integrated platforms) then
> > the PF will
> > +               treat itself like yet another additional VF in all
> > fair resource
> > +               allocations and will also try to apply default
> > provisioning
> > +               values described below for its own configuration.
> > +
> 
> One alternative could be to expose two sets of defaults, the PF and
> VF 
> ones. With the advantage of allowing the "admin mode" / "minimal PF"
> to 
> be explicitly configurable instead of hardcoded. Should be more
> flexible.
> 
> If the discrete vs integrated distinction is wanted it could simply
> be 
> made by initialy populating (driver init) the respective defaults
> based 
> on the platform type.
> 
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /reset_defaults
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (WO) bool (1)
> > +
> > +               Writing to this file will reset all default
> > provisioning parameters
> > +               listed below to the default values.
> 
> Maybe this isn't required if you can say it is the responsibility of 
> whoever changes the defaults to either know what they are doing, or
> to 
> save and restore themselves if they. It is not a major concern but if
> writing kernel code can be avoided perhaps it can be considered.
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_contexts_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_doorbells_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_ggtt_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /resources/default_lmem_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /scheduling/default_exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /scheduling/default_preempt_timeout_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
> > /monitoring/default_page_fault_count
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent default provisioning that
> > should be used
> > +               for VFs automatic provisioning.
> > +
> > +               These values can be changed any time, but will have
> > no effect if
> > +               VFs are already provisioned.
> > +
> > +               default_contexts_quota: (RW) integer 0..U32_MAX
> > +                       The number of GuC context IDs to provide to
> > the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
> > +
> > +               default_doorbells_quota: (RW) integer 0..U32_MAX
> > +                       The number of GuC doorbells to provide to
> > the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
> > +
> > +               default_ggtt_quota: (RW) integer 0..U32_MAX
> > +                       The size of the GGTT address space (in
> > bytes) to provide to the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See "sriov_extensions/vfN/tileT/ggtt_quota"
> > for details.
> > +
> > +               default_lmem_quota: (RW) integer 0..U32_MAX
> > +                       The size of the LMEM (in bytes) to provide
> > to the VF.
> > +                       The default value is 0 (use fair
> > allocations).
> > +                       See "sriov_extensions/vfN/tileT/lmem_quota"
> > for details.
> > +
> > +               default_exec_quantum_ms: (RW) integer 0..U32_MAX
> > +                       The GT execution quantum (in millisecs)
> > assigned to the function.
> > +                       The default value is 0 (infinify).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
> > +
> > +               default_preempt_timeout_us: (RW) integer 0..U32_MAX
> > +                       The GT preemption timeout (in microsecs)
> > assigned to the function.
> > +                       The default value is 0 (infinity).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
> 
> I have a slight concern here on the usability of GuC specific
> tunables.
> 
> Whereas one can imagine an external entity (some admin, somewhere) to
> probably pretty much understand what it means to partition the local 
> memory, address space, and set the scheduling timeouts (all are
> intuitve 
> and obvious concepts), how are they suppose to approach the GuC 
> doorbells and contexts?
> 
> It could be a matter of adding more documentation for those two, or
> it 
> even could make sense to shove them under a guc prefix (or 
> subdirectory?) to signify the fact they are implementation details
> and 
> not a fundamental concept.
> 
> > +
> > +               default_cat_error_count: (RW) integer 0..U32_MAX
> > +               default_doorbell_time_us: (RW) integer 0..U32_MAX
> > +               default_engine_reset_count: (RW) integer 0..U32_MAX
> > +               default_h2g_time_us: (RW) integer 0..U32_MAX
> > +               default_irq_time_us: (RW) integer 0..U32_MAX
> > +               default_page_fault_count: (RW) integer 0..U32_MAX
> > +                       The monitoring threshold to be set for the
> > function.
> > +                       The default value is 0 (don't monitor).
> > +                       See
> > "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory appears on Xe device when:
> > +
> > +                - device supports SR-IOV, and
> > +                - device is a Physical Function (PF), and
> > +                - driver is enabled to support SR-IOV PF on given
> > device.
> > +
> > +               This directory is used as a root for all attributes
> > required to
> > +               manage both Physical Function (PF) and Virtual
> > Functions (VFs).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/strict
> > _scheduling_enabled
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) bool
> > +
> > +               This file represents a flag used to determine if
> > scheduling
> > +               parameters should be respected even if there is no
> > active
> > +               workloads submitted by the PF or VFs.
> > +
> > +               This flag is disabled by default, unless strict
> > scheduling is
> > +               not applicable on given platform. In such case this
> > file will
> > +               be read-only.
> > +
> > +               The change to this file may have no effect if VFs
> > are not yet enabled.
> > +               If strict scheduling can't be enabled in GuC then
> > write will fail with -EIO.
> 
> I think the semantics of this need to be documented ie. how it
> interacts 
> with exec_quantum_ms. If it does? I am guessing that it has to
> otherwise 
> I don't know what it would mean - presumably unused timeslices are
> not 
> given to other entities but time just goes wasted? But it is also a 
> question on over what time interval. Or that too is purely defined by
> the number of PF+VFs and their respective allocated quanta.
> 
> Also, would there be benefit, assuming it is possible with GuC, to
> allow 
> configuring it per PF/VF?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/monito
> > ring_period_ms
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) integer
> > +
> > +               This file represents the configuration knob used by
> > adverse event
> > +               monitoring. A value here is the period in millisecs
> > during which
> > +               events are counted and the total is checked against
> > a threshold.
> > +               See "sriov_extensions/vfN/tileT/gtX/thresholds" for
> > more details.
> > +
> > +               Default is 0 (monitoring is disabled).
> > +
> > +               If monitoring capability is not available, then
> > attempt to enable
> > +               will fail with -EPERM error. If monitoring can't be
> > enabled in
> > +               GuC then write will fail with -EIO.
> 
> Could the docs explain if there is a downside to enabling it, which
> is 
> probably why it isn't enabled by default? Because it does sound
> natural 
> that adverse events should be noticed.
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all attributes related to the
> > SR-IOV
> > +               Physical Function (PF).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all attributes related to the
> > SR-IOV
> > +               Virtual Function (VF).
> > +
> > +               Note that VF numbers (N) are 1-based as described
> > in PCI SR-IOV specification.
> > +               The Xe driver implementaton follows that naming
> > schema.
> > +
> > +               There will be "vf1", "vf2" up to "vfN" directories,
> > where N matches
> > +               value of the PCI "sriov_totalvfs" attribute.
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all SR-IOV attributes related
> > to the device tile.
> > +               The tile numbers (T) start from 0.
> > +
> > +               There is at least one "tile0/" directory present.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               This directory holds all SR-IOV attributes related
> > to the device GT.
> > +               The GT numbers (X) start from 0.
> > +
> > +               There is at least one "gt0/" directory present.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/dev
> > ice
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/de
> > vice
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (symbolic link)
> > +
> > +               Backlink to the PCI device entry representing given
> > function.
> > +               For PF this link is always present.
> > +               For VF this link is present only for currently
> > enabled VFs.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/pri
> > ority
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (RW) string
> > +
> > +               This file represents a GuC Scheduler knob to
> > override the default
> > +               round-robin or FIFO scheduler policies implemented
> > by the GuC.
> > +
> > +               The default value is "peer".
> > +
> > +               This flag can be changed, unless such change is not
> > applicable
> > +               for given platform or is not supported by current
> > GuC firmware.
> > +               In such case this file could be read-only or will
> > return -EPERM
> > +               on write attempt.
> > +
> > +               "immediate"
> > +                       GuC will Schedule PF workloads immediately
> > and PF
> > +                       workloads only until the PF's work queues
> > in GuC
> > +                       are empty.
> > +
> > +               "lazy"
> > +                       GuC will Schedule PF workloads at the next
> > opportune
> > +                       moment and PF workloads only until the PF
> > work queues
> > +                       in GuC are empty.
> > +
> > +               "peer"
> > +                       GuC Scheduler will treat PF and VFs with
> > equal priority.
> 
> Hmmm this is too very GuC specific and I wonder what is the usecase
> for 
> lazy? Lazy = "don't care when it runs, but when it runs it will run 
> everything queued so far", right? Feels a bit odd on first.
> 
> "Immediate" may also not be depending on preemption granularity and 
> workloads, right?
> 
> Are there any ideas to express the knobs in a more generic fashion?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/st
> > op
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               (WO) bool (1)
> > +
> > +               Write to this file will force GuC to stop handle
> > any requests from
> > +               this VF, but without triggering a FLR.
> > +               To recover, the full FLR must be issued using
> > generic "device/reset".
> > +
> > +               This file allows to implement custom policy
> > mechanism when VF is
> > +               misbehaving and triggering adverse events above
> > defined thresholds.
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/preempt_timeout_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/exec_quantum_ms
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/preempt_timeout_us
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent scheduling parameters of the
> > functions.
> > +
> > +               These scheduling parameters can be changed even if
> > VFs are enabled
> > +               and running, unless such change is not applicable
> > on given platform
> > +               due to fixed hardware or firmware assignment.
> > +
> > +               exec_quantum_ms: (RW) integer 0..U32_MAX
> > +                       The GT execution quantum in [ms] assigned
> > to the function.
> > +                       Requested quantum might be aligned per
> > HW/FW requirements.
> > +
> > +                       Default is 0 (unlimited).
> > +
> > +               preempt_timeout_us: (RW) integer 0..U32_MAX
> > +                       The GT preemption timeout in [us] assigned
> > to the function.
> > +                       Requested timeout might be aligned per
> > HW/FW requirements.
> > +
> > +                       Default is 0 (unlimited).
> 
> Alignment for the above two will be visible after read-back?
> 
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/ggtt_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/lmem_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/contexts_quota
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/doorbells_quota
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent shared resource assigned to
> > the functions.
> > +
> > +               These resource parameters can be changed, unless VF
> > is already running,
> > +               or such change is not applicable on given platform
> > due to fixed hardware
> > +               or firmware assignment.
> > +
> > +               Writes to these attributes may fail with:
> > +                       -EPERM if change is not applicable on give
> > HW/FW.
> > +                       -E2BIG if value larger that HW/FW limit.
> > +                       -EDQUOT if value is larger than maximum
> > quota defined by the PF.
> > +                       -ENOSPC if PF can't allocate required
> > quota.
> > +                       -EBUSY if the resource is currently in use
> > by the VF.
> > +                       -EIO if GuC refuses to change provisioning.
> 
> Why it would refuse if input is valid? In other words, what is the 
> user/admin supposed to do on -EIO?
> 
> > +
> > +               ggtt_quota: (RW) integer 0..U64_MAX
> > +                       The size of the GGTT address space (in
> > bytes) assigned to the VF.
> > +                       The value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               lmem_quota: (RW) integer 0..U64_MAX
> > +                       The size of the Local Memory (in bytes)
> > assigned to the VF.
> > +                       The value might be aligned per HW/FW
> > requirements.
> > +
> > +                       This attribute is only available on
> > discrete platforms.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               contexts_quota: (RW) 0..U16_MAX
> > +                       The number of GuC submission contexts
> > assigned to the VF.
> > +                       This value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +               doorbells_quota: (RW) 0..U16_MAX
> > +                       The number of GuC doorbells assigned to the
> > VF.
> > +                       This value might be aligned per HW/FW
> > requirements.
> > +
> > +                       Default is 0 (unprovisioned).
> > +
> > +
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
> > eT/gtX/thresholds/page_fault_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/cat_error_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/doorbell_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/engine_reset_count
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/h2g_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/irq_time_us
> > +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
> > leT/gtX/thresholds/page_fault_count
> > +Date:          2024
> > +KernelVersion: TBD
> > +Contact:       intel-xe@lists.freedesktop.org
> > +Description:
> > +               These files represent threshold values used by the
> > GuC to trigger
> > +               security events if adverse event monitoring is
> > enabled.
> 
> How are the security events delivered? There is mention of uevents in
> a 
> later paragraph - are they already defined or should be together with
> this so the link can be place here?
> 
> > +
> > +               These thresholds are checked every
> > "monitoring_period_ms".
> > +               Refer to GuC ABI for details about each threshold
> > category.
> 
> Is it possible to have a link here to GuC ABI?
> 
> Regards,
> 
> Tvrtko
> 
> > +
> > +               Default value for all thresholds is 0 (disabled).
> > +
> > +               cat_error_count: (RW) integer
> > +               doorbell_time_us: (RW) integer
> > +               engine_reset_count: (RW) integer
> > +               h2g_time_us: (RW) integer
> > +               irq_time_us: (RW) integer
> > +               page_fault_count: (RW) integer
> > diff --git a/Documentation/gpu/rfc/xe_sriov.rst
> > b/Documentation/gpu/rfc/xe_sriov.rst
> > new file mode 100644
> > index 000000000000..574f6414eabb
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/xe_sriov.rst
> > @@ -0,0 +1,192 @@
> > +.. SPDX-License-Identifier: MIT
> > +
> > +========================
> > +Xe – SR-IOV Support Plan
> > +========================
> > +
> > +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
> > Express (PCIe)
> > +specification suite is supported starting from 12th generation of
> > Intel Graphics
> > +processors.
> > +
> > +This document describes planned ABI of the new Xe driver (see
> > xe.rst) that will
> > +provide flexible configuration and management options related to
> > the SR-IOV.
> > +It will also highlight few most important changes to the Xe driver
> > +implementation to deal with Intel GPU SR-IOV specific
> > requirements.
> > +
> > +
> > +SR-IOV Capability
> > +=================
> > +
> > +Due to SR-IOV complexity and required co-operation between
> > hardware, firmware
> > +and kernel drivers, not all Xe architecture platforms might have
> > SR-IOV enabled
> > +or fully functional.
> > +
> > +To control at the driver level which platform will provide support
> > for SR-IOV,
> > +as we can't just rely on the PCI configuration data exposed by the
> > hardware,
> > +we will introduce "has_sriov" flag to the struct xe_device_desc
> > that describes
> > +a device capabilities that driver checks during the probe.
> > +
> > +Initially this flag will be set to disabled even on platforms that
> > we plan to
> > +support. We will enable this flag only once we finish merging all
> > required
> > +changes to the driver and related validated firmwares are also
> > made available.
> > +
> > +
> > +SR-IOV Platforms
> > +================
> > +
> > +Initially we plan to add SR-IOV functionality to the following SDV
> > platforms
> > +already supported by the Xe driver:
> > +
> > + - TGL (up to 7 VFs)
> > + - ADL (up to 7 VFs)
> > + - MTL (up to 7 VFs)
> > + - ATSM (up to 31 VFs)
> > + - PVC (up to 63 VFs)
> > +
> > +Newer platforms will be supported later, but we hope that enabling
> > will be
> > +much faster, as majority of the driver changes are either platform
> > agnostic
> > +or are similar between earlier platforms (hence we start with
> > SDVs).
> > +
> > +
> > +PF Mode
> > +=======
> > +
> > +Support in the driver for acting in Physical Function (PF) mode,
> > i.e. mode
> > +that allows configuration of VFs, depends on the CONFIG_PCI_IOV
> > and will be
> > +enabled by default.
> > +
> > +However, due to potentially conflicting requirements for SR-IOV
> > and other mega
> > +features, we might want to have an option to disable SR-IOV PF
> > mode support at
> > +the driver load time.
> > +
> > +Thus, we plan to use additional modparam named "sriov_totalvfs"
> > which if set to
> > +0 will force the driver to operate in the native (non-virtualized)
> > mode.
> > +The same modparam could be used to limit number of supported
> > Virtual Functions
> > +(VFs) by the driver compared to the hardware limit exposed in PCI
> > configuration.
> > +
> > +The name of this modparam corresponds to the existing PCI sysfs
> > attribute, that
> > +by default exposes hardware capability.
> > +
> > +The default value of this param will allow to support all possible
> > VFs as
> > +claimed by the hardware.
> > +
> > +This modparam will have no effect if driver is running on the VF
> > device.
> > +
> > +
> > +VFs Enabling
> > +============
> > +
> > +To enable or disable VFs we plan to rely on existing sysfs
> > attribute exposed by
> > +the PCI subsystem named "sriov_numvfs". We will provide all
> > necessary tweaks to
> > +provision VFs in our custom implementation of the
> > "sriov_configure" hook from
> > +the struct pci_driver.
> > +
> > +If for some reason, including explicit request to disable SR-IOV
> > PF mode using
> > +modparam, we will not be able to correctly support any VFs, driver
> > will change
> > +number of supported VFs, exposed to the userspace by
> > "sriov_totalvfs" attribute,
> > +to 0, thus preventing configuration of the VFs.
> > +
> > +
> > +VF Mode
> > +=======
> > +
> > +When driver is running on the VF device, then due to hardware
> > enforcements,
> > +access to the privileged registers is not possible. To avoid
> > relying on these
> > +registers, we plan to perform early detection if we are running on
> > the VF
> > +device using dedicated VF_CAP(0x1901f8) register and then use
> > global macro
> > +IS_SRIOV_VF(xe) to control the driver logic.
> > +
> > +To speed up merging of the required changes, we might first
> > introduce dummy
> > +macro that is always set to false, to prepare driver to avoid some
> > code paths
> > +before we finalize our VF mode detection and other VFs enabling
> > changes.
> > +
> > +
> > +Resources
> > +=========
> > +
> > +Most of the hardware (or firmware) resources available on the Xe
> > architecture,
> > +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
> > between PF and
> > +VFs and will require some provisioning steps to assign those
> > resources for use
> > +by the VF.
> > +
> > +Until VFs are provisioned with resources, the PF driver will be
> > able to use all
> > +resources, in the same way as it would be running in non-
> > virtualized mode.
> > +
> > +If some resource (of part or region of it) is assigned to specific
> > VF, then PF
> > +is not allowed to use that part or region of the resource, but can
> > continue to
> > +use whatever is left available.
> > +
> > +Those resources are usually fully virtualized, so they will not
> > require any
> > +special handling when used by the VF driver, except that VF driver
> > must know
> > +the assigned quota.
> > +
> > +The most notable exception is the GGTT address space, as on some
> > platforms,
> > +the VF driver must additionally know the real range that it can
> > access.
> > +
> > +Once the resources were assigned to the VF use and the VF driver
> > has started,
> > +then it is not allowed to change such provisioning, as that would
> > break the
> > +VF driver. To make changes the VF driver, which was using these
> > resources,
> > +must be unloaded (or the VM is terminated) and the VF device must
> > be reset
> > +using the FLR.
> > +
> > +
> > +Scheduling
> > +==========
> > +
> > +The workloads from PF driver and VF drivers must be submitted to
> > the hardware
> > +always by using the GuC submission mechanism. Unless VF has
> > exclusive access
> > +to the GT then submissions from different VFs are time-sliced and
> > controlled
> > +with additional "execution_quantum" and "preemption_timeout"
> > parameters.
> > +
> > +In contrast to the resource provisioning, those scheduling
> > parameters can be
> > +changed even if VF drivers are already running and are active.
> > +
> > +
> > +Automatic VFs Provisioning
> > +==========================
> > +
> > +To provide out-of-the box experience when user will be enabling
> > VFs using
> > +generic "sriov_numvfs" attribute without requiring complex
> > provisioning steps,
> > +the SR-IOV PF driver will implement automatic VFs resource
> > provisioning.
> > +
> > +By default, all VFs will be allocated with the fair amount of the
> > mandatory
> > +resources (like GGTT, GuC IDs) and with unrestricted scheduling
> > parameters.
> > +Such provisioning should be sufficient for most of the normal
> > usages, when
> > +no strict SLA is required.
> > +
> > +The PF driver will also expose some additional sysfs files to
> > allow adjusting
> > +this automatic VFs provisioning, like default values for most of
> > the
> > +provisioning parameters that PF will then apply for each enabled
> > VF.
> > +
> > +    Details about those extension can be found in
> > +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> > +
> > +
> > +Manual VFs Provisioning
> > +=======================
> > +
> > +If automatic VFs provisioning, which applies same configuration to
> > every VF,
> > +is not sufficient or there is a need for advanced customization of
> > some VF,
> > +the PF driver will also provide extended sysfs interface which
> > will allow
> > +control every provisioning attribute to the lowest feasible level.
> > +
> > +It is expected that these low-level attributes will be mostly used
> > by the
> > +advanced users or by the custom tools that will setup
> > configurations that
> > +meet predefined and validated SLA as required by the customers.
> > +
> > +    Details about those extension can be found in
> > +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
> > +
> > +
> > +VFs Monitoring
> > +==============
> > +
> > +In addition to the resource provisioning or changing scheduling
> > parameters,
> > +the PF driver might also allow configure some monitoring
> > parameters, like
> > +thresholds of adverse events or sample period, to track undesired
> > behavior
> > +of the VFs that could impact the whole system.
> > +
> > +Once those thresholds are setup and sampling period is defined,
> > the GuC will
> > +notify the PF driver about which VF is excessing the threshold and
> > then PF is
> > +able to trigger the uevent to notify the administrator (or VMM)
> > that could
> > +take some action against the VF.


^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-14 12:37   ` [Intel-xe] " Tvrtko Ursulin
@ 2023-11-14 16:55     ` Michal Wajdeczko
  -1 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 16:55 UTC (permalink / raw)
  To: Tvrtko Ursulin, intel-xe, dri-devel; +Cc: Oded Gabbay, Rodrigo Vivi



On 14.11.2023 13:37, Tvrtko Ursulin wrote:
> 
> On 10/11/2023 18:22, Michal Wajdeczko wrote:
>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe) specification suite is supported starting from 12th
>> generation of Intel Graphics processors.
>>
>> This RFC aims to explain how do we want to add support for SR-IOV
>> to the new Xe driver and to propose related additions to the sysfs.
>>
>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>> Cc: Oded Gabbay <ogabbay@kernel.org>
>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>> Cc: Daniel Vetter <daniel@ffwll.ch>
>> ---
>>   Documentation/gpu/rfc/index.rst             |   5 +
>>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>   3 files changed, 698 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>
>> diff --git a/Documentation/gpu/rfc/index.rst
>> b/Documentation/gpu/rfc/index.rst
>> index e4f7b005138d..fc5bc447f30d 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -35,3 +35,8 @@ host such documentation:
>>   .. toctree::
>>        xe.rst
>> +
>> +.. toctree::
>> +   :maxdepth: 1
>> +
>> +   xe_sriov.rst
>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> new file mode 100644
>> index 000000000000..77748204dd83
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> @@ -0,0 +1,501 @@
>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>> +..
>> +.. Intel Xe driver ABI (SR-IOV extensions)
>> +..
>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>> +    the PCI Express (PCIe) specification suite is supported
>> +    starting from 12th generation of Intel Graphics processors.
>> +
>> +    This document describes Xe driver specific additions.
>> +
>> +    For description of generic SR-IOV sysfs attributes see
>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_auto_provisioning
>> +    │   ├── admin_mode
>> +    │   ├── enabled
>> +    │   ├── reset_defaults
>> +    │   ├── resources
>> +    │   │   ├── default_contexts_quota
>> +    │   │   ├── default_doorbells_quota
>> +    │   │   ├── default_ggtt_quota
>> +    │   │   └── default_lmem_quota
>> +    │   ├── scheduling
>> +    │   │   ├── default_exec_quantum_ms
>> +    │   │   └── default_preempt_timeout_us
>> +    │   └── monitoring
>> +    │       ├── default_cat_error_count
>> +    │       ├── default_doorbell_time_us
>> +    │       ├── default_engine_reset_count
>> +    │       ├── default_h2g_time_us
>> +    │       ├── default_irq_time_us
>> +    │       └── default_page_fault_count
> 
> From the department of bike-shedding, one alternative could be to have a
> directory called defaults which avoids having to have the default_
> prefix on everything under it.

like this ?

    /sys/bus/pci/drivers/xe/BDF/
    ├── sriov_auto_provisioning
    │   ├── admin_mode
    │   ├── enabled
    │   └── defaults
    │       ├── resources
    │       │   ├── contexts_quota
    │       │   ├── doorbells_quota
    │       │   ├── ggtt_quota
    │       │   └── lmem_quota
    │       ├── scheduling
    │       │   ├── exec_quantum_ms
    │       │   └── preempt_timeout_us
    │       └── monitoring
    │           ├── cat_error_count
    │           ├── doorbell_time_us
    │           ├── engine_reset_count
    │           ├── h2g_time_us
    │           ├── irq_time_us
    │           └── page_fault_count


> 
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_extensions
> 
> Should this be xe_sriov_extensions or if not doesn't it need agreement
> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci?
> Sriov_auto_provisioning too I guess.

I didn't spot in ABI documents any driver specific additions to device
that were using extra prefix, nor the "reservation" placeholders.

but if we want to go with a prefix, then maybe all our additions at
device level should have one, including already added tileT:

	tile0 -> xe_tile0

or even to match existing mei file we should use "xe." prefix instead:

	xe.sriov_auto_provisioning
	xe.sriov_extension
	xe.tile0
	xe.mei-gsc...

> 
>> +    │   ├── monitoring_period_ms
>> +    │   ├── strict_scheduling_enabled
>> +    │   ├── pf
>> +    │   │   ├── device -> ../../../BDF
>> +    │   │   ├── priority
>> +    │   │   ├── tile0
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   ├── vf1
>> +    │   │   ├── device -> ../../../BDF+1
>> +    │   │   ├── stop
>> +    │   │   ├── tile0
>> +    │   │   │   ├── ggtt_quota
>> +    │   │   │   ├── lmem_quota
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── contexts_quota
>> +    │   │   │   │   ├── doorbells_quota
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   └── vfN
>> +..
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory appears on the device when:
>> +
>> +         - device supports SR-IOV, and
>> +         - device is a Physical Function (PF), and
>> +         - xe driver supports SR-IOV PF on given device, and
>> +         - xe driver supports automatic VFs provisioning.
>> +
>> +        This directory is used as a root for all attributes related to
>> +        automatic provisioning of SR-IOV Physical Function (PF) and/or
>> +        Virtual Functions (VFs).
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool (0, 1)
>> +
>> +        This file represents configuration flag for the automatic VFs
>> +        (un)provisioning that could be performed by the PF.
>> +
>> +        The default value is 1 (true).
>> +
>> +        This flag can be set to false, unless manual provisioning is not
>> +        applicable for given platform or it is not supported by current
>> +        PF implementation. In such cases -EPERM will be returned.
>> +
>> +        This flag will be automatically set to false when there will be
>> +        other attempts to change any of VF's resource provisioning.
>> +        See "sriov_extensions" section for details.
>> +
>> +        This flag can be set back to true if and only if all VFs are
>> +        fully unprovisioned, otherwise -EEXIST error will be returned.
>> +
>> +        false = "disabled"
>> +            When disabled, then PF will not attempt to do automatic
>> +            VFs provisioning when VFs are being enabled and will not
>> +            perform automatic unprovisioning of the VFs when VFs will
>> +            be disabled.
>> +
>> +        true = "enabled"
>> +            When enabled, then on VFs enabling PF will do automatic
>> +            VFs provisioning based on the default settings described
>> +            below.
>> +
>> +            If automatic VFs provisioning fails due to some reasons,
>> +            then VFs will not be enabled.
>> +
>> +            If enabled, all resources allocated during VFs enabling
>> +            will be released during VFs disabling (automatic
>> unprovisioning).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool (0, 1)
>> +
>> +        This file represents configuration flag for the automatic VFs
>> +        provisioning that could be performed by the PF.
>> +
>> +        The default value depends on the platform type.
>> +
>> +        This flag can be changed any time, but will have no effect if
>> +        VFs are already provisioned.
>> +
>> +        If enabled (default on discrete platforms) then the PF will
>> +        retain only minimum hardcoded resources for its own use when
>> +        doing VFs automatic provisioning and will not use any default
>> +        values described below for its own configuration.
>> +
>> +        If disabled (default on integrated platforms) then the PF will
>> +        treat itself like yet another additional VF in all fair resource
>> +        allocations and will also try to apply default provisioning
>> +        values described below for its own configuration.
>> +
> 
> One alternative could be to expose two sets of defaults, the PF and VF
> ones. With the advantage of allowing the "admin mode" / "minimal PF" to
> be explicitly configurable instead of hardcoded. Should be more flexible.

but this "admin_mode" is supposed to be configurable, only default
setting will depend on the platform.

the only "hardcoded" values will be what is needed (or already used) by
the PF for it's basic operations (load firmwares, communication, clears)

also note that when running in "minimal PF" mode then "defaults" are not
applicable to the PF, as PF is not treated like a VF, and IMO any
customization of PF scheduling parameters should be done directly using
"sriov_extensions/pf/tile/gt/*" attributes, not inside
"auto_provisioning/defaults"

only in "non-admin-mode" the PF will benefit from the default
scheduling/monitoring attributes, as then it will be treated as "yet
another VF"

regarding regular resources, PF will always be able to use what was not
allocated to other VFs (as primary goal is to have equal allocations on
each VF, PF will take any remaining anyway)

> 
> If the discrete vs integrated distinction is wanted it could simply be
> made by initialy populating (driver init) the respective defaults based
> on the platform type.
> 
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (WO) bool (1)
>> +
>> +        Writing to this file will reset all default provisioning
>> parameters
>> +        listed below to the default values.
> 
> Maybe this isn't required if you can say it is the responsibility of
> whoever changes the defaults to either know what they are doing, or to
> save and restore themselves if they. It is not a major concern but if
> writing kernel code can be avoided perhaps it can be considered.

true, it's optional, added only as a way to make sure no residual
changes impacts new auto-provisioning (without it we will have to ask to
reload the driver)

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent default provisioning that should be used
>> +        for VFs automatic provisioning.
>> +
>> +        These values can be changed any time, but will have no effect if
>> +        VFs are already provisioned.
>> +
>> +        default_contexts_quota: (RW) integer 0..U32_MAX
>> +            The number of GuC context IDs to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for
>> details.
>> +
>> +        default_doorbells_quota: (RW) integer 0..U32_MAX
>> +            The number of GuC doorbells to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for
>> details.
>> +
>> +        default_ggtt_quota: (RW) integer 0..U32_MAX
>> +            The size of the GGTT address space (in bytes) to provide
>> to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
>> +
>> +        default_lmem_quota: (RW) integer 0..U32_MAX
>> +            The size of the LMEM (in bytes) to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/lmem_quota" for details.
>> +
>> +        default_exec_quantum_ms: (RW) integer 0..U32_MAX
>> +            The GT execution quantum (in millisecs) assigned to the
>> function.
>> +            The default value is 0 (infinify).
>> +            See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for
>> details.
>> +
>> +        default_preempt_timeout_us: (RW) integer 0..U32_MAX
>> +            The GT preemption timeout (in microsecs) assigned to the
>> function.
>> +            The default value is 0 (infinity).
>> +            See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us"
>> for details.
> 
> I have a slight concern here on the usability of GuC specific tunables.
> 
> Whereas one can imagine an external entity (some admin, somewhere) to
> probably pretty much understand what it means to partition the local
> memory, address space, and set the scheduling timeouts (all are intuitve
> and obvious concepts), how are they suppose to approach the GuC
> doorbells and contexts?

GuC submission is mandatory for SRIOV feature to work and both GuC
doorbells and contexts IDs are almost the same kind of resources like
memory or address space.

but agree, they might be not something that were highly visible (on
Linux we don't use doorbells at all, and we had 64K of contexts so never
worried about running out of them).

> 
> It could be a matter of adding more documentation for those two, or it
> even could make sense to shove them under a guc prefix (or
> subdirectory?) to signify the fact they are implementation details and
> not a fundamental concept.

IMO "guc_" prefix would be better, as with yet another subdirectory we
might have too complex structure and GuC stuff is to some extend a
"fundamental concept" for current platforms.

> 
>> +
>> +        default_cat_error_count: (RW) integer 0..U32_MAX
>> +        default_doorbell_time_us: (RW) integer 0..U32_MAX
>> +        default_engine_reset_count: (RW) integer 0..U32_MAX
>> +        default_h2g_time_us: (RW) integer 0..U32_MAX
>> +        default_irq_time_us: (RW) integer 0..U32_MAX
>> +        default_page_fault_count: (RW) integer 0..U32_MAX
>> +            The monitoring threshold to be set for the function.
>> +            The default value is 0 (don't monitor).
>> +            See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory appears on Xe device when:
>> +
>> +         - device supports SR-IOV, and
>> +         - device is a Physical Function (PF), and
>> +         - driver is enabled to support SR-IOV PF on given device.
>> +
>> +        This directory is used as a root for all attributes required to
>> +        manage both Physical Function (PF) and Virtual Functions (VFs).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool
>> +
>> +        This file represents a flag used to determine if scheduling
>> +        parameters should be respected even if there is no active
>> +        workloads submitted by the PF or VFs.
>> +
>> +        This flag is disabled by default, unless strict scheduling is
>> +        not applicable on given platform. In such case this file will
>> +        be read-only.
>> +
>> +        The change to this file may have no effect if VFs are not yet
>> enabled.
>> +        If strict scheduling can't be enabled in GuC then write will
>> fail with -EIO.
> 
> I think the semantics of this need to be documented ie. how it interacts
> with exec_quantum_ms. If it does? I am guessing that it has to otherwise
> I don't know what it would mean - presumably unused timeslices are not
> given to other entities but time just goes wasted? But it is also a

if strict scheduling is disabled and given VF has no work, then it's
execution quantum expires immediately, allowing work from next VF to be
submitted to the GPU, thus maximizing GPU load.

if strict scheduling is set but given VF has no work, then VF's
timeslice is effectively wasted, as GPU will be idle, but at the benefit
that other VFs will see no fluctuations in their interval.

if you own all VMs then you probably don't want to set this ;)

> question on over what time interval. Or that too is purely defined by
> the number of PF+VFs and their respective allocated quanta.
> 
> Also, would there be benefit, assuming it is possible with GuC, to allow
> configuring it per PF/VF?

this is a policy setting, applicable per each GuC scheduler, not per PF
or VF, but moved level up here to make sure we apply same settings for
the whole device (all GuCs) in the same, otherwise it would be too messy

note that the is other related attribute named "priority" that GuC will
implement for the PF, but with later plans to extend this to VFs too.

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) integer
>> +
>> +        This file represents the configuration knob used by adverse
>> event
>> +        monitoring. A value here is the period in millisecs during which
>> +        events are counted and the total is checked against a threshold.
>> +        See "sriov_extensions/vfN/tileT/gtX/thresholds" for more
>> details.
>> +
>> +        Default is 0 (monitoring is disabled).
>> +
>> +        If monitoring capability is not available, then attempt to
>> enable
>> +        will fail with -EPERM error. If monitoring can't be enabled in
>> +        GuC then write will fail with -EIO.
> 
> Could the docs explain if there is a downside to enabling it, which is
> probably why it isn't enabled by default? Because it does sound natural
> that adverse events should be noticed.

the problem is that the definition what should be treated as unwanted VF
activity is very blurred and depends what kind of WLs will be run on VM,
as in some scenarios some sporadic issues are just "normal"

that's why we want to expose only knobs and notification mechanism,
without enforcing any policy on firmware or PF driver level, leaving
that to the administrator

> 
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all attributes related to the SR-IOV
>> +        Physical Function (PF).
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all attributes related to the SR-IOV
>> +        Virtual Function (VF).
>> +
>> +        Note that VF numbers (N) are 1-based as described in PCI
>> SR-IOV specification.
>> +        The Xe driver implementaton follows that naming schema.
>> +
>> +        There will be "vf1", "vf2" up to "vfN" directories, where N
>> matches
>> +        value of the PCI "sriov_totalvfs" attribute.
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all SR-IOV attributes related to the
>> device tile.
>> +        The tile numbers (T) start from 0.
>> +
>> +        There is at least one "tile0/" directory present.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all SR-IOV attributes related to the
>> device GT.
>> +        The GT numbers (X) start from 0.
>> +
>> +        There is at least one "gt0/" directory present.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (symbolic link)
>> +
>> +        Backlink to the PCI device entry representing given function.
>> +        For PF this link is always present.
>> +        For VF this link is present only for currently enabled VFs.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) string
>> +
>> +        This file represents a GuC Scheduler knob to override the
>> default
>> +        round-robin or FIFO scheduler policies implemented by the GuC.
>> +
>> +        The default value is "peer".
>> +
>> +        This flag can be changed, unless such change is not applicable
>> +        for given platform or is not supported by current GuC firmware.
>> +        In such case this file could be read-only or will return -EPERM
>> +        on write attempt.
>> +
>> +        "immediate"
>> +            GuC will Schedule PF workloads immediately and PF
>> +            workloads only until the PF's work queues in GuC
>> +            are empty.
>> +
>> +        "lazy"
>> +            GuC will Schedule PF workloads at the next opportune
>> +            moment and PF workloads only until the PF work queues
>> +            in GuC are empty.
>> +
>> +        "peer"
>> +            GuC Scheduler will treat PF and VFs with equal priority.
> 
> Hmmm this is too very GuC specific and I wonder what is the usecase for
> lazy? Lazy = "don't care when it runs, but when it runs it will run
> everything queued so far", right? Feels a bit odd on first.
> 
> "Immediate" may also not be depending on preemption granularity and
> workloads, right?
> 
> Are there any ideas to express the knobs in a more generic fashion?

will talk with GuC architect to provide a better explanation.

in the meantime, we can treat that as a "placeholder" for the knob that
will provide flexibility when GuC shall attempt to submit, usually
sporadic, workloads from the otherwise idle PF.

this is important in the "admin-mode" and with "strict-scheduling" as we
don't want to waste PF's execution_quantum

if you think that it is too unstable knob, we can try to hide it in
debugfs first, expose when more mature

> 
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (WO) bool (1)
>> +
>> +        Write to this file will force GuC to stop handle any requests
>> from
>> +        this VF, but without triggering a FLR.
>> +        To recover, the full FLR must be issued using generic
>> "device/reset".
>> +
>> +        This file allows to implement custom policy mechanism when VF is
>> +        misbehaving and triggering adverse events above defined
>> thresholds.
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent scheduling parameters of the functions.
>> +
>> +        These scheduling parameters can be changed even if VFs are
>> enabled
>> +        and running, unless such change is not applicable on given
>> platform
>> +        due to fixed hardware or firmware assignment.
>> +
>> +        exec_quantum_ms: (RW) integer 0..U32_MAX
>> +            The GT execution quantum in [ms] assigned to the function.
>> +            Requested quantum might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unlimited).
>> +
>> +        preempt_timeout_us: (RW) integer 0..U32_MAX
>> +            The GT preemption timeout in [us] assigned to the function.
>> +            Requested timeout might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unlimited).
> 
> Alignment for the above two will be visible after read-back?

unfortunately not, as GuC does not expose any interface to query final
values (sometimes they depend on HW clocks, some are FW implementation
details, that could change with new release)

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent shared resource assigned to the functions.
>> +
>> +        These resource parameters can be changed, unless VF is
>> already running,
>> +        or such change is not applicable on given platform due to
>> fixed hardware
>> +        or firmware assignment.
>> +
>> +        Writes to these attributes may fail with:
>> +            -EPERM if change is not applicable on give HW/FW.
>> +            -E2BIG if value larger that HW/FW limit.
>> +            -EDQUOT if value is larger than maximum quota defined by
>> the PF.
>> +            -ENOSPC if PF can't allocate required quota.
>> +            -EBUSY if the resource is currently in use by the VF.
>> +            -EIO if GuC refuses to change provisioning.
> 
> Why it would refuse if input is valid? In other words, what is the
> user/admin supposed to do on -EIO?

in early implementations we were fully relying on the GuC and return
-EIO on every error, as we didn't track the VF state that would allow us
to catch improper changes while VF is running (-EBUSY above) but even if
we (PF) do our best then GuC is still a decider here and may return
(unlikely) error.

as recovery we will be suggesting a VF FLR and then retry.

> 
>> +
>> +        ggtt_quota: (RW) integer 0..U64_MAX
>> +            The size of the GGTT address space (in bytes) assigned to
>> the VF.
>> +            The value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        lmem_quota: (RW) integer 0..U64_MAX
>> +            The size of the Local Memory (in bytes) assigned to the VF.
>> +            The value might be aligned per HW/FW requirements.
>> +
>> +            This attribute is only available on discrete platforms.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        contexts_quota: (RW) 0..U16_MAX
>> +            The number of GuC submission contexts assigned to the VF.
>> +            This value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        doorbells_quota: (RW) 0..U16_MAX
>> +            The number of GuC doorbells assigned to the VF.
>> +            This value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent threshold values used by the GuC to
>> trigger
>> +        security events if adverse event monitoring is enabled.
> 
> How are the security events delivered? There is mention of uevents in a
> later paragraph - are they already defined or should be together with
> this so the link can be place here?

chicken/egg problem - we don't have uevent fully defined as we were
waiting for these threshold definitions to be accepted, and without them
being available we didn't generate uevent, but in early code we had:

"THRESHOLD_EXCEEDED=1"
"THRESHOLD_ID=%#x"
"VF_ID=%u"

> 
>> +
>> +        These thresholds are checked every "monitoring_period_ms".
>> +        Refer to GuC ABI for details about each threshold category.
> 
> Is it possible to have a link here to GuC ABI?

(shared offline, as GuC spec is not public yet)

> 
> Regards,
> 
> Tvrtko

Thanks,
Michal


> 
>> +
>> +        Default value for all thresholds is 0 (disabled).
>> +
>> +        cat_error_count: (RW) integer
>> +        doorbell_time_us: (RW) integer
>> +        engine_reset_count: (RW) integer
>> +        h2g_time_us: (RW) integer
>> +        irq_time_us: (RW) integer
>> +        page_fault_count: (RW) integer
>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst
>> b/Documentation/gpu/rfc/xe_sriov.rst
>> new file mode 100644
>> index 000000000000..574f6414eabb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>> @@ -0,0 +1,192 @@
>> +.. SPDX-License-Identifier: MIT
>> +
>> +========================
>> +Xe – SR-IOV Support Plan
>> +========================
>> +
>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe)
>> +specification suite is supported starting from 12th generation of
>> Intel Graphics
>> +processors.
>> +
>> +This document describes planned ABI of the new Xe driver (see xe.rst)
>> that will
>> +provide flexible configuration and management options related to the
>> SR-IOV.
>> +It will also highlight few most important changes to the Xe driver
>> +implementation to deal with Intel GPU SR-IOV specific requirements.
>> +
>> +
>> +SR-IOV Capability
>> +=================
>> +
>> +Due to SR-IOV complexity and required co-operation between hardware,
>> firmware
>> +and kernel drivers, not all Xe architecture platforms might have
>> SR-IOV enabled
>> +or fully functional.
>> +
>> +To control at the driver level which platform will provide support
>> for SR-IOV,
>> +as we can't just rely on the PCI configuration data exposed by the
>> hardware,
>> +we will introduce "has_sriov" flag to the struct xe_device_desc that
>> describes
>> +a device capabilities that driver checks during the probe.
>> +
>> +Initially this flag will be set to disabled even on platforms that we
>> plan to
>> +support. We will enable this flag only once we finish merging all
>> required
>> +changes to the driver and related validated firmwares are also made
>> available.
>> +
>> +
>> +SR-IOV Platforms
>> +================
>> +
>> +Initially we plan to add SR-IOV functionality to the following SDV
>> platforms
>> +already supported by the Xe driver:
>> +
>> + - TGL (up to 7 VFs)
>> + - ADL (up to 7 VFs)
>> + - MTL (up to 7 VFs)
>> + - ATSM (up to 31 VFs)
>> + - PVC (up to 63 VFs)
>> +
>> +Newer platforms will be supported later, but we hope that enabling
>> will be
>> +much faster, as majority of the driver changes are either platform
>> agnostic
>> +or are similar between earlier platforms (hence we start with SDVs).
>> +
>> +
>> +PF Mode
>> +=======
>> +
>> +Support in the driver for acting in Physical Function (PF) mode, i.e.
>> mode
>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and
>> will be
>> +enabled by default.
>> +
>> +However, due to potentially conflicting requirements for SR-IOV and
>> other mega
>> +features, we might want to have an option to disable SR-IOV PF mode
>> support at
>> +the driver load time.
>> +
>> +Thus, we plan to use additional modparam named "sriov_totalvfs" which
>> if set to
>> +0 will force the driver to operate in the native (non-virtualized) mode.
>> +The same modparam could be used to limit number of supported Virtual
>> Functions
>> +(VFs) by the driver compared to the hardware limit exposed in PCI
>> configuration.
>> +
>> +The name of this modparam corresponds to the existing PCI sysfs
>> attribute, that
>> +by default exposes hardware capability.
>> +
>> +The default value of this param will allow to support all possible
>> VFs as
>> +claimed by the hardware.
>> +
>> +This modparam will have no effect if driver is running on the VF device.
>> +
>> +
>> +VFs Enabling
>> +============
>> +
>> +To enable or disable VFs we plan to rely on existing sysfs attribute
>> exposed by
>> +the PCI subsystem named "sriov_numvfs". We will provide all necessary
>> tweaks to
>> +provision VFs in our custom implementation of the "sriov_configure"
>> hook from
>> +the struct pci_driver.
>> +
>> +If for some reason, including explicit request to disable SR-IOV PF
>> mode using
>> +modparam, we will not be able to correctly support any VFs, driver
>> will change
>> +number of supported VFs, exposed to the userspace by "sriov_totalvfs"
>> attribute,
>> +to 0, thus preventing configuration of the VFs.
>> +
>> +
>> +VF Mode
>> +=======
>> +
>> +When driver is running on the VF device, then due to hardware
>> enforcements,
>> +access to the privileged registers is not possible. To avoid relying
>> on these
>> +registers, we plan to perform early detection if we are running on
>> the VF
>> +device using dedicated VF_CAP(0x1901f8) register and then use global
>> macro
>> +IS_SRIOV_VF(xe) to control the driver logic.
>> +
>> +To speed up merging of the required changes, we might first introduce
>> dummy
>> +macro that is always set to false, to prepare driver to avoid some
>> code paths
>> +before we finalize our VF mode detection and other VFs enabling changes.
>> +
>> +
>> +Resources
>> +=========
>> +
>> +Most of the hardware (or firmware) resources available on the Xe
>> architecture,
>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
>> between PF and
>> +VFs and will require some provisioning steps to assign those
>> resources for use
>> +by the VF.
>> +
>> +Until VFs are provisioned with resources, the PF driver will be able
>> to use all
>> +resources, in the same way as it would be running in non-virtualized
>> mode.
>> +
>> +If some resource (of part or region of it) is assigned to specific
>> VF, then PF
>> +is not allowed to use that part or region of the resource, but can
>> continue to
>> +use whatever is left available.
>> +
>> +Those resources are usually fully virtualized, so they will not
>> require any
>> +special handling when used by the VF driver, except that VF driver
>> must know
>> +the assigned quota.
>> +
>> +The most notable exception is the GGTT address space, as on some
>> platforms,
>> +the VF driver must additionally know the real range that it can access.
>> +
>> +Once the resources were assigned to the VF use and the VF driver has
>> started,
>> +then it is not allowed to change such provisioning, as that would
>> break the
>> +VF driver. To make changes the VF driver, which was using these
>> resources,
>> +must be unloaded (or the VM is terminated) and the VF device must be
>> reset
>> +using the FLR.
>> +
>> +
>> +Scheduling
>> +==========
>> +
>> +The workloads from PF driver and VF drivers must be submitted to the
>> hardware
>> +always by using the GuC submission mechanism. Unless VF has exclusive
>> access
>> +to the GT then submissions from different VFs are time-sliced and
>> controlled
>> +with additional "execution_quantum" and "preemption_timeout" parameters.
>> +
>> +In contrast to the resource provisioning, those scheduling parameters
>> can be
>> +changed even if VF drivers are already running and are active.
>> +
>> +
>> +Automatic VFs Provisioning
>> +==========================
>> +
>> +To provide out-of-the box experience when user will be enabling VFs
>> using
>> +generic "sriov_numvfs" attribute without requiring complex
>> provisioning steps,
>> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
>> +
>> +By default, all VFs will be allocated with the fair amount of the
>> mandatory
>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling
>> parameters.
>> +Such provisioning should be sufficient for most of the normal usages,
>> when
>> +no strict SLA is required.
>> +
>> +The PF driver will also expose some additional sysfs files to allow
>> adjusting
>> +this automatic VFs provisioning, like default values for most of the
>> +provisioning parameters that PF will then apply for each enabled VF.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +Manual VFs Provisioning
>> +=======================
>> +
>> +If automatic VFs provisioning, which applies same configuration to
>> every VF,
>> +is not sufficient or there is a need for advanced customization of
>> some VF,
>> +the PF driver will also provide extended sysfs interface which will
>> allow
>> +control every provisioning attribute to the lowest feasible level.
>> +
>> +It is expected that these low-level attributes will be mostly used by
>> the
>> +advanced users or by the custom tools that will setup configurations
>> that
>> +meet predefined and validated SLA as required by the customers.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +VFs Monitoring
>> +==============
>> +
>> +In addition to the resource provisioning or changing scheduling
>> parameters,
>> +the PF driver might also allow configure some monitoring parameters,
>> like
>> +thresholds of adverse events or sample period, to track undesired
>> behavior
>> +of the VFs that could impact the whole system.
>> +
>> +Once those thresholds are setup and sampling period is defined, the
>> GuC will
>> +notify the PF driver about which VF is excessing the threshold and
>> then PF is
>> +able to trigger the uevent to notify the administrator (or VMM) that
>> could
>> +take some action against the VF.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 16:55     ` Michal Wajdeczko
  0 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 16:55 UTC (permalink / raw)
  To: Tvrtko Ursulin, intel-xe, dri-devel; +Cc: Daniel Vetter, Rodrigo Vivi



On 14.11.2023 13:37, Tvrtko Ursulin wrote:
> 
> On 10/11/2023 18:22, Michal Wajdeczko wrote:
>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe) specification suite is supported starting from 12th
>> generation of Intel Graphics processors.
>>
>> This RFC aims to explain how do we want to add support for SR-IOV
>> to the new Xe driver and to propose related additions to the sysfs.
>>
>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>> Cc: Oded Gabbay <ogabbay@kernel.org>
>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>> Cc: Daniel Vetter <daniel@ffwll.ch>
>> ---
>>   Documentation/gpu/rfc/index.rst             |   5 +
>>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501 ++++++++++++++++++++
>>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>   3 files changed, 698 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>
>> diff --git a/Documentation/gpu/rfc/index.rst
>> b/Documentation/gpu/rfc/index.rst
>> index e4f7b005138d..fc5bc447f30d 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -35,3 +35,8 @@ host such documentation:
>>   .. toctree::
>>        xe.rst
>> +
>> +.. toctree::
>> +   :maxdepth: 1
>> +
>> +   xe_sriov.rst
>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> new file mode 100644
>> index 000000000000..77748204dd83
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>> @@ -0,0 +1,501 @@
>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>> +..
>> +.. Intel Xe driver ABI (SR-IOV extensions)
>> +..
>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>> +    the PCI Express (PCIe) specification suite is supported
>> +    starting from 12th generation of Intel Graphics processors.
>> +
>> +    This document describes Xe driver specific additions.
>> +
>> +    For description of generic SR-IOV sysfs attributes see
>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_auto_provisioning
>> +    │   ├── admin_mode
>> +    │   ├── enabled
>> +    │   ├── reset_defaults
>> +    │   ├── resources
>> +    │   │   ├── default_contexts_quota
>> +    │   │   ├── default_doorbells_quota
>> +    │   │   ├── default_ggtt_quota
>> +    │   │   └── default_lmem_quota
>> +    │   ├── scheduling
>> +    │   │   ├── default_exec_quantum_ms
>> +    │   │   └── default_preempt_timeout_us
>> +    │   └── monitoring
>> +    │       ├── default_cat_error_count
>> +    │       ├── default_doorbell_time_us
>> +    │       ├── default_engine_reset_count
>> +    │       ├── default_h2g_time_us
>> +    │       ├── default_irq_time_us
>> +    │       └── default_page_fault_count
> 
> From the department of bike-shedding, one alternative could be to have a
> directory called defaults which avoids having to have the default_
> prefix on everything under it.

like this ?

    /sys/bus/pci/drivers/xe/BDF/
    ├── sriov_auto_provisioning
    │   ├── admin_mode
    │   ├── enabled
    │   └── defaults
    │       ├── resources
    │       │   ├── contexts_quota
    │       │   ├── doorbells_quota
    │       │   ├── ggtt_quota
    │       │   └── lmem_quota
    │       ├── scheduling
    │       │   ├── exec_quantum_ms
    │       │   └── preempt_timeout_us
    │       └── monitoring
    │           ├── cat_error_count
    │           ├── doorbell_time_us
    │           ├── engine_reset_count
    │           ├── h2g_time_us
    │           ├── irq_time_us
    │           └── page_fault_count


> 
>> +
>> +    /sys/bus/pci/drivers/xe/BDF/
>> +    ├── sriov_extensions
> 
> Should this be xe_sriov_extensions or if not doesn't it need agreement
> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci?
> Sriov_auto_provisioning too I guess.

I didn't spot in ABI documents any driver specific additions to device
that were using extra prefix, nor the "reservation" placeholders.

but if we want to go with a prefix, then maybe all our additions at
device level should have one, including already added tileT:

	tile0 -> xe_tile0

or even to match existing mei file we should use "xe." prefix instead:

	xe.sriov_auto_provisioning
	xe.sriov_extension
	xe.tile0
	xe.mei-gsc...

> 
>> +    │   ├── monitoring_period_ms
>> +    │   ├── strict_scheduling_enabled
>> +    │   ├── pf
>> +    │   │   ├── device -> ../../../BDF
>> +    │   │   ├── priority
>> +    │   │   ├── tile0
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   ├── vf1
>> +    │   │   ├── device -> ../../../BDF+1
>> +    │   │   ├── stop
>> +    │   │   ├── tile0
>> +    │   │   │   ├── ggtt_quota
>> +    │   │   │   ├── lmem_quota
>> +    │   │   │   ├── gt0
>> +    │   │   │   │   ├── contexts_quota
>> +    │   │   │   │   ├── doorbells_quota
>> +    │   │   │   │   ├── exec_quantum_ms
>> +    │   │   │   │   ├── preempt_timeout_us
>> +    │   │   │   │   └── thresholds
>> +    │   │   │   │       ├── cat_error_count
>> +    │   │   │   │       ├── doorbell_time_us
>> +    │   │   │   │       ├── engine_reset_count
>> +    │   │   │   │       ├── h2g_time_us
>> +    │   │   │   │       ├── irq_time_us
>> +    │   │   │   │       └── page_fault_count
>> +    │   │   │   └── gtX
>> +    │   │   └── tileT
>> +    │   └── vfN
>> +..
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory appears on the device when:
>> +
>> +         - device supports SR-IOV, and
>> +         - device is a Physical Function (PF), and
>> +         - xe driver supports SR-IOV PF on given device, and
>> +         - xe driver supports automatic VFs provisioning.
>> +
>> +        This directory is used as a root for all attributes related to
>> +        automatic provisioning of SR-IOV Physical Function (PF) and/or
>> +        Virtual Functions (VFs).
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/enabled
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool (0, 1)
>> +
>> +        This file represents configuration flag for the automatic VFs
>> +        (un)provisioning that could be performed by the PF.
>> +
>> +        The default value is 1 (true).
>> +
>> +        This flag can be set to false, unless manual provisioning is not
>> +        applicable for given platform or it is not supported by current
>> +        PF implementation. In such cases -EPERM will be returned.
>> +
>> +        This flag will be automatically set to false when there will be
>> +        other attempts to change any of VF's resource provisioning.
>> +        See "sriov_extensions" section for details.
>> +
>> +        This flag can be set back to true if and only if all VFs are
>> +        fully unprovisioned, otherwise -EEXIST error will be returned.
>> +
>> +        false = "disabled"
>> +            When disabled, then PF will not attempt to do automatic
>> +            VFs provisioning when VFs are being enabled and will not
>> +            perform automatic unprovisioning of the VFs when VFs will
>> +            be disabled.
>> +
>> +        true = "enabled"
>> +            When enabled, then on VFs enabling PF will do automatic
>> +            VFs provisioning based on the default settings described
>> +            below.
>> +
>> +            If automatic VFs provisioning fails due to some reasons,
>> +            then VFs will not be enabled.
>> +
>> +            If enabled, all resources allocated during VFs enabling
>> +            will be released during VFs disabling (automatic
>> unprovisioning).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/admin_mode
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool (0, 1)
>> +
>> +        This file represents configuration flag for the automatic VFs
>> +        provisioning that could be performed by the PF.
>> +
>> +        The default value depends on the platform type.
>> +
>> +        This flag can be changed any time, but will have no effect if
>> +        VFs are already provisioned.
>> +
>> +        If enabled (default on discrete platforms) then the PF will
>> +        retain only minimum hardcoded resources for its own use when
>> +        doing VFs automatic provisioning and will not use any default
>> +        values described below for its own configuration.
>> +
>> +        If disabled (default on integrated platforms) then the PF will
>> +        treat itself like yet another additional VF in all fair resource
>> +        allocations and will also try to apply default provisioning
>> +        values described below for its own configuration.
>> +
> 
> One alternative could be to expose two sets of defaults, the PF and VF
> ones. With the advantage of allowing the "admin mode" / "minimal PF" to
> be explicitly configurable instead of hardcoded. Should be more flexible.

but this "admin_mode" is supposed to be configurable, only default
setting will depend on the platform.

the only "hardcoded" values will be what is needed (or already used) by
the PF for it's basic operations (load firmwares, communication, clears)

also note that when running in "minimal PF" mode then "defaults" are not
applicable to the PF, as PF is not treated like a VF, and IMO any
customization of PF scheduling parameters should be done directly using
"sriov_extensions/pf/tile/gt/*" attributes, not inside
"auto_provisioning/defaults"

only in "non-admin-mode" the PF will benefit from the default
scheduling/monitoring attributes, as then it will be treated as "yet
another VF"

regarding regular resources, PF will always be able to use what was not
allocated to other VFs (as primary goal is to have equal allocations on
each VF, PF will take any remaining anyway)

> 
> If the discrete vs integrated distinction is wanted it could simply be
> made by initialy populating (driver init) the respective defaults based
> on the platform type.
> 
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/reset_defaults
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (WO) bool (1)
>> +
>> +        Writing to this file will reset all default provisioning
>> parameters
>> +        listed below to the default values.
> 
> Maybe this isn't required if you can say it is the responsibility of
> whoever changes the defaults to either know what they are doing, or to
> save and restore themselves if they. It is not a major concern but if
> writing kernel code can be avoided perhaps it can be considered.

true, it's optional, added only as a way to make sure no residual
changes impacts new auto-provisioning (without it we will have to ask to
reload the driver)

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_contexts_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_doorbells_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_ggtt_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/resources/default_lmem_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/scheduling/default_preempt_timeout_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning/monitoring/default_page_fault_count
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent default provisioning that should be used
>> +        for VFs automatic provisioning.
>> +
>> +        These values can be changed any time, but will have no effect if
>> +        VFs are already provisioned.
>> +
>> +        default_contexts_quota: (RW) integer 0..U32_MAX
>> +            The number of GuC context IDs to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/gtX/contexts_quota" for
>> details.
>> +
>> +        default_doorbells_quota: (RW) integer 0..U32_MAX
>> +            The number of GuC doorbells to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for
>> details.
>> +
>> +        default_ggtt_quota: (RW) integer 0..U32_MAX
>> +            The size of the GGTT address space (in bytes) to provide
>> to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/ggtt_quota" for details.
>> +
>> +        default_lmem_quota: (RW) integer 0..U32_MAX
>> +            The size of the LMEM (in bytes) to provide to the VF.
>> +            The default value is 0 (use fair allocations).
>> +            See "sriov_extensions/vfN/tileT/lmem_quota" for details.
>> +
>> +        default_exec_quantum_ms: (RW) integer 0..U32_MAX
>> +            The GT execution quantum (in millisecs) assigned to the
>> function.
>> +            The default value is 0 (infinify).
>> +            See "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for
>> details.
>> +
>> +        default_preempt_timeout_us: (RW) integer 0..U32_MAX
>> +            The GT preemption timeout (in microsecs) assigned to the
>> function.
>> +            The default value is 0 (infinity).
>> +            See "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us"
>> for details.
> 
> I have a slight concern here on the usability of GuC specific tunables.
> 
> Whereas one can imagine an external entity (some admin, somewhere) to
> probably pretty much understand what it means to partition the local
> memory, address space, and set the scheduling timeouts (all are intuitve
> and obvious concepts), how are they suppose to approach the GuC
> doorbells and contexts?

GuC submission is mandatory for SRIOV feature to work and both GuC
doorbells and contexts IDs are almost the same kind of resources like
memory or address space.

but agree, they might be not something that were highly visible (on
Linux we don't use doorbells at all, and we had 64K of contexts so never
worried about running out of them).

> 
> It could be a matter of adding more documentation for those two, or it
> even could make sense to shove them under a guc prefix (or
> subdirectory?) to signify the fact they are implementation details and
> not a fundamental concept.

IMO "guc_" prefix would be better, as with yet another subdirectory we
might have too complex structure and GuC stuff is to some extend a
"fundamental concept" for current platforms.

> 
>> +
>> +        default_cat_error_count: (RW) integer 0..U32_MAX
>> +        default_doorbell_time_us: (RW) integer 0..U32_MAX
>> +        default_engine_reset_count: (RW) integer 0..U32_MAX
>> +        default_h2g_time_us: (RW) integer 0..U32_MAX
>> +        default_irq_time_us: (RW) integer 0..U32_MAX
>> +        default_page_fault_count: (RW) integer 0..U32_MAX
>> +            The monitoring threshold to be set for the function.
>> +            The default value is 0 (don't monitor).
>> +            See "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory appears on Xe device when:
>> +
>> +         - device supports SR-IOV, and
>> +         - device is a Physical Function (PF), and
>> +         - driver is enabled to support SR-IOV PF on given device.
>> +
>> +        This directory is used as a root for all attributes required to
>> +        manage both Physical Function (PF) and Virtual Functions (VFs).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/strict_scheduling_enabled
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) bool
>> +
>> +        This file represents a flag used to determine if scheduling
>> +        parameters should be respected even if there is no active
>> +        workloads submitted by the PF or VFs.
>> +
>> +        This flag is disabled by default, unless strict scheduling is
>> +        not applicable on given platform. In such case this file will
>> +        be read-only.
>> +
>> +        The change to this file may have no effect if VFs are not yet
>> enabled.
>> +        If strict scheduling can't be enabled in GuC then write will
>> fail with -EIO.
> 
> I think the semantics of this need to be documented ie. how it interacts
> with exec_quantum_ms. If it does? I am guessing that it has to otherwise
> I don't know what it would mean - presumably unused timeslices are not
> given to other entities but time just goes wasted? But it is also a

if strict scheduling is disabled and given VF has no work, then it's
execution quantum expires immediately, allowing work from next VF to be
submitted to the GPU, thus maximizing GPU load.

if strict scheduling is set but given VF has no work, then VF's
timeslice is effectively wasted, as GPU will be idle, but at the benefit
that other VFs will see no fluctuations in their interval.

if you own all VMs then you probably don't want to set this ;)

> question on over what time interval. Or that too is purely defined by
> the number of PF+VFs and their respective allocated quanta.
> 
> Also, would there be benefit, assuming it is possible with GuC, to allow
> configuring it per PF/VF?

this is a policy setting, applicable per each GuC scheduler, not per PF
or VF, but moved level up here to make sure we apply same settings for
the whole device (all GuCs) in the same, otherwise it would be too messy

note that the is other related attribute named "priority" that GuC will
implement for the PF, but with later plans to extend this to VFs too.

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/monitoring_period_ms
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) integer
>> +
>> +        This file represents the configuration knob used by adverse
>> event
>> +        monitoring. A value here is the period in millisecs during which
>> +        events are counted and the total is checked against a threshold.
>> +        See "sriov_extensions/vfN/tileT/gtX/thresholds" for more
>> details.
>> +
>> +        Default is 0 (monitoring is disabled).
>> +
>> +        If monitoring capability is not available, then attempt to
>> enable
>> +        will fail with -EPERM error. If monitoring can't be enabled in
>> +        GuC then write will fail with -EIO.
> 
> Could the docs explain if there is a downside to enabling it, which is
> probably why it isn't enabled by default? Because it does sound natural
> that adverse events should be noticed.

the problem is that the definition what should be treated as unwanted VF
activity is very blurred and depends what kind of WLs will be run on VM,
as in some scenarios some sporadic issues are just "normal"

that's why we want to expose only knobs and notification mechanism,
without enforcing any policy on firmware or PF driver level, leaving
that to the administrator

> 
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all attributes related to the SR-IOV
>> +        Physical Function (PF).
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all attributes related to the SR-IOV
>> +        Virtual Function (VF).
>> +
>> +        Note that VF numbers (N) are 1-based as described in PCI
>> SR-IOV specification.
>> +        The Xe driver implementaton follows that naming schema.
>> +
>> +        There will be "vf1", "vf2" up to "vfN" directories, where N
>> matches
>> +        value of the PCI "sriov_totalvfs" attribute.
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all SR-IOV attributes related to the
>> device tile.
>> +        The tile numbers (T) start from 0.
>> +
>> +        There is at least one "tile0/" directory present.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        This directory holds all SR-IOV attributes related to the
>> device GT.
>> +        The GT numbers (X) start from 0.
>> +
>> +        There is at least one "gt0/" directory present.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/device
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/device
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (symbolic link)
>> +
>> +        Backlink to the PCI device entry representing given function.
>> +        For PF this link is always present.
>> +        For VF this link is present only for currently enabled VFs.
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/priority
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (RW) string
>> +
>> +        This file represents a GuC Scheduler knob to override the
>> default
>> +        round-robin or FIFO scheduler policies implemented by the GuC.
>> +
>> +        The default value is "peer".
>> +
>> +        This flag can be changed, unless such change is not applicable
>> +        for given platform or is not supported by current GuC firmware.
>> +        In such case this file could be read-only or will return -EPERM
>> +        on write attempt.
>> +
>> +        "immediate"
>> +            GuC will Schedule PF workloads immediately and PF
>> +            workloads only until the PF's work queues in GuC
>> +            are empty.
>> +
>> +        "lazy"
>> +            GuC will Schedule PF workloads at the next opportune
>> +            moment and PF workloads only until the PF work queues
>> +            in GuC are empty.
>> +
>> +        "peer"
>> +            GuC Scheduler will treat PF and VFs with equal priority.
> 
> Hmmm this is too very GuC specific and I wonder what is the usecase for
> lazy? Lazy = "don't care when it runs, but when it runs it will run
> everything queued so far", right? Feels a bit odd on first.
> 
> "Immediate" may also not be depending on preemption granularity and
> workloads, right?
> 
> Are there any ideas to express the knobs in a more generic fashion?

will talk with GuC architect to provide a better explanation.

in the meantime, we can treat that as a "placeholder" for the knob that
will provide flexibility when GuC shall attempt to submit, usually
sporadic, workloads from the otherwise idle PF.

this is important in the "admin-mode" and with "strict-scheduling" as we
don't want to waste PF's execution_quantum

if you think that it is too unstable knob, we can try to hide it in
debugfs first, expose when more mature

> 
>> +
>> +
>> +What:        /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/stop
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        (WO) bool (1)
>> +
>> +        Write to this file will force GuC to stop handle any requests
>> from
>> +        this VF, but without triggering a FLR.
>> +        To recover, the full FLR must be issued using generic
>> "device/reset".
>> +
>> +        This file allows to implement custom policy mechanism when VF is
>> +        misbehaving and triggering adverse events above defined
>> thresholds.
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/preempt_timeout_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/exec_quantum_ms
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/preempt_timeout_us
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent scheduling parameters of the functions.
>> +
>> +        These scheduling parameters can be changed even if VFs are
>> enabled
>> +        and running, unless such change is not applicable on given
>> platform
>> +        due to fixed hardware or firmware assignment.
>> +
>> +        exec_quantum_ms: (RW) integer 0..U32_MAX
>> +            The GT execution quantum in [ms] assigned to the function.
>> +            Requested quantum might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unlimited).
>> +
>> +        preempt_timeout_us: (RW) integer 0..U32_MAX
>> +            The GT preemption timeout in [us] assigned to the function.
>> +            Requested timeout might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unlimited).
> 
> Alignment for the above two will be visible after read-back?

unfortunately not, as GuC does not expose any interface to query final
values (sometimes they depend on HW clocks, some are FW implementation
details, that could change with new release)

> 
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/ggtt_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/lmem_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/contexts_quota
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/doorbells_quota
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent shared resource assigned to the functions.
>> +
>> +        These resource parameters can be changed, unless VF is
>> already running,
>> +        or such change is not applicable on given platform due to
>> fixed hardware
>> +        or firmware assignment.
>> +
>> +        Writes to these attributes may fail with:
>> +            -EPERM if change is not applicable on give HW/FW.
>> +            -E2BIG if value larger that HW/FW limit.
>> +            -EDQUOT if value is larger than maximum quota defined by
>> the PF.
>> +            -ENOSPC if PF can't allocate required quota.
>> +            -EBUSY if the resource is currently in use by the VF.
>> +            -EIO if GuC refuses to change provisioning.
> 
> Why it would refuse if input is valid? In other words, what is the
> user/admin supposed to do on -EIO?

in early implementations we were fully relying on the GuC and return
-EIO on every error, as we didn't track the VF state that would allow us
to catch improper changes while VF is running (-EBUSY above) but even if
we (PF) do our best then GuC is still a decider here and may return
(unlikely) error.

as recovery we will be suggesting a VF FLR and then retry.

> 
>> +
>> +        ggtt_quota: (RW) integer 0..U64_MAX
>> +            The size of the GGTT address space (in bytes) assigned to
>> the VF.
>> +            The value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        lmem_quota: (RW) integer 0..U64_MAX
>> +            The size of the Local Memory (in bytes) assigned to the VF.
>> +            The value might be aligned per HW/FW requirements.
>> +
>> +            This attribute is only available on discrete platforms.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        contexts_quota: (RW) 0..U16_MAX
>> +            The number of GuC submission contexts assigned to the VF.
>> +            This value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +        doorbells_quota: (RW) 0..U16_MAX
>> +            The number of GuC doorbells assigned to the VF.
>> +            This value might be aligned per HW/FW requirements.
>> +
>> +            Default is 0 (unprovisioned).
>> +
>> +
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/tileT/gtX/thresholds/page_fault_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/cat_error_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/doorbell_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/engine_reset_count
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/h2g_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/irq_time_us
>> +What:       
>> /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/tileT/gtX/thresholds/page_fault_count
>> +Date:        2024
>> +KernelVersion:    TBD
>> +Contact:    intel-xe@lists.freedesktop.org
>> +Description:
>> +        These files represent threshold values used by the GuC to
>> trigger
>> +        security events if adverse event monitoring is enabled.
> 
> How are the security events delivered? There is mention of uevents in a
> later paragraph - are they already defined or should be together with
> this so the link can be place here?

chicken/egg problem - we don't have uevent fully defined as we were
waiting for these threshold definitions to be accepted, and without them
being available we didn't generate uevent, but in early code we had:

"THRESHOLD_EXCEEDED=1"
"THRESHOLD_ID=%#x"
"VF_ID=%u"

> 
>> +
>> +        These thresholds are checked every "monitoring_period_ms".
>> +        Refer to GuC ABI for details about each threshold category.
> 
> Is it possible to have a link here to GuC ABI?

(shared offline, as GuC spec is not public yet)

> 
> Regards,
> 
> Tvrtko

Thanks,
Michal


> 
>> +
>> +        Default value for all thresholds is 0 (disabled).
>> +
>> +        cat_error_count: (RW) integer
>> +        doorbell_time_us: (RW) integer
>> +        engine_reset_count: (RW) integer
>> +        h2g_time_us: (RW) integer
>> +        irq_time_us: (RW) integer
>> +        page_fault_count: (RW) integer
>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst
>> b/Documentation/gpu/rfc/xe_sriov.rst
>> new file mode 100644
>> index 000000000000..574f6414eabb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>> @@ -0,0 +1,192 @@
>> +.. SPDX-License-Identifier: MIT
>> +
>> +========================
>> +Xe – SR-IOV Support Plan
>> +========================
>> +
>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>> Express (PCIe)
>> +specification suite is supported starting from 12th generation of
>> Intel Graphics
>> +processors.
>> +
>> +This document describes planned ABI of the new Xe driver (see xe.rst)
>> that will
>> +provide flexible configuration and management options related to the
>> SR-IOV.
>> +It will also highlight few most important changes to the Xe driver
>> +implementation to deal with Intel GPU SR-IOV specific requirements.
>> +
>> +
>> +SR-IOV Capability
>> +=================
>> +
>> +Due to SR-IOV complexity and required co-operation between hardware,
>> firmware
>> +and kernel drivers, not all Xe architecture platforms might have
>> SR-IOV enabled
>> +or fully functional.
>> +
>> +To control at the driver level which platform will provide support
>> for SR-IOV,
>> +as we can't just rely on the PCI configuration data exposed by the
>> hardware,
>> +we will introduce "has_sriov" flag to the struct xe_device_desc that
>> describes
>> +a device capabilities that driver checks during the probe.
>> +
>> +Initially this flag will be set to disabled even on platforms that we
>> plan to
>> +support. We will enable this flag only once we finish merging all
>> required
>> +changes to the driver and related validated firmwares are also made
>> available.
>> +
>> +
>> +SR-IOV Platforms
>> +================
>> +
>> +Initially we plan to add SR-IOV functionality to the following SDV
>> platforms
>> +already supported by the Xe driver:
>> +
>> + - TGL (up to 7 VFs)
>> + - ADL (up to 7 VFs)
>> + - MTL (up to 7 VFs)
>> + - ATSM (up to 31 VFs)
>> + - PVC (up to 63 VFs)
>> +
>> +Newer platforms will be supported later, but we hope that enabling
>> will be
>> +much faster, as majority of the driver changes are either platform
>> agnostic
>> +or are similar between earlier platforms (hence we start with SDVs).
>> +
>> +
>> +PF Mode
>> +=======
>> +
>> +Support in the driver for acting in Physical Function (PF) mode, i.e.
>> mode
>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV and
>> will be
>> +enabled by default.
>> +
>> +However, due to potentially conflicting requirements for SR-IOV and
>> other mega
>> +features, we might want to have an option to disable SR-IOV PF mode
>> support at
>> +the driver load time.
>> +
>> +Thus, we plan to use additional modparam named "sriov_totalvfs" which
>> if set to
>> +0 will force the driver to operate in the native (non-virtualized) mode.
>> +The same modparam could be used to limit number of supported Virtual
>> Functions
>> +(VFs) by the driver compared to the hardware limit exposed in PCI
>> configuration.
>> +
>> +The name of this modparam corresponds to the existing PCI sysfs
>> attribute, that
>> +by default exposes hardware capability.
>> +
>> +The default value of this param will allow to support all possible
>> VFs as
>> +claimed by the hardware.
>> +
>> +This modparam will have no effect if driver is running on the VF device.
>> +
>> +
>> +VFs Enabling
>> +============
>> +
>> +To enable or disable VFs we plan to rely on existing sysfs attribute
>> exposed by
>> +the PCI subsystem named "sriov_numvfs". We will provide all necessary
>> tweaks to
>> +provision VFs in our custom implementation of the "sriov_configure"
>> hook from
>> +the struct pci_driver.
>> +
>> +If for some reason, including explicit request to disable SR-IOV PF
>> mode using
>> +modparam, we will not be able to correctly support any VFs, driver
>> will change
>> +number of supported VFs, exposed to the userspace by "sriov_totalvfs"
>> attribute,
>> +to 0, thus preventing configuration of the VFs.
>> +
>> +
>> +VF Mode
>> +=======
>> +
>> +When driver is running on the VF device, then due to hardware
>> enforcements,
>> +access to the privileged registers is not possible. To avoid relying
>> on these
>> +registers, we plan to perform early detection if we are running on
>> the VF
>> +device using dedicated VF_CAP(0x1901f8) register and then use global
>> macro
>> +IS_SRIOV_VF(xe) to control the driver logic.
>> +
>> +To speed up merging of the required changes, we might first introduce
>> dummy
>> +macro that is always set to false, to prepare driver to avoid some
>> code paths
>> +before we finalize our VF mode detection and other VFs enabling changes.
>> +
>> +
>> +Resources
>> +=========
>> +
>> +Most of the hardware (or firmware) resources available on the Xe
>> architecture,
>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
>> between PF and
>> +VFs and will require some provisioning steps to assign those
>> resources for use
>> +by the VF.
>> +
>> +Until VFs are provisioned with resources, the PF driver will be able
>> to use all
>> +resources, in the same way as it would be running in non-virtualized
>> mode.
>> +
>> +If some resource (of part or region of it) is assigned to specific
>> VF, then PF
>> +is not allowed to use that part or region of the resource, but can
>> continue to
>> +use whatever is left available.
>> +
>> +Those resources are usually fully virtualized, so they will not
>> require any
>> +special handling when used by the VF driver, except that VF driver
>> must know
>> +the assigned quota.
>> +
>> +The most notable exception is the GGTT address space, as on some
>> platforms,
>> +the VF driver must additionally know the real range that it can access.
>> +
>> +Once the resources were assigned to the VF use and the VF driver has
>> started,
>> +then it is not allowed to change such provisioning, as that would
>> break the
>> +VF driver. To make changes the VF driver, which was using these
>> resources,
>> +must be unloaded (or the VM is terminated) and the VF device must be
>> reset
>> +using the FLR.
>> +
>> +
>> +Scheduling
>> +==========
>> +
>> +The workloads from PF driver and VF drivers must be submitted to the
>> hardware
>> +always by using the GuC submission mechanism. Unless VF has exclusive
>> access
>> +to the GT then submissions from different VFs are time-sliced and
>> controlled
>> +with additional "execution_quantum" and "preemption_timeout" parameters.
>> +
>> +In contrast to the resource provisioning, those scheduling parameters
>> can be
>> +changed even if VF drivers are already running and are active.
>> +
>> +
>> +Automatic VFs Provisioning
>> +==========================
>> +
>> +To provide out-of-the box experience when user will be enabling VFs
>> using
>> +generic "sriov_numvfs" attribute without requiring complex
>> provisioning steps,
>> +the SR-IOV PF driver will implement automatic VFs resource provisioning.
>> +
>> +By default, all VFs will be allocated with the fair amount of the
>> mandatory
>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling
>> parameters.
>> +Such provisioning should be sufficient for most of the normal usages,
>> when
>> +no strict SLA is required.
>> +
>> +The PF driver will also expose some additional sysfs files to allow
>> adjusting
>> +this automatic VFs provisioning, like default values for most of the
>> +provisioning parameters that PF will then apply for each enabled VF.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +Manual VFs Provisioning
>> +=======================
>> +
>> +If automatic VFs provisioning, which applies same configuration to
>> every VF,
>> +is not sufficient or there is a need for advanced customization of
>> some VF,
>> +the PF driver will also provide extended sysfs interface which will
>> allow
>> +control every provisioning attribute to the lowest feasible level.
>> +
>> +It is expected that these low-level attributes will be mostly used by
>> the
>> +advanced users or by the custom tools that will setup configurations
>> that
>> +meet predefined and validated SLA as required by the customers.
>> +
>> +    Details about those extension can be found in
>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>> +
>> +
>> +VFs Monitoring
>> +==============
>> +
>> +In addition to the resource provisioning or changing scheduling
>> parameters,
>> +the PF driver might also allow configure some monitoring parameters,
>> like
>> +thresholds of adverse events or sample period, to track undesired
>> behavior
>> +of the VFs that could impact the whole system.
>> +
>> +Once those thresholds are setup and sampling period is defined, the
>> GuC will
>> +notify the PF driver about which VF is excessing the threshold and
>> then PF is
>> +able to trigger the uevent to notify the administrator (or VMM) that
>> could
>> +take some action against the VF.

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
  2023-11-14 13:22     ` [Intel-xe] " Vivi, Rodrigo
@ 2023-11-14 16:59       ` Michal Wajdeczko
  -1 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 16:59 UTC (permalink / raw)
  To: Vivi, Rodrigo, intel-xe, dri-devel, tvrtko.ursulin; +Cc: ogabbay



On 14.11.2023 14:22, Vivi, Rodrigo wrote:
> On Tue, 2023-11-14 at 12:37 +0000, Tvrtko Ursulin wrote:
>>
>> On 10/11/2023 18:22, Michal Wajdeczko wrote:
>>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>>> Express (PCIe) specification suite is supported starting from 12th
>>> generation of Intel Graphics processors.
>>>
>>> This RFC aims to explain how do we want to add support for SR-IOV
>>> to the new Xe driver and to propose related additions to the sysfs.
>>>
>>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>>> Cc: Oded Gabbay <ogabbay@kernel.org>
>>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>>> Cc: Daniel Vetter <daniel@ffwll.ch>
>>> ---
>>>   Documentation/gpu/rfc/index.rst             |   5 +
>>>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501
>>> ++++++++++++++++++++
>>>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>>   3 files changed, 698 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/index.rst
>>> b/Documentation/gpu/rfc/index.rst
>>> index e4f7b005138d..fc5bc447f30d 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -35,3 +35,8 @@ host such documentation:
>>>   .. toctree::
>>>   
>>>      xe.rst
>>> +
>>> +.. toctree::
>>> +   :maxdepth: 1
>>> +
>>> +   xe_sriov.rst
>>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> new file mode 100644
>>> index 000000000000..77748204dd83
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> @@ -0,0 +1,501 @@
>>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>>> +..
>>> +.. Intel Xe driver ABI (SR-IOV extensions)
>>> +..
>>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>>> +    the PCI Express (PCIe) specification suite is supported
>>> +    starting from 12th generation of Intel Graphics processors.
>>> +
>>> +    This document describes Xe driver specific additions.
>>> +
>>> +    For description of generic SR-IOV sysfs attributes see
>>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_auto_provisioning
>>> +    │   ├── admin_mode
>>> +    │   ├── enabled
>>> +    │   ├── reset_defaults
>>> +    │   ├── resources
>>> +    │   │   ├── default_contexts_quota
>>> +    │   │   ├── default_doorbells_quota
>>> +    │   │   ├── default_ggtt_quota
>>> +    │   │   └── default_lmem_quota
>>> +    │   ├── scheduling
>>> +    │   │   ├── default_exec_quantum_ms
>>> +    │   │   └── default_preempt_timeout_us
>>> +    │   └── monitoring
>>> +    │       ├── default_cat_error_count
>>> +    │       ├── default_doorbell_time_us
>>> +    │       ├── default_engine_reset_count
>>> +    │       ├── default_h2g_time_us
>>> +    │       ├── default_irq_time_us
>>> +    │       └── default_page_fault_count
>>
>>  From the department of bike-shedding, one alternative could be to
>> have 
>> a directory called defaults which avoids having to have the default_ 
>> prefix on everything under it.
> 
> good idea. probably with a '.' prefix to make it hidden like we have
> for other stuff already.
> 
> '.defaults'

but then we will be inconsistent as in this 'other stuff' we use
".defaults" directory to hold RO attributes with min/default/max values,
while here we wanted to define RW attributes that will be applied to VFs

maybe ".template" instead ?

> 
>>
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_extensions
>>
>> Should this be xe_sriov_extensions or if not doesn't it need
>> agreement 
>> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
>> Sriov_auto_provisioning too I guess.
>>
>>> +    │   ├── monitoring_period_ms
>>> +    │   ├── strict_scheduling_enabled
>>> +    │   ├── pf
>>> +    │   │   ├── device -> ../../../BDF
>>> +    │   │   ├── priority
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├── engine_reset_count
>>> +    │   │   │   │       ├── h2g_time_us
>>> +    │   │   │   │       ├── irq_time_us
>>> +    │   │   │   │       └── page_fault_count
>>> +    │   │   │   └── gtX
>>> +    │   │   └── tileT
>>> +    │   ├── vf1
>>> +    │   │   ├── device -> ../../../BDF+1
>>> +    │   │   ├── stop
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── ggtt_quota
>>> +    │   │   │   ├── lmem_quota
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── contexts_quota
>>> +    │   │   │   │   ├── doorbells_quota
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├── engine_reset_count
>>> +    │   │   │   │       ├── h2g_time_us
>>> +    │   │   │   │       ├── irq_time_us
>>> +    │   │   │   │       └── page_fault_count
>>> +    │   │   │   └── gtX
>>> +    │   │   └── tileT
>>> +    │   └── vfN
>>> +..
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory appears on the device when:
>>> +
>>> +                - device supports SR-IOV, and
>>> +                - device is a Physical Function (PF), and
>>> +                - xe driver supports SR-IOV PF on given device,
>>> and
>>> +                - xe driver supports automatic VFs provisioning.
>>> +
>>> +               This directory is used as a root for all attributes
>>> related to
>>> +               automatic provisioning of SR-IOV Physical Function
>>> (PF) and/or
>>> +               Virtual Functions (VFs).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /enabled
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (RW) bool (0, 1)
>>> +
>>> +               This file represents configuration flag for the
>>> automatic VFs
>>> +               (un)provisioning that could be performed by the PF.
>>> +
>>> +               The default value is 1 (true).
>>> +
>>> +               This flag can be set to false, unless manual
>>> provisioning is not
>>> +               applicable for given platform or it is not
>>> supported by current
>>> +               PF implementation. In such cases -EPERM will be
>>> returned.
>>> +
>>> +               This flag will be automatically set to false when
>>> there will be
>>> +               other attempts to change any of VF's resource
>>> provisioning.
>>> +               See "sriov_extensions" section for details.
>>> +
>>> +               This flag can be set back to true if and only if
>>> all VFs are
>>> +               fully unprovisioned, otherwise -EEXIST error will
>>> be returned.
>>> +
>>> +               false = "disabled"
>>> +                       When disabled, then PF will not attempt to
>>> do automatic
>>> +                       VFs provisioning when VFs are being enabled
>>> and will not
>>> +                       perform automatic unprovisioning of the VFs
>>> when VFs will
>>> +                       be disabled.
>>> +
>>> +               true = "enabled"
>>> +                       When enabled, then on VFs enabling PF will
>>> do automatic
>>> +                       VFs provisioning based on the default
>>> settings described
>>> +                       below.
>>> +
>>> +                       If automatic VFs provisioning fails due to
>>> some reasons,
>>> +                       then VFs will not be enabled.
>>> +
>>> +                       If enabled, all resources allocated during
>>> VFs enabling
>>> +                       will be released during VFs disabling
>>> (automatic unprovisioning).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /admin_mode
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (RW) bool (0, 1)
>>> +
>>> +               This file represents configuration flag for the
>>> automatic VFs
>>> +               provisioning that could be performed by the PF.
>>> +
>>> +               The default value depends on the platform type.
>>> +
>>> +               This flag can be changed any time, but will have no
>>> effect if
>>> +               VFs are already provisioned.
>>> +
>>> +               If enabled (default on discrete platforms) then the
>>> PF will
>>> +               retain only minimum hardcoded resources for its own
>>> use when
>>> +               doing VFs automatic provisioning and will not use
>>> any default
>>> +               values described below for its own configuration.
>>> +
>>> +               If disabled (default on integrated platforms) then
>>> the PF will
>>> +               treat itself like yet another additional VF in all
>>> fair resource
>>> +               allocations and will also try to apply default
>>> provisioning
>>> +               values described below for its own configuration.
>>> +
>>
>> One alternative could be to expose two sets of defaults, the PF and
>> VF 
>> ones. With the advantage of allowing the "admin mode" / "minimal PF"
>> to 
>> be explicitly configurable instead of hardcoded. Should be more
>> flexible.
>>
>> If the discrete vs integrated distinction is wanted it could simply
>> be 
>> made by initialy populating (driver init) the respective defaults
>> based 
>> on the platform type.
>>
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /reset_defaults
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (WO) bool (1)
>>> +
>>> +               Writing to this file will reset all default
>>> provisioning parameters
>>> +               listed below to the default values.
>>
>> Maybe this isn't required if you can say it is the responsibility of 
>> whoever changes the defaults to either know what they are doing, or
>> to 
>> save and restore themselves if they. It is not a major concern but if
>> writing kernel code can be avoided perhaps it can be considered.
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_contexts_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_doorbells_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_ggtt_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /resources/default_lmem_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /scheduling/default_exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /scheduling/default_preempt_timeout_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_auto_provisioning
>>> /monitoring/default_page_fault_count
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               These files represent default provisioning that
>>> should be used
>>> +               for VFs automatic provisioning.
>>> +
>>> +               These values can be changed any time, but will have
>>> no effect if
>>> +               VFs are already provisioned.
>>> +
>>> +               default_contexts_quota: (RW) integer 0..U32_MAX
>>> +                       The number of GuC context IDs to provide to
>>> the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/contexts_quota" for details.
>>> +
>>> +               default_doorbells_quota: (RW) integer 0..U32_MAX
>>> +                       The number of GuC doorbells to provide to
>>> the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/doorbells_quota" for details.
>>> +
>>> +               default_ggtt_quota: (RW) integer 0..U32_MAX
>>> +                       The size of the GGTT address space (in
>>> bytes) to provide to the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See "sriov_extensions/vfN/tileT/ggtt_quota"
>>> for details.
>>> +
>>> +               default_lmem_quota: (RW) integer 0..U32_MAX
>>> +                       The size of the LMEM (in bytes) to provide
>>> to the VF.
>>> +                       The default value is 0 (use fair
>>> allocations).
>>> +                       See "sriov_extensions/vfN/tileT/lmem_quota"
>>> for details.
>>> +
>>> +               default_exec_quantum_ms: (RW) integer 0..U32_MAX
>>> +                       The GT execution quantum (in millisecs)
>>> assigned to the function.
>>> +                       The default value is 0 (infinify).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/exec_quantum_ms" for details.
>>> +
>>> +               default_preempt_timeout_us: (RW) integer 0..U32_MAX
>>> +                       The GT preemption timeout (in microsecs)
>>> assigned to the function.
>>> +                       The default value is 0 (infinity).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/preempt_timeout_us" for details.
>>
>> I have a slight concern here on the usability of GuC specific
>> tunables.
>>
>> Whereas one can imagine an external entity (some admin, somewhere) to
>> probably pretty much understand what it means to partition the local 
>> memory, address space, and set the scheduling timeouts (all are
>> intuitve 
>> and obvious concepts), how are they suppose to approach the GuC 
>> doorbells and contexts?
>>
>> It could be a matter of adding more documentation for those two, or
>> it 
>> even could make sense to shove them under a guc prefix (or 
>> subdirectory?) to signify the fact they are implementation details
>> and 
>> not a fundamental concept.
>>
>>> +
>>> +               default_cat_error_count: (RW) integer 0..U32_MAX
>>> +               default_doorbell_time_us: (RW) integer 0..U32_MAX
>>> +               default_engine_reset_count: (RW) integer 0..U32_MAX
>>> +               default_h2g_time_us: (RW) integer 0..U32_MAX
>>> +               default_irq_time_us: (RW) integer 0..U32_MAX
>>> +               default_page_fault_count: (RW) integer 0..U32_MAX
>>> +                       The monitoring threshold to be set for the
>>> function.
>>> +                       The default value is 0 (don't monitor).
>>> +                       See
>>> "sriov_extensions/vfN/tileT/gtX/thresholds" for details.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory appears on Xe device when:
>>> +
>>> +                - device supports SR-IOV, and
>>> +                - device is a Physical Function (PF), and
>>> +                - driver is enabled to support SR-IOV PF on given
>>> device.
>>> +
>>> +               This directory is used as a root for all attributes
>>> required to
>>> +               manage both Physical Function (PF) and Virtual
>>> Functions (VFs).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/strict
>>> _scheduling_enabled
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (RW) bool
>>> +
>>> +               This file represents a flag used to determine if
>>> scheduling
>>> +               parameters should be respected even if there is no
>>> active
>>> +               workloads submitted by the PF or VFs.
>>> +
>>> +               This flag is disabled by default, unless strict
>>> scheduling is
>>> +               not applicable on given platform. In such case this
>>> file will
>>> +               be read-only.
>>> +
>>> +               The change to this file may have no effect if VFs
>>> are not yet enabled.
>>> +               If strict scheduling can't be enabled in GuC then
>>> write will fail with -EIO.
>>
>> I think the semantics of this need to be documented ie. how it
>> interacts 
>> with exec_quantum_ms. If it does? I am guessing that it has to
>> otherwise 
>> I don't know what it would mean - presumably unused timeslices are
>> not 
>> given to other entities but time just goes wasted? But it is also a 
>> question on over what time interval. Or that too is purely defined by
>> the number of PF+VFs and their respective allocated quanta.
>>
>> Also, would there be benefit, assuming it is possible with GuC, to
>> allow 
>> configuring it per PF/VF?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/monito
>>> ring_period_ms
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (RW) integer
>>> +
>>> +               This file represents the configuration knob used by
>>> adverse event
>>> +               monitoring. A value here is the period in millisecs
>>> during which
>>> +               events are counted and the total is checked against
>>> a threshold.
>>> +               See "sriov_extensions/vfN/tileT/gtX/thresholds" for
>>> more details.
>>> +
>>> +               Default is 0 (monitoring is disabled).
>>> +
>>> +               If monitoring capability is not available, then
>>> attempt to enable
>>> +               will fail with -EPERM error. If monitoring can't be
>>> enabled in
>>> +               GuC then write will fail with -EIO.
>>
>> Could the docs explain if there is a downside to enabling it, which
>> is 
>> probably why it isn't enabled by default? Because it does sound
>> natural 
>> that adverse events should be noticed.
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory holds all attributes related to the
>>> SR-IOV
>>> +               Physical Function (PF).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory holds all attributes related to the
>>> SR-IOV
>>> +               Virtual Function (VF).
>>> +
>>> +               Note that VF numbers (N) are 1-based as described
>>> in PCI SR-IOV specification.
>>> +               The Xe driver implementaton follows that naming
>>> schema.
>>> +
>>> +               There will be "vf1", "vf2" up to "vfN" directories,
>>> where N matches
>>> +               value of the PCI "sriov_totalvfs" attribute.
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory holds all SR-IOV attributes related
>>> to the device tile.
>>> +               The tile numbers (T) start from 0.
>>> +
>>> +               There is at least one "tile0/" directory present.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               This directory holds all SR-IOV attributes related
>>> to the device GT.
>>> +               The GT numbers (X) start from 0.
>>> +
>>> +               There is at least one "gt0/" directory present.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/dev
>>> ice
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/de
>>> vice
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (symbolic link)
>>> +
>>> +               Backlink to the PCI device entry representing given
>>> function.
>>> +               For PF this link is always present.
>>> +               For VF this link is present only for currently
>>> enabled VFs.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/pri
>>> ority
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (RW) string
>>> +
>>> +               This file represents a GuC Scheduler knob to
>>> override the default
>>> +               round-robin or FIFO scheduler policies implemented
>>> by the GuC.
>>> +
>>> +               The default value is "peer".
>>> +
>>> +               This flag can be changed, unless such change is not
>>> applicable
>>> +               for given platform or is not supported by current
>>> GuC firmware.
>>> +               In such case this file could be read-only or will
>>> return -EPERM
>>> +               on write attempt.
>>> +
>>> +               "immediate"
>>> +                       GuC will Schedule PF workloads immediately
>>> and PF
>>> +                       workloads only until the PF's work queues
>>> in GuC
>>> +                       are empty.
>>> +
>>> +               "lazy"
>>> +                       GuC will Schedule PF workloads at the next
>>> opportune
>>> +                       moment and PF workloads only until the PF
>>> work queues
>>> +                       in GuC are empty.
>>> +
>>> +               "peer"
>>> +                       GuC Scheduler will treat PF and VFs with
>>> equal priority.
>>
>> Hmmm this is too very GuC specific and I wonder what is the usecase
>> for 
>> lazy? Lazy = "don't care when it runs, but when it runs it will run 
>> everything queued so far", right? Feels a bit odd on first.
>>
>> "Immediate" may also not be depending on preemption granularity and 
>> workloads, right?
>>
>> Are there any ideas to express the knobs in a more generic fashion?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/st
>>> op
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               (WO) bool (1)
>>> +
>>> +               Write to this file will force GuC to stop handle
>>> any requests from
>>> +               this VF, but without triggering a FLR.
>>> +               To recover, the full FLR must be issued using
>>> generic "device/reset".
>>> +
>>> +               This file allows to implement custom policy
>>> mechanism when VF is
>>> +               misbehaving and triggering adverse events above
>>> defined thresholds.
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/preempt_timeout_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/exec_quantum_ms
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/preempt_timeout_us
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               These files represent scheduling parameters of the
>>> functions.
>>> +
>>> +               These scheduling parameters can be changed even if
>>> VFs are enabled
>>> +               and running, unless such change is not applicable
>>> on given platform
>>> +               due to fixed hardware or firmware assignment.
>>> +
>>> +               exec_quantum_ms: (RW) integer 0..U32_MAX
>>> +                       The GT execution quantum in [ms] assigned
>>> to the function.
>>> +                       Requested quantum might be aligned per
>>> HW/FW requirements.
>>> +
>>> +                       Default is 0 (unlimited).
>>> +
>>> +               preempt_timeout_us: (RW) integer 0..U32_MAX
>>> +                       The GT preemption timeout in [us] assigned
>>> to the function.
>>> +                       Requested timeout might be aligned per
>>> HW/FW requirements.
>>> +
>>> +                       Default is 0 (unlimited).
>>
>> Alignment for the above two will be visible after read-back?
>>
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/ggtt_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/lmem_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/contexts_quota
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/doorbells_quota
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               These files represent shared resource assigned to
>>> the functions.
>>> +
>>> +               These resource parameters can be changed, unless VF
>>> is already running,
>>> +               or such change is not applicable on given platform
>>> due to fixed hardware
>>> +               or firmware assignment.
>>> +
>>> +               Writes to these attributes may fail with:
>>> +                       -EPERM if change is not applicable on give
>>> HW/FW.
>>> +                       -E2BIG if value larger that HW/FW limit.
>>> +                       -EDQUOT if value is larger than maximum
>>> quota defined by the PF.
>>> +                       -ENOSPC if PF can't allocate required
>>> quota.
>>> +                       -EBUSY if the resource is currently in use
>>> by the VF.
>>> +                       -EIO if GuC refuses to change provisioning.
>>
>> Why it would refuse if input is valid? In other words, what is the 
>> user/admin supposed to do on -EIO?
>>
>>> +
>>> +               ggtt_quota: (RW) integer 0..U64_MAX
>>> +                       The size of the GGTT address space (in
>>> bytes) assigned to the VF.
>>> +                       The value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               lmem_quota: (RW) integer 0..U64_MAX
>>> +                       The size of the Local Memory (in bytes)
>>> assigned to the VF.
>>> +                       The value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       This attribute is only available on
>>> discrete platforms.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               contexts_quota: (RW) 0..U16_MAX
>>> +                       The number of GuC submission contexts
>>> assigned to the VF.
>>> +                       This value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +               doorbells_quota: (RW) 0..U16_MAX
>>> +                       The number of GuC doorbells assigned to the
>>> VF.
>>> +                       This value might be aligned per HW/FW
>>> requirements.
>>> +
>>> +                       Default is 0 (unprovisioned).
>>> +
>>> +
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/pf/til
>>> eT/gtX/thresholds/page_fault_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/cat_error_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/doorbell_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/engine_reset_count
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/h2g_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/irq_time_us
>>> +What:          /sys/bus/pci/drivers/xe/.../sriov_extensions/vfN/ti
>>> leT/gtX/thresholds/page_fault_count
>>> +Date:          2024
>>> +KernelVersion: TBD
>>> +Contact:       intel-xe@lists.freedesktop.org
>>> +Description:
>>> +               These files represent threshold values used by the
>>> GuC to trigger
>>> +               security events if adverse event monitoring is
>>> enabled.
>>
>> How are the security events delivered? There is mention of uevents in
>> a 
>> later paragraph - are they already defined or should be together with
>> this so the link can be place here?
>>
>>> +
>>> +               These thresholds are checked every
>>> "monitoring_period_ms".
>>> +               Refer to GuC ABI for details about each threshold
>>> category.
>>
>> Is it possible to have a link here to GuC ABI?
>>
>> Regards,
>>
>> Tvrtko
>>
>>> +
>>> +               Default value for all thresholds is 0 (disabled).
>>> +
>>> +               cat_error_count: (RW) integer
>>> +               doorbell_time_us: (RW) integer
>>> +               engine_reset_count: (RW) integer
>>> +               h2g_time_us: (RW) integer
>>> +               irq_time_us: (RW) integer
>>> +               page_fault_count: (RW) integer
>>> diff --git a/Documentation/gpu/rfc/xe_sriov.rst
>>> b/Documentation/gpu/rfc/xe_sriov.rst
>>> new file mode 100644
>>> index 000000000000..574f6414eabb
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/xe_sriov.rst
>>> @@ -0,0 +1,192 @@
>>> +.. SPDX-License-Identifier: MIT
>>> +
>>> +========================
>>> +Xe – SR-IOV Support Plan
>>> +========================
>>> +
>>> +The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>>> Express (PCIe)
>>> +specification suite is supported starting from 12th generation of
>>> Intel Graphics
>>> +processors.
>>> +
>>> +This document describes planned ABI of the new Xe driver (see
>>> xe.rst) that will
>>> +provide flexible configuration and management options related to
>>> the SR-IOV.
>>> +It will also highlight few most important changes to the Xe driver
>>> +implementation to deal with Intel GPU SR-IOV specific
>>> requirements.
>>> +
>>> +
>>> +SR-IOV Capability
>>> +=================
>>> +
>>> +Due to SR-IOV complexity and required co-operation between
>>> hardware, firmware
>>> +and kernel drivers, not all Xe architecture platforms might have
>>> SR-IOV enabled
>>> +or fully functional.
>>> +
>>> +To control at the driver level which platform will provide support
>>> for SR-IOV,
>>> +as we can't just rely on the PCI configuration data exposed by the
>>> hardware,
>>> +we will introduce "has_sriov" flag to the struct xe_device_desc
>>> that describes
>>> +a device capabilities that driver checks during the probe.
>>> +
>>> +Initially this flag will be set to disabled even on platforms that
>>> we plan to
>>> +support. We will enable this flag only once we finish merging all
>>> required
>>> +changes to the driver and related validated firmwares are also
>>> made available.
>>> +
>>> +
>>> +SR-IOV Platforms
>>> +================
>>> +
>>> +Initially we plan to add SR-IOV functionality to the following SDV
>>> platforms
>>> +already supported by the Xe driver:
>>> +
>>> + - TGL (up to 7 VFs)
>>> + - ADL (up to 7 VFs)
>>> + - MTL (up to 7 VFs)
>>> + - ATSM (up to 31 VFs)
>>> + - PVC (up to 63 VFs)
>>> +
>>> +Newer platforms will be supported later, but we hope that enabling
>>> will be
>>> +much faster, as majority of the driver changes are either platform
>>> agnostic
>>> +or are similar between earlier platforms (hence we start with
>>> SDVs).
>>> +
>>> +
>>> +PF Mode
>>> +=======
>>> +
>>> +Support in the driver for acting in Physical Function (PF) mode,
>>> i.e. mode
>>> +that allows configuration of VFs, depends on the CONFIG_PCI_IOV
>>> and will be
>>> +enabled by default.
>>> +
>>> +However, due to potentially conflicting requirements for SR-IOV
>>> and other mega
>>> +features, we might want to have an option to disable SR-IOV PF
>>> mode support at
>>> +the driver load time.
>>> +
>>> +Thus, we plan to use additional modparam named "sriov_totalvfs"
>>> which if set to
>>> +0 will force the driver to operate in the native (non-virtualized)
>>> mode.
>>> +The same modparam could be used to limit number of supported
>>> Virtual Functions
>>> +(VFs) by the driver compared to the hardware limit exposed in PCI
>>> configuration.
>>> +
>>> +The name of this modparam corresponds to the existing PCI sysfs
>>> attribute, that
>>> +by default exposes hardware capability.
>>> +
>>> +The default value of this param will allow to support all possible
>>> VFs as
>>> +claimed by the hardware.
>>> +
>>> +This modparam will have no effect if driver is running on the VF
>>> device.
>>> +
>>> +
>>> +VFs Enabling
>>> +============
>>> +
>>> +To enable or disable VFs we plan to rely on existing sysfs
>>> attribute exposed by
>>> +the PCI subsystem named "sriov_numvfs". We will provide all
>>> necessary tweaks to
>>> +provision VFs in our custom implementation of the
>>> "sriov_configure" hook from
>>> +the struct pci_driver.
>>> +
>>> +If for some reason, including explicit request to disable SR-IOV
>>> PF mode using
>>> +modparam, we will not be able to correctly support any VFs, driver
>>> will change
>>> +number of supported VFs, exposed to the userspace by
>>> "sriov_totalvfs" attribute,
>>> +to 0, thus preventing configuration of the VFs.
>>> +
>>> +
>>> +VF Mode
>>> +=======
>>> +
>>> +When driver is running on the VF device, then due to hardware
>>> enforcements,
>>> +access to the privileged registers is not possible. To avoid
>>> relying on these
>>> +registers, we plan to perform early detection if we are running on
>>> the VF
>>> +device using dedicated VF_CAP(0x1901f8) register and then use
>>> global macro
>>> +IS_SRIOV_VF(xe) to control the driver logic.
>>> +
>>> +To speed up merging of the required changes, we might first
>>> introduce dummy
>>> +macro that is always set to false, to prepare driver to avoid some
>>> code paths
>>> +before we finalize our VF mode detection and other VFs enabling
>>> changes.
>>> +
>>> +
>>> +Resources
>>> +=========
>>> +
>>> +Most of the hardware (or firmware) resources available on the Xe
>>> architecture,
>>> +like GGTT, LMEM, GuC context IDs, GuC doorbells, will be shared
>>> between PF and
>>> +VFs and will require some provisioning steps to assign those
>>> resources for use
>>> +by the VF.
>>> +
>>> +Until VFs are provisioned with resources, the PF driver will be
>>> able to use all
>>> +resources, in the same way as it would be running in non-
>>> virtualized mode.
>>> +
>>> +If some resource (of part or region of it) is assigned to specific
>>> VF, then PF
>>> +is not allowed to use that part or region of the resource, but can
>>> continue to
>>> +use whatever is left available.
>>> +
>>> +Those resources are usually fully virtualized, so they will not
>>> require any
>>> +special handling when used by the VF driver, except that VF driver
>>> must know
>>> +the assigned quota.
>>> +
>>> +The most notable exception is the GGTT address space, as on some
>>> platforms,
>>> +the VF driver must additionally know the real range that it can
>>> access.
>>> +
>>> +Once the resources were assigned to the VF use and the VF driver
>>> has started,
>>> +then it is not allowed to change such provisioning, as that would
>>> break the
>>> +VF driver. To make changes the VF driver, which was using these
>>> resources,
>>> +must be unloaded (or the VM is terminated) and the VF device must
>>> be reset
>>> +using the FLR.
>>> +
>>> +
>>> +Scheduling
>>> +==========
>>> +
>>> +The workloads from PF driver and VF drivers must be submitted to
>>> the hardware
>>> +always by using the GuC submission mechanism. Unless VF has
>>> exclusive access
>>> +to the GT then submissions from different VFs are time-sliced and
>>> controlled
>>> +with additional "execution_quantum" and "preemption_timeout"
>>> parameters.
>>> +
>>> +In contrast to the resource provisioning, those scheduling
>>> parameters can be
>>> +changed even if VF drivers are already running and are active.
>>> +
>>> +
>>> +Automatic VFs Provisioning
>>> +==========================
>>> +
>>> +To provide out-of-the box experience when user will be enabling
>>> VFs using
>>> +generic "sriov_numvfs" attribute without requiring complex
>>> provisioning steps,
>>> +the SR-IOV PF driver will implement automatic VFs resource
>>> provisioning.
>>> +
>>> +By default, all VFs will be allocated with the fair amount of the
>>> mandatory
>>> +resources (like GGTT, GuC IDs) and with unrestricted scheduling
>>> parameters.
>>> +Such provisioning should be sufficient for most of the normal
>>> usages, when
>>> +no strict SLA is required.
>>> +
>>> +The PF driver will also expose some additional sysfs files to
>>> allow adjusting
>>> +this automatic VFs provisioning, like default values for most of
>>> the
>>> +provisioning parameters that PF will then apply for each enabled
>>> VF.
>>> +
>>> +    Details about those extension can be found in
>>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>>> +
>>> +
>>> +Manual VFs Provisioning
>>> +=======================
>>> +
>>> +If automatic VFs provisioning, which applies same configuration to
>>> every VF,
>>> +is not sufficient or there is a need for advanced customization of
>>> some VF,
>>> +the PF driver will also provide extended sysfs interface which
>>> will allow
>>> +control every provisioning attribute to the lowest feasible level.
>>> +
>>> +It is expected that these low-level attributes will be mostly used
>>> by the
>>> +advanced users or by the custom tools that will setup
>>> configurations that
>>> +meet predefined and validated SLA as required by the customers.
>>> +
>>> +    Details about those extension can be found in
>>> +    :download:`Preliminary Xe driver ABI <sysfs-driver-xe-sriov>`.
>>> +
>>> +
>>> +VFs Monitoring
>>> +==============
>>> +
>>> +In addition to the resource provisioning or changing scheduling
>>> parameters,
>>> +the PF driver might also allow configure some monitoring
>>> parameters, like
>>> +thresholds of adverse events or sample period, to track undesired
>>> behavior
>>> +of the VFs that could impact the whole system.
>>> +
>>> +Once those thresholds are setup and sampling period is defined,
>>> the GuC will
>>> +notify the PF driver about which VF is excessing the threshold and
>>> then PF is
>>> +able to trigger the uevent to notify the administrator (or VMM)
>>> that could
>>> +take some action against the VF.
> 

^ permalink raw reply	[flat|nested] 30+ messages in thread

* Re: [Intel-xe] [PATCH] drm/doc/rfc: SR-IOV support on the new Xe driver
@ 2023-11-14 16:59       ` Michal Wajdeczko
  0 siblings, 0 replies; 30+ messages in thread
From: Michal Wajdeczko @ 2023-11-14 16:59 UTC (permalink / raw)
  To: Vivi, Rodrigo, intel-xe, dri-devel, tvrtko.ursulin; +Cc: daniel



On 14.11.2023 14:22, Vivi, Rodrigo wrote:
> On Tue, 2023-11-14 at 12:37 +0000, Tvrtko Ursulin wrote:
>>
>> On 10/11/2023 18:22, Michal Wajdeczko wrote:
>>> The Single Root I/O Virtualization (SR-IOV) extension to the PCI
>>> Express (PCIe) specification suite is supported starting from 12th
>>> generation of Intel Graphics processors.
>>>
>>> This RFC aims to explain how do we want to add support for SR-IOV
>>> to the new Xe driver and to propose related additions to the sysfs.
>>>
>>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
>>> Cc: Oded Gabbay <ogabbay@kernel.org>
>>> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
>>> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
>>> Cc: Daniel Vetter <daniel@ffwll.ch>
>>> ---
>>>   Documentation/gpu/rfc/index.rst             |   5 +
>>>   Documentation/gpu/rfc/sysfs-driver-xe-sriov | 501
>>> ++++++++++++++++++++
>>>   Documentation/gpu/rfc/xe_sriov.rst          | 192 ++++++++
>>>   3 files changed, 698 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>>   create mode 100644 Documentation/gpu/rfc/xe_sriov.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/index.rst
>>> b/Documentation/gpu/rfc/index.rst
>>> index e4f7b005138d..fc5bc447f30d 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -35,3 +35,8 @@ host such documentation:
>>>   .. toctree::
>>>   
>>>      xe.rst
>>> +
>>> +.. toctree::
>>> +   :maxdepth: 1
>>> +
>>> +   xe_sriov.rst
>>> diff --git a/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> new file mode 100644
>>> index 000000000000..77748204dd83
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/sysfs-driver-xe-sriov
>>> @@ -0,0 +1,501 @@
>>> +.. Documentation/ABI/testing/sysfs-driver-xe-sriov
>>> +..
>>> +.. Intel Xe driver ABI (SR-IOV extensions)
>>> +..
>>> +    The Single Root I/O Virtualization (SR-IOV) extension to
>>> +    the PCI Express (PCIe) specification suite is supported
>>> +    starting from 12th generation of Intel Graphics processors.
>>> +
>>> +    This document describes Xe driver specific additions.
>>> +
>>> +    For description of generic SR-IOV sysfs attributes see
>>> +    "Documentation/ABI/testing/sysfs-bus-pci" document.
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_auto_provisioning
>>> +    │   ├── admin_mode
>>> +    │   ├── enabled
>>> +    │   ├── reset_defaults
>>> +    │   ├── resources
>>> +    │   │   ├── default_contexts_quota
>>> +    │   │   ├── default_doorbells_quota
>>> +    │   │   ├── default_ggtt_quota
>>> +    │   │   └── default_lmem_quota
>>> +    │   ├── scheduling
>>> +    │   │   ├── default_exec_quantum_ms
>>> +    │   │   └── default_preempt_timeout_us
>>> +    │   └── monitoring
>>> +    │       ├── default_cat_error_count
>>> +    │       ├── default_doorbell_time_us
>>> +    │       ├── default_engine_reset_count
>>> +    │       ├── default_h2g_time_us
>>> +    │       ├── default_irq_time_us
>>> +    │       └── default_page_fault_count
>>
>>  From the department of bike-shedding, one alternative could be to
>> have 
>> a directory called defaults which avoids having to have the default_ 
>> prefix on everything under it.
> 
> good idea. probably with a '.' prefix to make it hidden like we have
> for other stuff already.
> 
> '.defaults'

but then we will be inconsistent as in this 'other stuff' we use
".defaults" directory to hold RO attributes with min/default/max values,
while here we wanted to define RW attributes that will be applied to VFs

maybe ".template" instead ?

> 
>>
>>> +
>>> +    /sys/bus/pci/drivers/xe/BDF/
>>> +    ├── sriov_extensions
>>
>> Should this be xe_sriov_extensions or if not doesn't it need
>> agreement 
>> to reserve the keyword in Documentation/ABI/testing/sysfs-bus-pci? 
>> Sriov_auto_provisioning too I guess.
>>
>>> +    │   ├── monitoring_period_ms
>>> +    │   ├── strict_scheduling_enabled
>>> +    │   ├── pf
>>> +    │   │   ├── device -> ../../../BDF
>>> +    │   │   ├── priority
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├── engine_reset_count
>>> +    │   │   │   │       ├── h2g_time_us
>>> +    │   │   │   │       ├── irq_time_us
>>> +    │   │   │   │       └── page_fault_count
>>> +    │   │   │   └── gtX
>>> +    │   │   └── tileT
>>> +    │   ├── vf1
>>> +    │   │   ├── device -> ../../../BDF+1
>>> +    │   │   ├── stop
>>> +    │   │   ├── tile0
>>> +    │   │   │   ├── ggtt_quota
>>> +    │   │   │   ├── lmem_quota
>>> +    │   │   │   ├── gt0
>>> +    │   │   │   │   ├── contexts_quota
>>> +    │   │   │   │   ├── doorbells_quota
>>> +    │   │   │   │   ├── exec_quantum_ms
>>> +    │   │   │   │   ├── preempt_timeout_us
>>> +    │   │   │   │   └── thresholds
>>> +    │   │   │   │       ├── cat_error_count
>>> +    │   │   │   │       ├── doorbell_time_us
>>> +    │   │   │   │       ├γ