* [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-22 20:06 ` Alex Deucher
0 siblings, 0 replies; 12+ messages in thread
From: Alex Deucher via iommu @ 2022-04-22 20:06 UTC (permalink / raw)
To: linux-doc, linux-kernel, corbet, hpa, x86, dave.hansen, bp,
mingo, tglx, joro, Suravee.Suthikulpanit, will, iommu,
robin.murphy, Vasant.Hegde
Cc: Alex Deucher
Add preliminary documentation for AMD IOMMU and combine
with the existing Intel IOMMU documentation and clean
up and modernize some of the existing documentation to
align with the current state of the kernel.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
a device) and document proper DMA API. Also correct the fact that
the AMD IOMMU is not limited to managing PCI devices.
v3: Fix spelling and rework text as suggested by Vasant
v4: Combine Intel and AMD documents into a single document as suggested
by Dave Hansen
v5: Clarify that keywords are related to ACPI, grammatical fixes
v6: Make more stuff common based on feedback from Robin
Documentation/x86/index.rst | 2 +-
Documentation/x86/intel-iommu.rst | 115 ------------------------
Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
3 files changed, 144 insertions(+), 116 deletions(-)
delete mode 100644 Documentation/x86/intel-iommu.rst
create mode 100644 Documentation/x86/iommu.rst
diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
index f498f1d36cd3..6f8409fe0674 100644
--- a/Documentation/x86/index.rst
+++ b/Documentation/x86/index.rst
@@ -21,7 +21,7 @@ x86-specific Documentation
tlb
mtrr
pat
- intel-iommu
+ iommu
intel_txt
amd-memory-encryption
pti
diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
deleted file mode 100644
index 099f13d51d5f..000000000000
--- a/Documentation/x86/intel-iommu.rst
+++ /dev/null
@@ -1,115 +0,0 @@
-===================
-Linux IOMMU Support
-===================
-
-The architecture spec can be obtained from the below location.
-
-http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
-
-This guide gives a quick cheat sheet for some basic understanding.
-
-Some Keywords
-
-- DMAR - DMA remapping
-- DRHD - DMA Remapping Hardware Unit Definition
-- RMRR - Reserved memory Region Reporting Structure
-- ZLR - Zero length reads from PCI devices
-- IOVA - IO Virtual address.
-
-Basic stuff
------------
-
-ACPI enumerates and lists the different DMA engines in the platform, and
-device scope relationships between PCI devices and which DMA engine controls
-them.
-
-What is RMRR?
--------------
-
-There are some devices the BIOS controls, for e.g USB devices to perform
-PS2 emulation. The regions of memory used for these devices are marked
-reserved in the e820 map. When we turn on DMA translation, DMA to those
-regions will fail. Hence BIOS uses RMRR to specify these regions along with
-devices that need to access these regions. OS is expected to setup
-unity mappings for these regions for these devices to access these regions.
-
-How is IOVA generated?
-----------------------
-
-Well behaved drivers call pci_map_*() calls before sending command to device
-that needs to perform DMA. Once DMA is completed and mapping is no longer
-required, device performs a pci_unmap_*() calls to unmap the region.
-
-The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
-device has its own domain (hence protection). Devices under p2p bridges
-share the virtual address with all devices under the p2p bridge due to
-transaction id aliasing for p2p bridges.
-
-IOVA generation is pretty generic. We used the same technique as vmalloc()
-but these are not global address spaces, but separate for each domain.
-Different DMA engines may support different number of domains.
-
-We also allocate guard pages with each mapping, so we can attempt to catch
-any overflow that might happen.
-
-
-Graphics Problems?
-------------------
-If you encounter issues with graphics devices, you can try adding
-option intel_iommu=igfx_off to turn off the integrated graphics engine.
-If this fixes anything, please ensure you file a bug reporting the problem.
-
-Some exceptions to IOVA
------------------------
-Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
-The same is true for peer to peer transactions. Hence we reserve the
-address from PCI MMIO ranges so they are not allocated for IOVA addresses.
-
-
-Fault reporting
----------------
-When errors are reported, the DMA engine signals via an interrupt. The fault
-reason and device that caused it with fault reason is printed on console.
-
-See below for sample.
-
-
-Boot Message Sample
--------------------
-
-Something like this gets printed indicating presence of DMAR tables
-in ACPI.
-
-ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
-
-When DMAR is being processed and initialized by ACPI, prints DMAR locations
-and any RMRR's processed::
-
- ACPI DMAR:Host address width 36
- ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
- ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
- ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
- ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
- ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
-
-When DMAR is enabled for use, you will notice..
-
-PCI-DMA: Using DMAR IOMMU
--------------------------
-
-Fault reporting
-^^^^^^^^^^^^^^^
-
-::
-
- DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
- DMAR:[fault reason 05] PTE Write access is not set
- DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
- DMAR:[fault reason 05] PTE Write access is not set
-
-TBD
-----
-
-- For compatibility testing, could use unity map domain for all devices, just
- provide a 1-1 for all useful memory under a single domain for all devices.
-- API for paravirt ops for abstracting functionality for VMM folks.
diff --git a/Documentation/x86/iommu.rst b/Documentation/x86/iommu.rst
new file mode 100644
index 000000000000..ed87d76a38d5
--- /dev/null
+++ b/Documentation/x86/iommu.rst
@@ -0,0 +1,143 @@
+=================
+x86 IOMMU Support
+=================
+
+The architecture specs can be obtained from the below locations.
+
+- Intel: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
+- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
+
+This guide gives a quick cheat sheet for some basic understanding.
+
+Basic stuff
+-----------
+
+ACPI enumerates and lists the different IOMMUs on the platform, and
+device scope relationships between devices and which IOMMU controls
+them.
+
+Some ACPI Keywords:
+
+- DMAR - Intel DMA Remapping table
+- DRHD - Intel DMA Remapping Hardware Unit Definition
+- RMRR - Intel Reserved Memory Region Reporting Structure
+- IVRS - AMD I/O Virtualization Reporting Structure
+- IVDB - AMD I/O Virtualization Definition Block
+- IVHD - AMD I/O Virtualization Hardware Definition
+
+What is Intel RMRR?
+^^^^^^^^^^^^^^^^^^^
+
+There are some devices the BIOS controls, for e.g USB devices to perform
+PS2 emulation. The regions of memory used for these devices are marked
+reserved in the e820 map. When we turn on DMA translation, DMA to those
+regions will fail. Hence BIOS uses RMRR to specify these regions along with
+devices that need to access these regions. OS is expected to setup
+unity mappings for these regions for these devices to access these regions.
+
+What is AMD IVRS?
+^^^^^^^^^^^^^^^^^
+
+The architecture defines an ACPI-compatible data structure called an I/O
+Virtualization Reporting Structure (IVRS) that is used to convey information
+related to I/O virtualization to system software. The IVRS describes the
+configuration and capabilities of the IOMMUs contained in the platform as
+well as information about the devices that each IOMMU virtualizes.
+
+The IVRS provides information about the following:
+
+- IOMMUs present in the platform including their capabilities and proper configuration
+- System I/O topology relevant to each IOMMU
+- Peripheral devices that cannot be otherwise enumerated
+- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are generally exclusion ranges to be configured by system software.
+
+How is an I/O Virtual Address (IOVA) generated?
+-----------------------------------------------
+
+Well behaved drivers call dma_map_*() calls before sending command to device
+that needs to perform DMA. Once DMA is completed and mapping is no longer
+required, driver performs dma_unmap_*() calls to unmap the region.
+
+Some exceptions to IOVAs
+------------------------
+
+Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
+The same is true for peer to peer transactions. Hence we reserve the
+address from PCI MMIO ranges so they are not allocated for IOVA addresses.
+
+Graphics Problems?
+------------------
+
+If you encounter issues with integrated graphics devices, you can try
+adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
+to the kernel command line to use a 1:1 mapping for the IOMMU in
+general. On Intel you can also try "intel_iommu=igfx_off" to turn off
+translation specifically for the integrated graphics engine only. If
+this fixes anything, please ensure you file a bug reporting the problem.
+
+Fault reporting
+---------------
+When errors are reported, the IOMMU signals via an interrupt. The fault
+reason and device that caused it is printed on the console.
+
+
+Kernel Log Samples
+------------------
+
+Intel Boot Messages
+^^^^^^^^^^^^^^^^^^^
+
+Something like this gets printed indicating presence of DMAR tables
+in ACPI:
+
+::
+
+ ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
+
+When DMAR is being processed and initialized by ACPI, prints DMAR locations
+and any RMRR's processed:
+
+::
+
+ ACPI DMAR:Host address width 36
+ ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
+ ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
+ ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
+ ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
+ ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
+
+When DMAR is enabled for use, you will notice:
+
+::
+
+ PCI-DMA: Using DMAR IOMMU
+
+Intel Fault reporting
+^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+ DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
+ DMAR:[fault reason 05] PTE Write access is not set
+ DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
+ DMAR:[fault reason 05] PTE Write access is not set
+
+AMD Boot Messages
+^^^^^^^^^^^^^^^^^
+
+Something like this gets printed indicating presence of the IOMMU:
+
+::
+
+ AMD-Vi: Found IOMMU cap 0x40
+ AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC GA_vAPIC
+ AMD-Vi: Interrupt remapping enabled
+ AMD-Vi: Virtual APIC enabled
+
+AMD Fault reporting
+^^^^^^^^^^^^^^^^^^^
+
+::
+
+ AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
+ AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
--
2.35.1
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-22 20:06 ` Alex Deucher
0 siblings, 0 replies; 12+ messages in thread
From: Alex Deucher @ 2022-04-22 20:06 UTC (permalink / raw)
To: linux-doc, linux-kernel, corbet, hpa, x86, dave.hansen, bp,
mingo, tglx, joro, Suravee.Suthikulpanit, will, iommu,
robin.murphy, Vasant.Hegde
Cc: Alex Deucher
Add preliminary documentation for AMD IOMMU and combine
with the existing Intel IOMMU documentation and clean
up and modernize some of the existing documentation to
align with the current state of the kernel.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
a device) and document proper DMA API. Also correct the fact that
the AMD IOMMU is not limited to managing PCI devices.
v3: Fix spelling and rework text as suggested by Vasant
v4: Combine Intel and AMD documents into a single document as suggested
by Dave Hansen
v5: Clarify that keywords are related to ACPI, grammatical fixes
v6: Make more stuff common based on feedback from Robin
Documentation/x86/index.rst | 2 +-
Documentation/x86/intel-iommu.rst | 115 ------------------------
Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
3 files changed, 144 insertions(+), 116 deletions(-)
delete mode 100644 Documentation/x86/intel-iommu.rst
create mode 100644 Documentation/x86/iommu.rst
diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
index f498f1d36cd3..6f8409fe0674 100644
--- a/Documentation/x86/index.rst
+++ b/Documentation/x86/index.rst
@@ -21,7 +21,7 @@ x86-specific Documentation
tlb
mtrr
pat
- intel-iommu
+ iommu
intel_txt
amd-memory-encryption
pti
diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
deleted file mode 100644
index 099f13d51d5f..000000000000
--- a/Documentation/x86/intel-iommu.rst
+++ /dev/null
@@ -1,115 +0,0 @@
-===================
-Linux IOMMU Support
-===================
-
-The architecture spec can be obtained from the below location.
-
-http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
-
-This guide gives a quick cheat sheet for some basic understanding.
-
-Some Keywords
-
-- DMAR - DMA remapping
-- DRHD - DMA Remapping Hardware Unit Definition
-- RMRR - Reserved memory Region Reporting Structure
-- ZLR - Zero length reads from PCI devices
-- IOVA - IO Virtual address.
-
-Basic stuff
------------
-
-ACPI enumerates and lists the different DMA engines in the platform, and
-device scope relationships between PCI devices and which DMA engine controls
-them.
-
-What is RMRR?
--------------
-
-There are some devices the BIOS controls, for e.g USB devices to perform
-PS2 emulation. The regions of memory used for these devices are marked
-reserved in the e820 map. When we turn on DMA translation, DMA to those
-regions will fail. Hence BIOS uses RMRR to specify these regions along with
-devices that need to access these regions. OS is expected to setup
-unity mappings for these regions for these devices to access these regions.
-
-How is IOVA generated?
-----------------------
-
-Well behaved drivers call pci_map_*() calls before sending command to device
-that needs to perform DMA. Once DMA is completed and mapping is no longer
-required, device performs a pci_unmap_*() calls to unmap the region.
-
-The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
-device has its own domain (hence protection). Devices under p2p bridges
-share the virtual address with all devices under the p2p bridge due to
-transaction id aliasing for p2p bridges.
-
-IOVA generation is pretty generic. We used the same technique as vmalloc()
-but these are not global address spaces, but separate for each domain.
-Different DMA engines may support different number of domains.
-
-We also allocate guard pages with each mapping, so we can attempt to catch
-any overflow that might happen.
-
-
-Graphics Problems?
-------------------
-If you encounter issues with graphics devices, you can try adding
-option intel_iommu=igfx_off to turn off the integrated graphics engine.
-If this fixes anything, please ensure you file a bug reporting the problem.
-
-Some exceptions to IOVA
------------------------
-Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
-The same is true for peer to peer transactions. Hence we reserve the
-address from PCI MMIO ranges so they are not allocated for IOVA addresses.
-
-
-Fault reporting
----------------
-When errors are reported, the DMA engine signals via an interrupt. The fault
-reason and device that caused it with fault reason is printed on console.
-
-See below for sample.
-
-
-Boot Message Sample
--------------------
-
-Something like this gets printed indicating presence of DMAR tables
-in ACPI.
-
-ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
-
-When DMAR is being processed and initialized by ACPI, prints DMAR locations
-and any RMRR's processed::
-
- ACPI DMAR:Host address width 36
- ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
- ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
- ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
- ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
- ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
-
-When DMAR is enabled for use, you will notice..
-
-PCI-DMA: Using DMAR IOMMU
--------------------------
-
-Fault reporting
-^^^^^^^^^^^^^^^
-
-::
-
- DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
- DMAR:[fault reason 05] PTE Write access is not set
- DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
- DMAR:[fault reason 05] PTE Write access is not set
-
-TBD
-----
-
-- For compatibility testing, could use unity map domain for all devices, just
- provide a 1-1 for all useful memory under a single domain for all devices.
-- API for paravirt ops for abstracting functionality for VMM folks.
diff --git a/Documentation/x86/iommu.rst b/Documentation/x86/iommu.rst
new file mode 100644
index 000000000000..ed87d76a38d5
--- /dev/null
+++ b/Documentation/x86/iommu.rst
@@ -0,0 +1,143 @@
+=================
+x86 IOMMU Support
+=================
+
+The architecture specs can be obtained from the below locations.
+
+- Intel: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
+- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
+
+This guide gives a quick cheat sheet for some basic understanding.
+
+Basic stuff
+-----------
+
+ACPI enumerates and lists the different IOMMUs on the platform, and
+device scope relationships between devices and which IOMMU controls
+them.
+
+Some ACPI Keywords:
+
+- DMAR - Intel DMA Remapping table
+- DRHD - Intel DMA Remapping Hardware Unit Definition
+- RMRR - Intel Reserved Memory Region Reporting Structure
+- IVRS - AMD I/O Virtualization Reporting Structure
+- IVDB - AMD I/O Virtualization Definition Block
+- IVHD - AMD I/O Virtualization Hardware Definition
+
+What is Intel RMRR?
+^^^^^^^^^^^^^^^^^^^
+
+There are some devices the BIOS controls, for e.g USB devices to perform
+PS2 emulation. The regions of memory used for these devices are marked
+reserved in the e820 map. When we turn on DMA translation, DMA to those
+regions will fail. Hence BIOS uses RMRR to specify these regions along with
+devices that need to access these regions. OS is expected to setup
+unity mappings for these regions for these devices to access these regions.
+
+What is AMD IVRS?
+^^^^^^^^^^^^^^^^^
+
+The architecture defines an ACPI-compatible data structure called an I/O
+Virtualization Reporting Structure (IVRS) that is used to convey information
+related to I/O virtualization to system software. The IVRS describes the
+configuration and capabilities of the IOMMUs contained in the platform as
+well as information about the devices that each IOMMU virtualizes.
+
+The IVRS provides information about the following:
+
+- IOMMUs present in the platform including their capabilities and proper configuration
+- System I/O topology relevant to each IOMMU
+- Peripheral devices that cannot be otherwise enumerated
+- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are generally exclusion ranges to be configured by system software.
+
+How is an I/O Virtual Address (IOVA) generated?
+-----------------------------------------------
+
+Well behaved drivers call dma_map_*() calls before sending command to device
+that needs to perform DMA. Once DMA is completed and mapping is no longer
+required, driver performs dma_unmap_*() calls to unmap the region.
+
+Some exceptions to IOVAs
+------------------------
+
+Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
+The same is true for peer to peer transactions. Hence we reserve the
+address from PCI MMIO ranges so they are not allocated for IOVA addresses.
+
+Graphics Problems?
+------------------
+
+If you encounter issues with integrated graphics devices, you can try
+adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
+to the kernel command line to use a 1:1 mapping for the IOMMU in
+general. On Intel you can also try "intel_iommu=igfx_off" to turn off
+translation specifically for the integrated graphics engine only. If
+this fixes anything, please ensure you file a bug reporting the problem.
+
+Fault reporting
+---------------
+When errors are reported, the IOMMU signals via an interrupt. The fault
+reason and device that caused it is printed on the console.
+
+
+Kernel Log Samples
+------------------
+
+Intel Boot Messages
+^^^^^^^^^^^^^^^^^^^
+
+Something like this gets printed indicating presence of DMAR tables
+in ACPI:
+
+::
+
+ ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
+
+When DMAR is being processed and initialized by ACPI, prints DMAR locations
+and any RMRR's processed:
+
+::
+
+ ACPI DMAR:Host address width 36
+ ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
+ ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
+ ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
+ ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
+ ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
+
+When DMAR is enabled for use, you will notice:
+
+::
+
+ PCI-DMA: Using DMAR IOMMU
+
+Intel Fault reporting
+^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+ DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
+ DMAR:[fault reason 05] PTE Write access is not set
+ DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
+ DMAR:[fault reason 05] PTE Write access is not set
+
+AMD Boot Messages
+^^^^^^^^^^^^^^^^^
+
+Something like this gets printed indicating presence of the IOMMU:
+
+::
+
+ AMD-Vi: Found IOMMU cap 0x40
+ AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC GA_vAPIC
+ AMD-Vi: Interrupt remapping enabled
+ AMD-Vi: Virtual APIC enabled
+
+AMD Fault reporting
+^^^^^^^^^^^^^^^^^^^
+
+::
+
+ AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
+ AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
--
2.35.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
2022-04-22 20:06 ` Alex Deucher
@ 2022-04-22 21:26 ` Robin Murphy
-1 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2022-04-22 21:26 UTC (permalink / raw)
To: Alex Deucher, linux-doc, linux-kernel, corbet, hpa, x86,
dave.hansen, bp, mingo, tglx, joro, Suravee.Suthikulpanit, will,
iommu, Vasant.Hegde
On 2022-04-22 21:06, Alex Deucher wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
FWIW,
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
>
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> index f498f1d36cd3..6f8409fe0674 100644
> --- a/Documentation/x86/index.rst
> +++ b/Documentation/x86/index.rst
> @@ -21,7 +21,7 @@ x86-specific Documentation
> tlb
> mtrr
> pat
> - intel-iommu
> + iommu
> intel_txt
> amd-memory-encryption
> pti
> diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
> deleted file mode 100644
> index 099f13d51d5f..000000000000
> --- a/Documentation/x86/intel-iommu.rst
> +++ /dev/null
> @@ -1,115 +0,0 @@
> -===================
> -Linux IOMMU Support
> -===================
> -
> -The architecture spec can be obtained from the below location.
> -
> -http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> -
> -This guide gives a quick cheat sheet for some basic understanding.
> -
> -Some Keywords
> -
> -- DMAR - DMA remapping
> -- DRHD - DMA Remapping Hardware Unit Definition
> -- RMRR - Reserved memory Region Reporting Structure
> -- ZLR - Zero length reads from PCI devices
> -- IOVA - IO Virtual address.
> -
> -Basic stuff
> ------------
> -
> -ACPI enumerates and lists the different DMA engines in the platform, and
> -device scope relationships between PCI devices and which DMA engine controls
> -them.
> -
> -What is RMRR?
> --------------
> -
> -There are some devices the BIOS controls, for e.g USB devices to perform
> -PS2 emulation. The regions of memory used for these devices are marked
> -reserved in the e820 map. When we turn on DMA translation, DMA to those
> -regions will fail. Hence BIOS uses RMRR to specify these regions along with
> -devices that need to access these regions. OS is expected to setup
> -unity mappings for these regions for these devices to access these regions.
> -
> -How is IOVA generated?
> -----------------------
> -
> -Well behaved drivers call pci_map_*() calls before sending command to device
> -that needs to perform DMA. Once DMA is completed and mapping is no longer
> -required, device performs a pci_unmap_*() calls to unmap the region.
> -
> -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
> -device has its own domain (hence protection). Devices under p2p bridges
> -share the virtual address with all devices under the p2p bridge due to
> -transaction id aliasing for p2p bridges.
> -
> -IOVA generation is pretty generic. We used the same technique as vmalloc()
> -but these are not global address spaces, but separate for each domain.
> -Different DMA engines may support different number of domains.
> -
> -We also allocate guard pages with each mapping, so we can attempt to catch
> -any overflow that might happen.
> -
> -
> -Graphics Problems?
> -------------------
> -If you encounter issues with graphics devices, you can try adding
> -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> -If this fixes anything, please ensure you file a bug reporting the problem.
> -
> -Some exceptions to IOVA
> ------------------------
> -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> -The same is true for peer to peer transactions. Hence we reserve the
> -address from PCI MMIO ranges so they are not allocated for IOVA addresses.
> -
> -
> -Fault reporting
> ----------------
> -When errors are reported, the DMA engine signals via an interrupt. The fault
> -reason and device that caused it with fault reason is printed on console.
> -
> -See below for sample.
> -
> -
> -Boot Message Sample
> --------------------
> -
> -Something like this gets printed indicating presence of DMAR tables
> -in ACPI.
> -
> -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
> -
> -When DMAR is being processed and initialized by ACPI, prints DMAR locations
> -and any RMRR's processed::
> -
> - ACPI DMAR:Host address width 36
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> - ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> - ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> -
> -When DMAR is enabled for use, you will notice..
> -
> -PCI-DMA: Using DMAR IOMMU
> --------------------------
> -
> -Fault reporting
> -^^^^^^^^^^^^^^^
> -
> -::
> -
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> -
> -TBD
> -----
> -
> -- For compatibility testing, could use unity map domain for all devices, just
> - provide a 1-1 for all useful memory under a single domain for all devices.
> -- API for paravirt ops for abstracting functionality for VMM folks.
> diff --git a/Documentation/x86/iommu.rst b/Documentation/x86/iommu.rst
> new file mode 100644
> index 000000000000..ed87d76a38d5
> --- /dev/null
> +++ b/Documentation/x86/iommu.rst
> @@ -0,0 +1,143 @@
> +=================
> +x86 IOMMU Support
> +=================
> +
> +The architecture specs can be obtained from the below locations.
> +
> +- Intel: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> +- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
> +
> +This guide gives a quick cheat sheet for some basic understanding.
> +
> +Basic stuff
> +-----------
> +
> +ACPI enumerates and lists the different IOMMUs on the platform, and
> +device scope relationships between devices and which IOMMU controls
> +them.
> +
> +Some ACPI Keywords:
> +
> +- DMAR - Intel DMA Remapping table
> +- DRHD - Intel DMA Remapping Hardware Unit Definition
> +- RMRR - Intel Reserved Memory Region Reporting Structure
> +- IVRS - AMD I/O Virtualization Reporting Structure
> +- IVDB - AMD I/O Virtualization Definition Block
> +- IVHD - AMD I/O Virtualization Hardware Definition
> +
> +What is Intel RMRR?
> +^^^^^^^^^^^^^^^^^^^
> +
> +There are some devices the BIOS controls, for e.g USB devices to perform
> +PS2 emulation. The regions of memory used for these devices are marked
> +reserved in the e820 map. When we turn on DMA translation, DMA to those
> +regions will fail. Hence BIOS uses RMRR to specify these regions along with
> +devices that need to access these regions. OS is expected to setup
> +unity mappings for these regions for these devices to access these regions.
> +
> +What is AMD IVRS?
> +^^^^^^^^^^^^^^^^^
> +
> +The architecture defines an ACPI-compatible data structure called an I/O
> +Virtualization Reporting Structure (IVRS) that is used to convey information
> +related to I/O virtualization to system software. The IVRS describes the
> +configuration and capabilities of the IOMMUs contained in the platform as
> +well as information about the devices that each IOMMU virtualizes.
> +
> +The IVRS provides information about the following:
> +
> +- IOMMUs present in the platform including their capabilities and proper configuration
> +- System I/O topology relevant to each IOMMU
> +- Peripheral devices that cannot be otherwise enumerated
> +- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are generally exclusion ranges to be configured by system software.
> +
> +How is an I/O Virtual Address (IOVA) generated?
> +-----------------------------------------------
> +
> +Well behaved drivers call dma_map_*() calls before sending command to device
> +that needs to perform DMA. Once DMA is completed and mapping is no longer
> +required, driver performs dma_unmap_*() calls to unmap the region.
> +
> +Some exceptions to IOVAs
> +------------------------
> +
> +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> +The same is true for peer to peer transactions. Hence we reserve the
> +address from PCI MMIO ranges so they are not allocated for IOVA addresses.
> +
> +Graphics Problems?
> +------------------
> +
> +If you encounter issues with integrated graphics devices, you can try
> +adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
> +to the kernel command line to use a 1:1 mapping for the IOMMU in
> +general. On Intel you can also try "intel_iommu=igfx_off" to turn off
> +translation specifically for the integrated graphics engine only. If
> +this fixes anything, please ensure you file a bug reporting the problem.
> +
> +Fault reporting
> +---------------
> +When errors are reported, the IOMMU signals via an interrupt. The fault
> +reason and device that caused it is printed on the console.
> +
> +
> +Kernel Log Samples
> +------------------
> +
> +Intel Boot Messages
> +^^^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of DMAR tables
> +in ACPI:
> +
> +::
> +
> + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
> +
> +When DMAR is being processed and initialized by ACPI, prints DMAR locations
> +and any RMRR's processed:
> +
> +::
> +
> + ACPI DMAR:Host address width 36
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> + ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> + ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> +
> +When DMAR is enabled for use, you will notice:
> +
> +::
> +
> + PCI-DMA: Using DMAR IOMMU
> +
> +Intel Fault reporting
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> +
> +AMD Boot Messages
> +^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of the IOMMU:
> +
> +::
> +
> + AMD-Vi: Found IOMMU cap 0x40
> + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC GA_vAPIC
> + AMD-Vi: Interrupt remapping enabled
> + AMD-Vi: Virtual APIC enabled
> +
> +AMD Fault reporting
> +^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
> + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-22 21:26 ` Robin Murphy
0 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2022-04-22 21:26 UTC (permalink / raw)
To: Alex Deucher, linux-doc, linux-kernel, corbet, hpa, x86,
dave.hansen, bp, mingo, tglx, joro, Suravee.Suthikulpanit, will,
iommu, Vasant.Hegde
On 2022-04-22 21:06, Alex Deucher wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
FWIW,
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
>
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> index f498f1d36cd3..6f8409fe0674 100644
> --- a/Documentation/x86/index.rst
> +++ b/Documentation/x86/index.rst
> @@ -21,7 +21,7 @@ x86-specific Documentation
> tlb
> mtrr
> pat
> - intel-iommu
> + iommu
> intel_txt
> amd-memory-encryption
> pti
> diff --git a/Documentation/x86/intel-iommu.rst b/Documentation/x86/intel-iommu.rst
> deleted file mode 100644
> index 099f13d51d5f..000000000000
> --- a/Documentation/x86/intel-iommu.rst
> +++ /dev/null
> @@ -1,115 +0,0 @@
> -===================
> -Linux IOMMU Support
> -===================
> -
> -The architecture spec can be obtained from the below location.
> -
> -http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> -
> -This guide gives a quick cheat sheet for some basic understanding.
> -
> -Some Keywords
> -
> -- DMAR - DMA remapping
> -- DRHD - DMA Remapping Hardware Unit Definition
> -- RMRR - Reserved memory Region Reporting Structure
> -- ZLR - Zero length reads from PCI devices
> -- IOVA - IO Virtual address.
> -
> -Basic stuff
> ------------
> -
> -ACPI enumerates and lists the different DMA engines in the platform, and
> -device scope relationships between PCI devices and which DMA engine controls
> -them.
> -
> -What is RMRR?
> --------------
> -
> -There are some devices the BIOS controls, for e.g USB devices to perform
> -PS2 emulation. The regions of memory used for these devices are marked
> -reserved in the e820 map. When we turn on DMA translation, DMA to those
> -regions will fail. Hence BIOS uses RMRR to specify these regions along with
> -devices that need to access these regions. OS is expected to setup
> -unity mappings for these regions for these devices to access these regions.
> -
> -How is IOVA generated?
> -----------------------
> -
> -Well behaved drivers call pci_map_*() calls before sending command to device
> -that needs to perform DMA. Once DMA is completed and mapping is no longer
> -required, device performs a pci_unmap_*() calls to unmap the region.
> -
> -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
> -device has its own domain (hence protection). Devices under p2p bridges
> -share the virtual address with all devices under the p2p bridge due to
> -transaction id aliasing for p2p bridges.
> -
> -IOVA generation is pretty generic. We used the same technique as vmalloc()
> -but these are not global address spaces, but separate for each domain.
> -Different DMA engines may support different number of domains.
> -
> -We also allocate guard pages with each mapping, so we can attempt to catch
> -any overflow that might happen.
> -
> -
> -Graphics Problems?
> -------------------
> -If you encounter issues with graphics devices, you can try adding
> -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> -If this fixes anything, please ensure you file a bug reporting the problem.
> -
> -Some exceptions to IOVA
> ------------------------
> -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> -The same is true for peer to peer transactions. Hence we reserve the
> -address from PCI MMIO ranges so they are not allocated for IOVA addresses.
> -
> -
> -Fault reporting
> ----------------
> -When errors are reported, the DMA engine signals via an interrupt. The fault
> -reason and device that caused it with fault reason is printed on console.
> -
> -See below for sample.
> -
> -
> -Boot Message Sample
> --------------------
> -
> -Something like this gets printed indicating presence of DMAR tables
> -in ACPI.
> -
> -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
> -
> -When DMAR is being processed and initialized by ACPI, prints DMAR locations
> -and any RMRR's processed::
> -
> - ACPI DMAR:Host address width 36
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> - ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> - ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> -
> -When DMAR is enabled for use, you will notice..
> -
> -PCI-DMA: Using DMAR IOMMU
> --------------------------
> -
> -Fault reporting
> -^^^^^^^^^^^^^^^
> -
> -::
> -
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> -
> -TBD
> -----
> -
> -- For compatibility testing, could use unity map domain for all devices, just
> - provide a 1-1 for all useful memory under a single domain for all devices.
> -- API for paravirt ops for abstracting functionality for VMM folks.
> diff --git a/Documentation/x86/iommu.rst b/Documentation/x86/iommu.rst
> new file mode 100644
> index 000000000000..ed87d76a38d5
> --- /dev/null
> +++ b/Documentation/x86/iommu.rst
> @@ -0,0 +1,143 @@
> +=================
> +x86 IOMMU Support
> +=================
> +
> +The architecture specs can be obtained from the below locations.
> +
> +- Intel: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> +- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
> +
> +This guide gives a quick cheat sheet for some basic understanding.
> +
> +Basic stuff
> +-----------
> +
> +ACPI enumerates and lists the different IOMMUs on the platform, and
> +device scope relationships between devices and which IOMMU controls
> +them.
> +
> +Some ACPI Keywords:
> +
> +- DMAR - Intel DMA Remapping table
> +- DRHD - Intel DMA Remapping Hardware Unit Definition
> +- RMRR - Intel Reserved Memory Region Reporting Structure
> +- IVRS - AMD I/O Virtualization Reporting Structure
> +- IVDB - AMD I/O Virtualization Definition Block
> +- IVHD - AMD I/O Virtualization Hardware Definition
> +
> +What is Intel RMRR?
> +^^^^^^^^^^^^^^^^^^^
> +
> +There are some devices the BIOS controls, for e.g USB devices to perform
> +PS2 emulation. The regions of memory used for these devices are marked
> +reserved in the e820 map. When we turn on DMA translation, DMA to those
> +regions will fail. Hence BIOS uses RMRR to specify these regions along with
> +devices that need to access these regions. OS is expected to setup
> +unity mappings for these regions for these devices to access these regions.
> +
> +What is AMD IVRS?
> +^^^^^^^^^^^^^^^^^
> +
> +The architecture defines an ACPI-compatible data structure called an I/O
> +Virtualization Reporting Structure (IVRS) that is used to convey information
> +related to I/O virtualization to system software. The IVRS describes the
> +configuration and capabilities of the IOMMUs contained in the platform as
> +well as information about the devices that each IOMMU virtualizes.
> +
> +The IVRS provides information about the following:
> +
> +- IOMMUs present in the platform including their capabilities and proper configuration
> +- System I/O topology relevant to each IOMMU
> +- Peripheral devices that cannot be otherwise enumerated
> +- Memory regions used by SMI/SMM, platform firmware, and platform hardware. These are generally exclusion ranges to be configured by system software.
> +
> +How is an I/O Virtual Address (IOVA) generated?
> +-----------------------------------------------
> +
> +Well behaved drivers call dma_map_*() calls before sending command to device
> +that needs to perform DMA. Once DMA is completed and mapping is no longer
> +required, driver performs dma_unmap_*() calls to unmap the region.
> +
> +Some exceptions to IOVAs
> +------------------------
> +
> +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> +The same is true for peer to peer transactions. Hence we reserve the
> +address from PCI MMIO ranges so they are not allocated for IOVA addresses.
> +
> +Graphics Problems?
> +------------------
> +
> +If you encounter issues with integrated graphics devices, you can try
> +adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
> +to the kernel command line to use a 1:1 mapping for the IOMMU in
> +general. On Intel you can also try "intel_iommu=igfx_off" to turn off
> +translation specifically for the integrated graphics engine only. If
> +this fixes anything, please ensure you file a bug reporting the problem.
> +
> +Fault reporting
> +---------------
> +When errors are reported, the IOMMU signals via an interrupt. The fault
> +reason and device that caused it is printed on the console.
> +
> +
> +Kernel Log Samples
> +------------------
> +
> +Intel Boot Messages
> +^^^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of DMAR tables
> +in ACPI:
> +
> +::
> +
> + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @ 0x000000007f5b5ef0
> +
> +When DMAR is being processed and initialized by ACPI, prints DMAR locations
> +and any RMRR's processed:
> +
> +::
> +
> + ACPI DMAR:Host address width 36
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> + ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> + ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> +
> +When DMAR is enabled for use, you will notice:
> +
> +::
> +
> + PCI-DMA: Using DMAR IOMMU
> +
> +Intel Fault reporting
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> +
> +AMD Boot Messages
> +^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of the IOMMU:
> +
> +::
> +
> + AMD-Vi: Found IOMMU cap 0x40
> + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC GA_vAPIC
> + AMD-Vi: Interrupt remapping enabled
> + AMD-Vi: Virtual APIC enabled
> +
> +AMD Fault reporting
> +^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
> + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
2022-04-22 20:06 ` Alex Deucher
@ 2022-04-26 16:44 ` Jacob Pan
-1 siblings, 0 replies; 12+ messages in thread
From: Jacob Pan @ 2022-04-26 16:44 UTC (permalink / raw)
To: Alex Deucher
Cc: linux-doc, linux-kernel, corbet, hpa, x86, dave.hansen, bp,
mingo, tglx, joro, Suravee.Suthikulpanit, will, iommu,
robin.murphy, Vasant.Hegde, jacob.jun.pan, Lu, Baolu
Hi Alex,
Thanks for doing this, really helps to catch up the current state. Please
see my comments inline.
On Fri, 22 Apr 2022 16:06:07 -0400, Alex Deucher
<alexander.deucher@amd.com> wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
>
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> index f498f1d36cd3..6f8409fe0674 100644
> --- a/Documentation/x86/index.rst
> +++ b/Documentation/x86/index.rst
> @@ -21,7 +21,7 @@ x86-specific Documentation
> tlb
> mtrr
> pat
> - intel-iommu
> + iommu
> intel_txt
> amd-memory-encryption
> pti
> diff --git a/Documentation/x86/intel-iommu.rst
> b/Documentation/x86/intel-iommu.rst deleted file mode 100644
> index 099f13d51d5f..000000000000
> --- a/Documentation/x86/intel-iommu.rst
> +++ /dev/null
> @@ -1,115 +0,0 @@
> -===================
> -Linux IOMMU Support
> -===================
> -
> -The architecture spec can be obtained from the below location.
> -
> -http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> -
> -This guide gives a quick cheat sheet for some basic understanding.
> -
> -Some Keywords
> -
> -- DMAR - DMA remapping
> -- DRHD - DMA Remapping Hardware Unit Definition
> -- RMRR - Reserved memory Region Reporting Structure
> -- ZLR - Zero length reads from PCI devices
> -- IOVA - IO Virtual address.
> -
I feel this combined document only focus on IOVA and DMA APIs, it is
considered as legacy DMA after scalable mode is introduced by Intel to
support DMA with PASID, shared virtual addressing (SVA).
Perhaps, we can also combine ./Documentation/x86/sva.rst
With scalable mode, it affects boot messages, fault reporting, etc. I am
not saying no to this document, just suggesting. I don't know where AMD is
at in terms of PASID support but there are lots of things in common between
VT-d and ARM's SMMU in terms of PASID/SVA. Should we broaden the purpose of
this document even further?
> -Basic stuff
> ------------
> -
> -ACPI enumerates and lists the different DMA engines in the platform, and
> -device scope relationships between PCI devices and which DMA engine
> controls -them.
> -
> -What is RMRR?
> --------------
> -
> -There are some devices the BIOS controls, for e.g USB devices to perform
> -PS2 emulation. The regions of memory used for these devices are marked
> -reserved in the e820 map. When we turn on DMA translation, DMA to those
> -regions will fail. Hence BIOS uses RMRR to specify these regions along
> with -devices that need to access these regions. OS is expected to setup
> -unity mappings for these regions for these devices to access these
> regions. -
> -How is IOVA generated?
> -----------------------
> -
> -Well behaved drivers call pci_map_*() calls before sending command to
> device -that needs to perform DMA. Once DMA is completed and mapping is
> no longer -required, device performs a pci_unmap_*() calls to unmap the
> region. -
> -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
> -device has its own domain (hence protection). Devices under p2p bridges
> -share the virtual address with all devices under the p2p bridge due to
> -transaction id aliasing for p2p bridges.
> -
> -IOVA generation is pretty generic. We used the same technique as
> vmalloc() -but these are not global address spaces, but separate for each
> domain. -Different DMA engines may support different number of domains.
> -
> -We also allocate guard pages with each mapping, so we can attempt to
> catch -any overflow that might happen.
> -
> -
> -Graphics Problems?
> -------------------
> -If you encounter issues with graphics devices, you can try adding
> -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> -If this fixes anything, please ensure you file a bug reporting the
> problem. -
> -Some exceptions to IOVA
> ------------------------
> -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> -The same is true for peer to peer transactions. Hence we reserve the
> -address from PCI MMIO ranges so they are not allocated for IOVA
> addresses. -
> -
> -Fault reporting
> ----------------
> -When errors are reported, the DMA engine signals via an interrupt. The
> fault -reason and device that caused it with fault reason is printed on
> console. -
> -See below for sample.
> -
> -
> -Boot Message Sample
> --------------------
> -
> -Something like this gets printed indicating presence of DMAR tables
> -in ACPI.
> -
> -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> 0x000000007f5b5ef0 -
> -When DMAR is being processed and initialized by ACPI, prints DMAR
> locations -and any RMRR's processed::
> -
> - ACPI DMAR:Host address width 36
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> - ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> - ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> -
> -When DMAR is enabled for use, you will notice..
> -
> -PCI-DMA: Using DMAR IOMMU
> --------------------------
> -
> -Fault reporting
> -^^^^^^^^^^^^^^^
> -
> -::
> -
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> -
> -TBD
> -----
> -
> -- For compatibility testing, could use unity map domain for all devices,
> just
> - provide a 1-1 for all useful memory under a single domain for all
> devices. -- API for paravirt ops for abstracting functionality for VMM
> folks. diff --git a/Documentation/x86/iommu.rst
> b/Documentation/x86/iommu.rst new file mode 100644
> index 000000000000..ed87d76a38d5
> --- /dev/null
> +++ b/Documentation/x86/iommu.rst
> @@ -0,0 +1,143 @@
> +=================
> +x86 IOMMU Support
> +=================
> +
> +The architecture specs can be obtained from the below locations.
> +
> +- Intel:
> http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> +- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf +
> +This guide gives a quick cheat sheet for some basic understanding.
> +
> +Basic stuff
> +-----------
> +
> +ACPI enumerates and lists the different IOMMUs on the platform, and
> +device scope relationships between devices and which IOMMU controls
> +them.
> +
> +Some ACPI Keywords:
> +
> +- DMAR - Intel DMA Remapping table
> +- DRHD - Intel DMA Remapping Hardware Unit Definition
> +- RMRR - Intel Reserved Memory Region Reporting Structure
> +- IVRS - AMD I/O Virtualization Reporting Structure
> +- IVDB - AMD I/O Virtualization Definition Block
> +- IVHD - AMD I/O Virtualization Hardware Definition
> +
> +What is Intel RMRR?
> +^^^^^^^^^^^^^^^^^^^
> +
> +There are some devices the BIOS controls, for e.g USB devices to perform
> +PS2 emulation. The regions of memory used for these devices are marked
> +reserved in the e820 map. When we turn on DMA translation, DMA to those
> +regions will fail. Hence BIOS uses RMRR to specify these regions along
> with +devices that need to access these regions. OS is expected to setup
> +unity mappings for these regions for these devices to access these
> regions. +
> +What is AMD IVRS?
> +^^^^^^^^^^^^^^^^^
> +
> +The architecture defines an ACPI-compatible data structure called an I/O
> +Virtualization Reporting Structure (IVRS) that is used to convey
> information +related to I/O virtualization to system software. The IVRS
> describes the +configuration and capabilities of the IOMMUs contained in
> the platform as +well as information about the devices that each IOMMU
> virtualizes. +
> +The IVRS provides information about the following:
> +
> +- IOMMUs present in the platform including their capabilities and proper
> configuration +- System I/O topology relevant to each IOMMU
> +- Peripheral devices that cannot be otherwise enumerated
> +- Memory regions used by SMI/SMM, platform firmware, and platform
> hardware. These are generally exclusion ranges to be configured by system
> software. + +How is an I/O Virtual Address (IOVA) generated?
> +-----------------------------------------------
> +
> +Well behaved drivers call dma_map_*() calls before sending command to
> device +that needs to perform DMA. Once DMA is completed and mapping is
> no longer +required, driver performs dma_unmap_*() calls to unmap the
> region. +
> +Some exceptions to IOVAs
> +------------------------
> +
> +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> +The same is true for peer to peer transactions. Hence we reserve the
> +address from PCI MMIO ranges so they are not allocated for IOVA
> addresses. +
> +Graphics Problems?
> +------------------
> +
> +If you encounter issues with integrated graphics devices, you can try
> +adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
> +to the kernel command line to use a 1:1 mapping for the IOMMU in
> +general. On Intel you can also try "intel_iommu=igfx_off" to turn off
> +translation specifically for the integrated graphics engine only. If
> +this fixes anything, please ensure you file a bug reporting the problem.
> +
> +Fault reporting
> +---------------
> +When errors are reported, the IOMMU signals via an interrupt. The fault
> +reason and device that caused it is printed on the console.
> +
> +
> +Kernel Log Samples
> +------------------
> +
> +Intel Boot Messages
> +^^^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of DMAR tables
> +in ACPI:
> +
> +::
> +
> + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> 0x000000007f5b5ef0 +
> +When DMAR is being processed and initialized by ACPI, prints DMAR
> locations +and any RMRR's processed:
> +
> +::
> +
> + ACPI DMAR:Host address width 36
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> + ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> + ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> +
> +When DMAR is enabled for use, you will notice:
> +
> +::
> +
> + PCI-DMA: Using DMAR IOMMU
> +
> +Intel Fault reporting
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> +
> +AMD Boot Messages
> +^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of the IOMMU:
> +
> +::
> +
> + AMD-Vi: Found IOMMU cap 0x40
> + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC
> GA_vAPIC
> + AMD-Vi: Interrupt remapping enabled
> + AMD-Vi: Virtual APIC enabled
> +
> +AMD Fault reporting
> +^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007
> address=0xffffc02000 flags=0x0000]
> + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007
> address=0xffffc02000 flags=0x0000]
Thanks,
Jacob
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-26 16:44 ` Jacob Pan
0 siblings, 0 replies; 12+ messages in thread
From: Jacob Pan @ 2022-04-26 16:44 UTC (permalink / raw)
To: Alex Deucher
Cc: dave.hansen, Lu, Baolu, corbet, robin.murphy, x86, linux-doc,
linux-kernel, Vasant.Hegde, iommu, mingo, bp, jacob.jun.pan, hpa,
tglx, will
Hi Alex,
Thanks for doing this, really helps to catch up the current state. Please
see my comments inline.
On Fri, 22 Apr 2022 16:06:07 -0400, Alex Deucher
<alexander.deucher@amd.com> wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
>
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> index f498f1d36cd3..6f8409fe0674 100644
> --- a/Documentation/x86/index.rst
> +++ b/Documentation/x86/index.rst
> @@ -21,7 +21,7 @@ x86-specific Documentation
> tlb
> mtrr
> pat
> - intel-iommu
> + iommu
> intel_txt
> amd-memory-encryption
> pti
> diff --git a/Documentation/x86/intel-iommu.rst
> b/Documentation/x86/intel-iommu.rst deleted file mode 100644
> index 099f13d51d5f..000000000000
> --- a/Documentation/x86/intel-iommu.rst
> +++ /dev/null
> @@ -1,115 +0,0 @@
> -===================
> -Linux IOMMU Support
> -===================
> -
> -The architecture spec can be obtained from the below location.
> -
> -http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> -
> -This guide gives a quick cheat sheet for some basic understanding.
> -
> -Some Keywords
> -
> -- DMAR - DMA remapping
> -- DRHD - DMA Remapping Hardware Unit Definition
> -- RMRR - Reserved memory Region Reporting Structure
> -- ZLR - Zero length reads from PCI devices
> -- IOVA - IO Virtual address.
> -
I feel this combined document only focus on IOVA and DMA APIs, it is
considered as legacy DMA after scalable mode is introduced by Intel to
support DMA with PASID, shared virtual addressing (SVA).
Perhaps, we can also combine ./Documentation/x86/sva.rst
With scalable mode, it affects boot messages, fault reporting, etc. I am
not saying no to this document, just suggesting. I don't know where AMD is
at in terms of PASID support but there are lots of things in common between
VT-d and ARM's SMMU in terms of PASID/SVA. Should we broaden the purpose of
this document even further?
> -Basic stuff
> ------------
> -
> -ACPI enumerates and lists the different DMA engines in the platform, and
> -device scope relationships between PCI devices and which DMA engine
> controls -them.
> -
> -What is RMRR?
> --------------
> -
> -There are some devices the BIOS controls, for e.g USB devices to perform
> -PS2 emulation. The regions of memory used for these devices are marked
> -reserved in the e820 map. When we turn on DMA translation, DMA to those
> -regions will fail. Hence BIOS uses RMRR to specify these regions along
> with -devices that need to access these regions. OS is expected to setup
> -unity mappings for these regions for these devices to access these
> regions. -
> -How is IOVA generated?
> -----------------------
> -
> -Well behaved drivers call pci_map_*() calls before sending command to
> device -that needs to perform DMA. Once DMA is completed and mapping is
> no longer -required, device performs a pci_unmap_*() calls to unmap the
> region. -
> -The Intel IOMMU driver allocates a virtual address per domain. Each PCIE
> -device has its own domain (hence protection). Devices under p2p bridges
> -share the virtual address with all devices under the p2p bridge due to
> -transaction id aliasing for p2p bridges.
> -
> -IOVA generation is pretty generic. We used the same technique as
> vmalloc() -but these are not global address spaces, but separate for each
> domain. -Different DMA engines may support different number of domains.
> -
> -We also allocate guard pages with each mapping, so we can attempt to
> catch -any overflow that might happen.
> -
> -
> -Graphics Problems?
> -------------------
> -If you encounter issues with graphics devices, you can try adding
> -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> -If this fixes anything, please ensure you file a bug reporting the
> problem. -
> -Some exceptions to IOVA
> ------------------------
> -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> -The same is true for peer to peer transactions. Hence we reserve the
> -address from PCI MMIO ranges so they are not allocated for IOVA
> addresses. -
> -
> -Fault reporting
> ----------------
> -When errors are reported, the DMA engine signals via an interrupt. The
> fault -reason and device that caused it with fault reason is printed on
> console. -
> -See below for sample.
> -
> -
> -Boot Message Sample
> --------------------
> -
> -Something like this gets printed indicating presence of DMAR tables
> -in ACPI.
> -
> -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> 0x000000007f5b5ef0 -
> -When DMAR is being processed and initialized by ACPI, prints DMAR
> locations -and any RMRR's processed::
> -
> - ACPI DMAR:Host address width 36
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> - ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> - ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> -
> -When DMAR is enabled for use, you will notice..
> -
> -PCI-DMA: Using DMAR IOMMU
> --------------------------
> -
> -Fault reporting
> -^^^^^^^^^^^^^^^
> -
> -::
> -
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> - DMAR:[fault reason 05] PTE Write access is not set
> -
> -TBD
> -----
> -
> -- For compatibility testing, could use unity map domain for all devices,
> just
> - provide a 1-1 for all useful memory under a single domain for all
> devices. -- API for paravirt ops for abstracting functionality for VMM
> folks. diff --git a/Documentation/x86/iommu.rst
> b/Documentation/x86/iommu.rst new file mode 100644
> index 000000000000..ed87d76a38d5
> --- /dev/null
> +++ b/Documentation/x86/iommu.rst
> @@ -0,0 +1,143 @@
> +=================
> +x86 IOMMU Support
> +=================
> +
> +The architecture specs can be obtained from the below locations.
> +
> +- Intel:
> http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
> +- AMD: https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf +
> +This guide gives a quick cheat sheet for some basic understanding.
> +
> +Basic stuff
> +-----------
> +
> +ACPI enumerates and lists the different IOMMUs on the platform, and
> +device scope relationships between devices and which IOMMU controls
> +them.
> +
> +Some ACPI Keywords:
> +
> +- DMAR - Intel DMA Remapping table
> +- DRHD - Intel DMA Remapping Hardware Unit Definition
> +- RMRR - Intel Reserved Memory Region Reporting Structure
> +- IVRS - AMD I/O Virtualization Reporting Structure
> +- IVDB - AMD I/O Virtualization Definition Block
> +- IVHD - AMD I/O Virtualization Hardware Definition
> +
> +What is Intel RMRR?
> +^^^^^^^^^^^^^^^^^^^
> +
> +There are some devices the BIOS controls, for e.g USB devices to perform
> +PS2 emulation. The regions of memory used for these devices are marked
> +reserved in the e820 map. When we turn on DMA translation, DMA to those
> +regions will fail. Hence BIOS uses RMRR to specify these regions along
> with +devices that need to access these regions. OS is expected to setup
> +unity mappings for these regions for these devices to access these
> regions. +
> +What is AMD IVRS?
> +^^^^^^^^^^^^^^^^^
> +
> +The architecture defines an ACPI-compatible data structure called an I/O
> +Virtualization Reporting Structure (IVRS) that is used to convey
> information +related to I/O virtualization to system software. The IVRS
> describes the +configuration and capabilities of the IOMMUs contained in
> the platform as +well as information about the devices that each IOMMU
> virtualizes. +
> +The IVRS provides information about the following:
> +
> +- IOMMUs present in the platform including their capabilities and proper
> configuration +- System I/O topology relevant to each IOMMU
> +- Peripheral devices that cannot be otherwise enumerated
> +- Memory regions used by SMI/SMM, platform firmware, and platform
> hardware. These are generally exclusion ranges to be configured by system
> software. + +How is an I/O Virtual Address (IOVA) generated?
> +-----------------------------------------------
> +
> +Well behaved drivers call dma_map_*() calls before sending command to
> device +that needs to perform DMA. Once DMA is completed and mapping is
> no longer +required, driver performs dma_unmap_*() calls to unmap the
> region. +
> +Some exceptions to IOVAs
> +------------------------
> +
> +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> +The same is true for peer to peer transactions. Hence we reserve the
> +address from PCI MMIO ranges so they are not allocated for IOVA
> addresses. +
> +Graphics Problems?
> +------------------
> +
> +If you encounter issues with integrated graphics devices, you can try
> +adding the option "iommu.passthrough=1", or the equivalent "iommu=pt",
> +to the kernel command line to use a 1:1 mapping for the IOMMU in
> +general. On Intel you can also try "intel_iommu=igfx_off" to turn off
> +translation specifically for the integrated graphics engine only. If
> +this fixes anything, please ensure you file a bug reporting the problem.
> +
> +Fault reporting
> +---------------
> +When errors are reported, the IOMMU signals via an interrupt. The fault
> +reason and device that caused it is printed on the console.
> +
> +
> +Kernel Log Samples
> +------------------
> +
> +Intel Boot Messages
> +^^^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of DMAR tables
> +in ACPI:
> +
> +::
> +
> + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> 0x000000007f5b5ef0 +
> +When DMAR is being processed and initialized by ACPI, prints DMAR
> locations +and any RMRR's processed:
> +
> +::
> +
> + ACPI DMAR:Host address width 36
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> + ACPI DMAR:RMRR base: 0x00000000000ed000 end: 0x00000000000effff
> + ACPI DMAR:RMRR base: 0x000000007f600000 end: 0x000000007fffffff
> +
> +When DMAR is enabled for use, you will notice:
> +
> +::
> +
> + PCI-DMA: Using DMAR IOMMU
> +
> +Intel Fault reporting
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> + DMAR:[fault reason 05] PTE Write access is not set
> +
> +AMD Boot Messages
> +^^^^^^^^^^^^^^^^^
> +
> +Something like this gets printed indicating presence of the IOMMU:
> +
> +::
> +
> + AMD-Vi: Found IOMMU cap 0x40
> + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC
> GA_vAPIC
> + AMD-Vi: Interrupt remapping enabled
> + AMD-Vi: Virtual APIC enabled
> +
> +AMD Fault reporting
> +^^^^^^^^^^^^^^^^^^^
> +
> +::
> +
> + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007
> address=0xffffc02000 flags=0x0000]
> + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007
> address=0xffffc02000 flags=0x0000]
Thanks,
Jacob
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH] Documentation: x86: rework IOMMU documentation
2022-04-26 16:44 ` Jacob Pan
@ 2022-04-27 17:12 ` Deucher, Alexander via iommu
-1 siblings, 0 replies; 12+ messages in thread
From: Deucher, Alexander @ 2022-04-27 17:12 UTC (permalink / raw)
To: Jacob Pan
Cc: linux-doc, linux-kernel, corbet, hpa, x86, dave.hansen, bp,
mingo, tglx, joro, Suthikulpanit, Suravee, will, iommu,
robin.murphy, Hegde, Vasant, Lu, Baolu
[Public]
> -----Original Message-----
> From: Jacob Pan <jacob.jun.pan@intel.com>
> Sent: Tuesday, April 26, 2022 12:45 PM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org;
> corbet@lwn.net; hpa@zytor.com; x86@kernel.org;
> dave.hansen@linux.intel.com; bp@alien8.de; mingo@redhat.com;
> tglx@linutronix.de; joro@8bytes.org; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; will@kernel.org; iommu@lists.linux-
> foundation.org; robin.murphy@arm.com; Hegde, Vasant
> <Vasant.Hegde@amd.com>; jacob.jun.pan@intel.com; Lu, Baolu
> <baolu.lu@intel.com>
> Subject: Re: [PATCH] Documentation: x86: rework IOMMU documentation
>
> Hi Alex,
>
> Thanks for doing this, really helps to catch up the current state. Please see my
> comments inline.
>
> On Fri, 22 Apr 2022 16:06:07 -0400, Alex Deucher
> <alexander.deucher@amd.com> wrote:
>
> > Add preliminary documentation for AMD IOMMU and combine with the
> > existing Intel IOMMU documentation and clean up and modernize some of
> > the existing documentation to align with the current state of the
> > kernel.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >
> > V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> > a device) and document proper DMA API. Also correct the fact that
> > the AMD IOMMU is not limited to managing PCI devices.
> > v3: Fix spelling and rework text as suggested by Vasant
> > v4: Combine Intel and AMD documents into a single document as suggested
> > by Dave Hansen
> > v5: Clarify that keywords are related to ACPI, grammatical fixes
> > v6: Make more stuff common based on feedback from Robin
> >
> > Documentation/x86/index.rst | 2 +-
> > Documentation/x86/intel-iommu.rst | 115 ------------------------
> > Documentation/x86/iommu.rst | 143
> ++++++++++++++++++++++++++++++
> > 3 files changed, 144 insertions(+), 116 deletions(-) delete mode
> > 100644 Documentation/x86/intel-iommu.rst create mode 100644
> > Documentation/x86/iommu.rst
> >
> > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> > index f498f1d36cd3..6f8409fe0674 100644
> > --- a/Documentation/x86/index.rst
> > +++ b/Documentation/x86/index.rst
> > @@ -21,7 +21,7 @@ x86-specific Documentation
> > tlb
> > mtrr
> > pat
> > - intel-iommu
> > + iommu
> > intel_txt
> > amd-memory-encryption
> > pti
> > diff --git a/Documentation/x86/intel-iommu.rst
> > b/Documentation/x86/intel-iommu.rst deleted file mode 100644 index
> > 099f13d51d5f..000000000000
> > --- a/Documentation/x86/intel-iommu.rst
> > +++ /dev/null
> > @@ -1,115 +0,0 @@
> > -===================
> > -Linux IOMMU Support
> > -===================
> > -
> > -The architecture spec can be obtained from the below location.
> > -
> > -
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >
> intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments
> %2Fprodu
> > ct-specifications%2Fvt-directed-io-
> spec.pdf&data=05%7C01%7Calexand
> >
> er.deucher%40amd.com%7C929847a4b2524432d1a608da27a3c9b0%7C3dd
> 8961fe488
> >
> 4e608e11a82d994e183d%7C0%7C0%7C637865881851295857%7CUnknow
> n%7CTWFpbGZs
> >
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D
> >
> %7C3000%7C%7C%7C&sdata=KaPkvBSHWbn1cKBRzyk9H%2BuNDll%2Bq
> 3wPfR3SFVA
> > LwkU%3D&reserved=0
> > -
> > -This guide gives a quick cheat sheet for some basic understanding.
> > -
> > -Some Keywords
> > -
> > -- DMAR - DMA remapping
> > -- DRHD - DMA Remapping Hardware Unit Definition
> > -- RMRR - Reserved memory Region Reporting Structure
> > -- ZLR - Zero length reads from PCI devices
> > -- IOVA - IO Virtual address.
> > -
> I feel this combined document only focus on IOVA and DMA APIs, it is
> considered as legacy DMA after scalable mode is introduced by Intel to
> support DMA with PASID, shared virtual addressing (SVA).
> Perhaps, we can also combine ./Documentation/x86/sva.rst
I think it would make sense to take that up in a separate patch set.
>
> With scalable mode, it affects boot messages, fault reporting, etc. I am not
> saying no to this document, just suggesting. I don't know where AMD is at in
> terms of PASID support but there are lots of things in common between VT-d
> and ARM's SMMU in terms of PASID/SVA. Should we broaden the purpose of
> this document even further?
I think that would make sense for a future clean up. I'd like to land the current clean up first.
AMD's IOMMU driver has supported PASID for probably 8-10 years. When we originally added it no other vendors were interested in supporting it so we made it a private API which was used by other AMD drivers that needed it at the time. Suravee can probably comment on the status of our support for the cross vendor API.
Alex
>
> > -Basic stuff
> > ------------
> > -
> > -ACPI enumerates and lists the different DMA engines in the platform,
> > and -device scope relationships between PCI devices and which DMA
> > engine controls -them.
> > -
> > -What is RMRR?
> > --------------
> > -
> > -There are some devices the BIOS controls, for e.g USB devices to
> > perform
> > -PS2 emulation. The regions of memory used for these devices are
> > marked -reserved in the e820 map. When we turn on DMA translation,
> DMA
> > to those -regions will fail. Hence BIOS uses RMRR to specify these
> > regions along with -devices that need to access these regions. OS is
> > expected to setup -unity mappings for these regions for these devices
> > to access these regions. - -How is IOVA generated?
> > -----------------------
> > -
> > -Well behaved drivers call pci_map_*() calls before sending command to
> > device -that needs to perform DMA. Once DMA is completed and mapping
> > is no longer -required, device performs a pci_unmap_*() calls to unmap
> > the region. - -The Intel IOMMU driver allocates a virtual address per
> > domain. Each PCIE -device has its own domain (hence protection).
> > Devices under p2p bridges -share the virtual address with all devices
> > under the p2p bridge due to -transaction id aliasing for p2p bridges.
> > -
> > -IOVA generation is pretty generic. We used the same technique as
> > vmalloc() -but these are not global address spaces, but separate for
> > each domain. -Different DMA engines may support different number of
> domains.
> > -
> > -We also allocate guard pages with each mapping, so we can attempt to
> > catch -any overflow that might happen.
> > -
> > -
> > -Graphics Problems?
> > -------------------
> > -If you encounter issues with graphics devices, you can try adding
> > -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> > -If this fixes anything, please ensure you file a bug reporting the
> > problem. - -Some exceptions to IOVA
> > ------------------------
> > -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> > -The same is true for peer to peer transactions. Hence we reserve the
> > -address from PCI MMIO ranges so they are not allocated for IOVA
> > addresses. -
> > -
> > -Fault reporting
> > ----------------
> > -When errors are reported, the DMA engine signals via an interrupt.
> > The fault -reason and device that caused it with fault reason is
> > printed on console. - -See below for sample.
> > -
> > -
> > -Boot Message Sample
> > --------------------
> > -
> > -Something like this gets printed indicating presence of DMAR tables
> > -in ACPI.
> > -
> > -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> > 0x000000007f5b5ef0 -
> > -When DMAR is being processed and initialized by ACPI, prints DMAR
> > locations -and any RMRR's processed::
> > -
> > - ACPI DMAR:Host address width 36
> > - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> > - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> > - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> > - ACPI DMAR:RMRR base: 0x00000000000ed000 end:
> 0x00000000000effff
> > - ACPI DMAR:RMRR base: 0x000000007f600000 end:
> 0x000000007fffffff
> > -
> > -When DMAR is enabled for use, you will notice..
> > -
> > -PCI-DMA: Using DMAR IOMMU
> > --------------------------
> > -
> > -Fault reporting
> > -^^^^^^^^^^^^^^^
> > -
> > -::
> > -
> > - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > - DMAR:[fault reason 05] PTE Write access is not set
> > - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > - DMAR:[fault reason 05] PTE Write access is not set
> > -
> > -TBD
> > -----
> > -
> > -- For compatibility testing, could use unity map domain for all
> > devices, just
> > - provide a 1-1 for all useful memory under a single domain for all
> > devices. -- API for paravirt ops for abstracting functionality for VMM
> > folks. diff --git a/Documentation/x86/iommu.rst
> > b/Documentation/x86/iommu.rst new file mode 100644 index
> > 000000000000..ed87d76a38d5
> > --- /dev/null
> > +++ b/Documentation/x86/iommu.rst
> > @@ -0,0 +1,143 @@
> > +=================
> > +x86 IOMMU Support
> > +=================
> > +
> > +The architecture specs can be obtained from the below locations.
> > +
> > +- Intel:
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.i
> >
> ntel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%
> 2Fproduc
> > t-specifications%2Fvt-directed-io-
> spec.pdf&data=05%7C01%7Calexande
> >
> r.deucher%40amd.com%7C929847a4b2524432d1a608da27a3c9b0%7C3dd8
> 961fe4884
> >
> e608e11a82d994e183d%7C0%7C0%7C637865881851295857%7CUnknown
> %7CTWFpbGZsb
> >
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%
> >
> 7C3000%7C%7C%7C&sdata=KaPkvBSHWbn1cKBRzyk9H%2BuNDll%2Bq3
> wPfR3SFVAL
> > wkU%3D&reserved=0
> > +- AMD:
> >
> +https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> .amd.com%2Fsystem%2Ffiles%2FTechDocs%2F48882_IOMMU.pdf&data
> =05%7C01%7Calexander.deucher%40amd.com%7C929847a4b2524432d1a6
> 08da27a3c9b0%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
> 7865881851295857%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&
> amp;sdata=QFq8HBY6cm2XaZ50u5LQA3RX3NX%2BwVsMqfJFDH9bFDs%3D&a
> mp;reserved=0 + This guide gives a quick cheat sheet for some basic
> understanding.
> > +
> > +Basic stuff
> > +-----------
> > +
> > +ACPI enumerates and lists the different IOMMUs on the platform, and
> > +device scope relationships between devices and which IOMMU controls
> > +them.
> > +
> > +Some ACPI Keywords:
> > +
> > +- DMAR - Intel DMA Remapping table
> > +- DRHD - Intel DMA Remapping Hardware Unit Definition
> > +- RMRR - Intel Reserved Memory Region Reporting Structure
> > +- IVRS - AMD I/O Virtualization Reporting Structure
> > +- IVDB - AMD I/O Virtualization Definition Block
> > +- IVHD - AMD I/O Virtualization Hardware Definition
> > +
> > +What is Intel RMRR?
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +There are some devices the BIOS controls, for e.g USB devices to
> > +perform
> > +PS2 emulation. The regions of memory used for these devices are
> > +marked reserved in the e820 map. When we turn on DMA translation,
> DMA
> > +to those regions will fail. Hence BIOS uses RMRR to specify these
> > +regions along
> > with +devices that need to access these regions. OS is expected to
> > setup
> > +unity mappings for these regions for these devices to access these
> > regions. +
> > +What is AMD IVRS?
> > +^^^^^^^^^^^^^^^^^
> > +
> > +The architecture defines an ACPI-compatible data structure called an
> > +I/O Virtualization Reporting Structure (IVRS) that is used to convey
> > information +related to I/O virtualization to system software. The
> > IVRS describes the +configuration and capabilities of the IOMMUs
> > contained in the platform as +well as information about the devices
> > that each IOMMU virtualizes. +
> > +The IVRS provides information about the following:
> > +
> > +- IOMMUs present in the platform including their capabilities and
> > +proper
> > configuration +- System I/O topology relevant to each IOMMU
> > +- Peripheral devices that cannot be otherwise enumerated
> > +- Memory regions used by SMI/SMM, platform firmware, and platform
> > hardware. These are generally exclusion ranges to be configured by
> > system software. + +How is an I/O Virtual Address (IOVA) generated?
> > +-----------------------------------------------
> > +
> > +Well behaved drivers call dma_map_*() calls before sending command to
> > device +that needs to perform DMA. Once DMA is completed and mapping
> > is no longer +required, driver performs dma_unmap_*() calls to unmap
> > the region. +
> > +Some exceptions to IOVAs
> > +------------------------
> > +
> > +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> > +The same is true for peer to peer transactions. Hence we reserve the
> > +address from PCI MMIO ranges so they are not allocated for IOVA
> > addresses. +
> > +Graphics Problems?
> > +------------------
> > +
> > +If you encounter issues with integrated graphics devices, you can try
> > +adding the option "iommu.passthrough=1", or the equivalent
> > +"iommu=pt", to the kernel command line to use a 1:1 mapping for the
> > +IOMMU in general. On Intel you can also try "intel_iommu=igfx_off"
> > +to turn off translation specifically for the integrated graphics
> > +engine only. If this fixes anything, please ensure you file a bug reporting
> the problem.
> > +
> > +Fault reporting
> > +---------------
> > +When errors are reported, the IOMMU signals via an interrupt. The
> > +fault reason and device that caused it is printed on the console.
> > +
> > +
> > +Kernel Log Samples
> > +------------------
> > +
> > +Intel Boot Messages
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +Something like this gets printed indicating presence of DMAR tables
> > +in ACPI:
> > +
> > +::
> > +
> > + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT
> 0x00000097) @
> > 0x000000007f5b5ef0 +
> > +When DMAR is being processed and initialized by ACPI, prints DMAR
> > locations +and any RMRR's processed:
> > +
> > +::
> > +
> > + ACPI DMAR:Host address width 36
> > + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> > + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> > + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> > + ACPI DMAR:RMRR base: 0x00000000000ed000 end:
> 0x00000000000effff
> > + ACPI DMAR:RMRR base: 0x000000007f600000 end:
> 0x000000007fffffff
> > +
> > +When DMAR is enabled for use, you will notice:
> > +
> > +::
> > +
> > + PCI-DMA: Using DMAR IOMMU
> > +
> > +Intel Fault reporting
> > +^^^^^^^^^^^^^^^^^^^^^
> > +
> > +::
> > +
> > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > + DMAR:[fault reason 05] PTE Write access is not set
> > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > + DMAR:[fault reason 05] PTE Write access is not set
> > +
> > +AMD Boot Messages
> > +^^^^^^^^^^^^^^^^^
> > +
> > +Something like this gets printed indicating presence of the IOMMU:
> > +
> > +::
> > +
> > + AMD-Vi: Found IOMMU cap 0x40
> > + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC
> > GA_vAPIC
> > + AMD-Vi: Interrupt remapping enabled
> > + AMD-Vi: Virtual APIC enabled
> > +
> > +AMD Fault reporting
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +::
> > +
> > + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007
> > address=0xffffc02000 flags=0x0000]
> > + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0
> domain=0x0007
> > address=0xffffc02000 flags=0x0000]
>
>
> Thanks,
>
> Jacob
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-27 17:12 ` Deucher, Alexander via iommu
0 siblings, 0 replies; 12+ messages in thread
From: Deucher, Alexander via iommu @ 2022-04-27 17:12 UTC (permalink / raw)
To: Jacob Pan
Cc: dave.hansen, Lu, Baolu, corbet, robin.murphy, x86, linux-doc,
linux-kernel, Hegde, Vasant, iommu, mingo, bp, hpa, tglx, will
[Public]
> -----Original Message-----
> From: Jacob Pan <jacob.jun.pan@intel.com>
> Sent: Tuesday, April 26, 2022 12:45 PM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org;
> corbet@lwn.net; hpa@zytor.com; x86@kernel.org;
> dave.hansen@linux.intel.com; bp@alien8.de; mingo@redhat.com;
> tglx@linutronix.de; joro@8bytes.org; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; will@kernel.org; iommu@lists.linux-
> foundation.org; robin.murphy@arm.com; Hegde, Vasant
> <Vasant.Hegde@amd.com>; jacob.jun.pan@intel.com; Lu, Baolu
> <baolu.lu@intel.com>
> Subject: Re: [PATCH] Documentation: x86: rework IOMMU documentation
>
> Hi Alex,
>
> Thanks for doing this, really helps to catch up the current state. Please see my
> comments inline.
>
> On Fri, 22 Apr 2022 16:06:07 -0400, Alex Deucher
> <alexander.deucher@amd.com> wrote:
>
> > Add preliminary documentation for AMD IOMMU and combine with the
> > existing Intel IOMMU documentation and clean up and modernize some of
> > the existing documentation to align with the current state of the
> > kernel.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> > ---
> >
> > V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> > a device) and document proper DMA API. Also correct the fact that
> > the AMD IOMMU is not limited to managing PCI devices.
> > v3: Fix spelling and rework text as suggested by Vasant
> > v4: Combine Intel and AMD documents into a single document as suggested
> > by Dave Hansen
> > v5: Clarify that keywords are related to ACPI, grammatical fixes
> > v6: Make more stuff common based on feedback from Robin
> >
> > Documentation/x86/index.rst | 2 +-
> > Documentation/x86/intel-iommu.rst | 115 ------------------------
> > Documentation/x86/iommu.rst | 143
> ++++++++++++++++++++++++++++++
> > 3 files changed, 144 insertions(+), 116 deletions(-) delete mode
> > 100644 Documentation/x86/intel-iommu.rst create mode 100644
> > Documentation/x86/iommu.rst
> >
> > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> > index f498f1d36cd3..6f8409fe0674 100644
> > --- a/Documentation/x86/index.rst
> > +++ b/Documentation/x86/index.rst
> > @@ -21,7 +21,7 @@ x86-specific Documentation
> > tlb
> > mtrr
> > pat
> > - intel-iommu
> > + iommu
> > intel_txt
> > amd-memory-encryption
> > pti
> > diff --git a/Documentation/x86/intel-iommu.rst
> > b/Documentation/x86/intel-iommu.rst deleted file mode 100644 index
> > 099f13d51d5f..000000000000
> > --- a/Documentation/x86/intel-iommu.rst
> > +++ /dev/null
> > @@ -1,115 +0,0 @@
> > -===================
> > -Linux IOMMU Support
> > -===================
> > -
> > -The architecture spec can be obtained from the below location.
> > -
> > -
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> >
> intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments
> %2Fprodu
> > ct-specifications%2Fvt-directed-io-
> spec.pdf&data=05%7C01%7Calexand
> >
> er.deucher%40amd.com%7C929847a4b2524432d1a608da27a3c9b0%7C3dd
> 8961fe488
> >
> 4e608e11a82d994e183d%7C0%7C0%7C637865881851295857%7CUnknow
> n%7CTWFpbGZs
> >
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D
> >
> %7C3000%7C%7C%7C&sdata=KaPkvBSHWbn1cKBRzyk9H%2BuNDll%2Bq
> 3wPfR3SFVA
> > LwkU%3D&reserved=0
> > -
> > -This guide gives a quick cheat sheet for some basic understanding.
> > -
> > -Some Keywords
> > -
> > -- DMAR - DMA remapping
> > -- DRHD - DMA Remapping Hardware Unit Definition
> > -- RMRR - Reserved memory Region Reporting Structure
> > -- ZLR - Zero length reads from PCI devices
> > -- IOVA - IO Virtual address.
> > -
> I feel this combined document only focus on IOVA and DMA APIs, it is
> considered as legacy DMA after scalable mode is introduced by Intel to
> support DMA with PASID, shared virtual addressing (SVA).
> Perhaps, we can also combine ./Documentation/x86/sva.rst
I think it would make sense to take that up in a separate patch set.
>
> With scalable mode, it affects boot messages, fault reporting, etc. I am not
> saying no to this document, just suggesting. I don't know where AMD is at in
> terms of PASID support but there are lots of things in common between VT-d
> and ARM's SMMU in terms of PASID/SVA. Should we broaden the purpose of
> this document even further?
I think that would make sense for a future clean up. I'd like to land the current clean up first.
AMD's IOMMU driver has supported PASID for probably 8-10 years. When we originally added it no other vendors were interested in supporting it so we made it a private API which was used by other AMD drivers that needed it at the time. Suravee can probably comment on the status of our support for the cross vendor API.
Alex
>
> > -Basic stuff
> > ------------
> > -
> > -ACPI enumerates and lists the different DMA engines in the platform,
> > and -device scope relationships between PCI devices and which DMA
> > engine controls -them.
> > -
> > -What is RMRR?
> > --------------
> > -
> > -There are some devices the BIOS controls, for e.g USB devices to
> > perform
> > -PS2 emulation. The regions of memory used for these devices are
> > marked -reserved in the e820 map. When we turn on DMA translation,
> DMA
> > to those -regions will fail. Hence BIOS uses RMRR to specify these
> > regions along with -devices that need to access these regions. OS is
> > expected to setup -unity mappings for these regions for these devices
> > to access these regions. - -How is IOVA generated?
> > -----------------------
> > -
> > -Well behaved drivers call pci_map_*() calls before sending command to
> > device -that needs to perform DMA. Once DMA is completed and mapping
> > is no longer -required, device performs a pci_unmap_*() calls to unmap
> > the region. - -The Intel IOMMU driver allocates a virtual address per
> > domain. Each PCIE -device has its own domain (hence protection).
> > Devices under p2p bridges -share the virtual address with all devices
> > under the p2p bridge due to -transaction id aliasing for p2p bridges.
> > -
> > -IOVA generation is pretty generic. We used the same technique as
> > vmalloc() -but these are not global address spaces, but separate for
> > each domain. -Different DMA engines may support different number of
> domains.
> > -
> > -We also allocate guard pages with each mapping, so we can attempt to
> > catch -any overflow that might happen.
> > -
> > -
> > -Graphics Problems?
> > -------------------
> > -If you encounter issues with graphics devices, you can try adding
> > -option intel_iommu=igfx_off to turn off the integrated graphics engine.
> > -If this fixes anything, please ensure you file a bug reporting the
> > problem. - -Some exceptions to IOVA
> > ------------------------
> > -Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> > -The same is true for peer to peer transactions. Hence we reserve the
> > -address from PCI MMIO ranges so they are not allocated for IOVA
> > addresses. -
> > -
> > -Fault reporting
> > ----------------
> > -When errors are reported, the DMA engine signals via an interrupt.
> > The fault -reason and device that caused it with fault reason is
> > printed on console. - -See below for sample.
> > -
> > -
> > -Boot Message Sample
> > --------------------
> > -
> > -Something like this gets printed indicating presence of DMAR tables
> > -in ACPI.
> > -
> > -ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT 0x00000097) @
> > 0x000000007f5b5ef0 -
> > -When DMAR is being processed and initialized by ACPI, prints DMAR
> > locations -and any RMRR's processed::
> > -
> > - ACPI DMAR:Host address width 36
> > - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> > - ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> > - ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> > - ACPI DMAR:RMRR base: 0x00000000000ed000 end:
> 0x00000000000effff
> > - ACPI DMAR:RMRR base: 0x000000007f600000 end:
> 0x000000007fffffff
> > -
> > -When DMAR is enabled for use, you will notice..
> > -
> > -PCI-DMA: Using DMAR IOMMU
> > --------------------------
> > -
> > -Fault reporting
> > -^^^^^^^^^^^^^^^
> > -
> > -::
> > -
> > - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > - DMAR:[fault reason 05] PTE Write access is not set
> > - DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > - DMAR:[fault reason 05] PTE Write access is not set
> > -
> > -TBD
> > -----
> > -
> > -- For compatibility testing, could use unity map domain for all
> > devices, just
> > - provide a 1-1 for all useful memory under a single domain for all
> > devices. -- API for paravirt ops for abstracting functionality for VMM
> > folks. diff --git a/Documentation/x86/iommu.rst
> > b/Documentation/x86/iommu.rst new file mode 100644 index
> > 000000000000..ed87d76a38d5
> > --- /dev/null
> > +++ b/Documentation/x86/iommu.rst
> > @@ -0,0 +1,143 @@
> > +=================
> > +x86 IOMMU Support
> > +=================
> > +
> > +The architecture specs can be obtained from the below locations.
> > +
> > +- Intel:
> >
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.i
> >
> ntel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%
> 2Fproduc
> > t-specifications%2Fvt-directed-io-
> spec.pdf&data=05%7C01%7Calexande
> >
> r.deucher%40amd.com%7C929847a4b2524432d1a608da27a3c9b0%7C3dd8
> 961fe4884
> >
> e608e11a82d994e183d%7C0%7C0%7C637865881851295857%7CUnknown
> %7CTWFpbGZsb
> >
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%
> >
> 7C3000%7C%7C%7C&sdata=KaPkvBSHWbn1cKBRzyk9H%2BuNDll%2Bq3
> wPfR3SFVAL
> > wkU%3D&reserved=0
> > +- AMD:
> >
> +https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> .amd.com%2Fsystem%2Ffiles%2FTechDocs%2F48882_IOMMU.pdf&data
> =05%7C01%7Calexander.deucher%40amd.com%7C929847a4b2524432d1a6
> 08da27a3c9b0%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
> 7865881851295857%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&
> amp;sdata=QFq8HBY6cm2XaZ50u5LQA3RX3NX%2BwVsMqfJFDH9bFDs%3D&a
> mp;reserved=0 + This guide gives a quick cheat sheet for some basic
> understanding.
> > +
> > +Basic stuff
> > +-----------
> > +
> > +ACPI enumerates and lists the different IOMMUs on the platform, and
> > +device scope relationships between devices and which IOMMU controls
> > +them.
> > +
> > +Some ACPI Keywords:
> > +
> > +- DMAR - Intel DMA Remapping table
> > +- DRHD - Intel DMA Remapping Hardware Unit Definition
> > +- RMRR - Intel Reserved Memory Region Reporting Structure
> > +- IVRS - AMD I/O Virtualization Reporting Structure
> > +- IVDB - AMD I/O Virtualization Definition Block
> > +- IVHD - AMD I/O Virtualization Hardware Definition
> > +
> > +What is Intel RMRR?
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +There are some devices the BIOS controls, for e.g USB devices to
> > +perform
> > +PS2 emulation. The regions of memory used for these devices are
> > +marked reserved in the e820 map. When we turn on DMA translation,
> DMA
> > +to those regions will fail. Hence BIOS uses RMRR to specify these
> > +regions along
> > with +devices that need to access these regions. OS is expected to
> > setup
> > +unity mappings for these regions for these devices to access these
> > regions. +
> > +What is AMD IVRS?
> > +^^^^^^^^^^^^^^^^^
> > +
> > +The architecture defines an ACPI-compatible data structure called an
> > +I/O Virtualization Reporting Structure (IVRS) that is used to convey
> > information +related to I/O virtualization to system software. The
> > IVRS describes the +configuration and capabilities of the IOMMUs
> > contained in the platform as +well as information about the devices
> > that each IOMMU virtualizes. +
> > +The IVRS provides information about the following:
> > +
> > +- IOMMUs present in the platform including their capabilities and
> > +proper
> > configuration +- System I/O topology relevant to each IOMMU
> > +- Peripheral devices that cannot be otherwise enumerated
> > +- Memory regions used by SMI/SMM, platform firmware, and platform
> > hardware. These are generally exclusion ranges to be configured by
> > system software. + +How is an I/O Virtual Address (IOVA) generated?
> > +-----------------------------------------------
> > +
> > +Well behaved drivers call dma_map_*() calls before sending command to
> > device +that needs to perform DMA. Once DMA is completed and mapping
> > is no longer +required, driver performs dma_unmap_*() calls to unmap
> > the region. +
> > +Some exceptions to IOVAs
> > +------------------------
> > +
> > +Interrupt ranges are not address translated, (0xfee00000 - 0xfeefffff).
> > +The same is true for peer to peer transactions. Hence we reserve the
> > +address from PCI MMIO ranges so they are not allocated for IOVA
> > addresses. +
> > +Graphics Problems?
> > +------------------
> > +
> > +If you encounter issues with integrated graphics devices, you can try
> > +adding the option "iommu.passthrough=1", or the equivalent
> > +"iommu=pt", to the kernel command line to use a 1:1 mapping for the
> > +IOMMU in general. On Intel you can also try "intel_iommu=igfx_off"
> > +to turn off translation specifically for the integrated graphics
> > +engine only. If this fixes anything, please ensure you file a bug reporting
> the problem.
> > +
> > +Fault reporting
> > +---------------
> > +When errors are reported, the IOMMU signals via an interrupt. The
> > +fault reason and device that caused it is printed on the console.
> > +
> > +
> > +Kernel Log Samples
> > +------------------
> > +
> > +Intel Boot Messages
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +Something like this gets printed indicating presence of DMAR tables
> > +in ACPI:
> > +
> > +::
> > +
> > + ACPI: DMAR (v001 A M I OEMDMAR 0x00000001 MSFT
> 0x00000097) @
> > 0x000000007f5b5ef0 +
> > +When DMAR is being processed and initialized by ACPI, prints DMAR
> > locations +and any RMRR's processed:
> > +
> > +::
> > +
> > + ACPI DMAR:Host address width 36
> > + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> > + ACPI DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> > + ACPI DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> > + ACPI DMAR:RMRR base: 0x00000000000ed000 end:
> 0x00000000000effff
> > + ACPI DMAR:RMRR base: 0x000000007f600000 end:
> 0x000000007fffffff
> > +
> > +When DMAR is enabled for use, you will notice:
> > +
> > +::
> > +
> > + PCI-DMA: Using DMAR IOMMU
> > +
> > +Intel Fault reporting
> > +^^^^^^^^^^^^^^^^^^^^^
> > +
> > +::
> > +
> > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > + DMAR:[fault reason 05] PTE Write access is not set
> > + DMAR:[DMA Write] Request device [00:02.0] fault addr 6df084000
> > + DMAR:[fault reason 05] PTE Write access is not set
> > +
> > +AMD Boot Messages
> > +^^^^^^^^^^^^^^^^^
> > +
> > +Something like this gets printed indicating presence of the IOMMU:
> > +
> > +::
> > +
> > + AMD-Vi: Found IOMMU cap 0x40
> > + AMD-Vi: Extended features (0x4f77ef22294ada): PPR NX GT IA GA PC
> > GA_vAPIC
> > + AMD-Vi: Interrupt remapping enabled
> > + AMD-Vi: Virtual APIC enabled
> > +
> > +AMD Fault reporting
> > +^^^^^^^^^^^^^^^^^^^
> > +
> > +::
> > +
> > + AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007
> > address=0xffffc02000 flags=0x0000]
> > + AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0
> domain=0x0007
> > address=0xffffc02000 flags=0x0000]
>
>
> Thanks,
>
> Jacob
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
2022-04-22 20:06 ` Alex Deucher
@ 2022-04-28 8:16 ` Joerg Roedel
-1 siblings, 0 replies; 12+ messages in thread
From: Joerg Roedel @ 2022-04-28 8:16 UTC (permalink / raw)
To: Alex Deucher, corbet
Cc: linux-doc, linux-kernel, corbet, hpa, x86, dave.hansen, bp,
mingo, tglx, Suravee.Suthikulpanit, will, iommu, robin.murphy,
Vasant.Hegde
On Fri, Apr 22, 2022 at 04:06:07PM -0400, Alex Deucher wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
Acked-by: Joerg Roedel <jroedel@suse.de>
Jonathan, will you merge that through the documentation tree?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-28 8:16 ` Joerg Roedel
0 siblings, 0 replies; 12+ messages in thread
From: Joerg Roedel @ 2022-04-28 8:16 UTC (permalink / raw)
To: Alex Deucher, corbet
Cc: dave.hansen, corbet, robin.murphy, x86, linux-doc, linux-kernel,
Vasant.Hegde, iommu, mingo, bp, hpa, tglx, will
On Fri, Apr 22, 2022 at 04:06:07PM -0400, Alex Deucher wrote:
> Add preliminary documentation for AMD IOMMU and combine
> with the existing Intel IOMMU documentation and clean
> up and modernize some of the existing documentation to
> align with the current state of the kernel.
>
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>
> V2: Incorporate feedback from Robin to clarify IOMMU vs DMA engine (e.g.,
> a device) and document proper DMA API. Also correct the fact that
> the AMD IOMMU is not limited to managing PCI devices.
> v3: Fix spelling and rework text as suggested by Vasant
> v4: Combine Intel and AMD documents into a single document as suggested
> by Dave Hansen
> v5: Clarify that keywords are related to ACPI, grammatical fixes
> v6: Make more stuff common based on feedback from Robin
>
> Documentation/x86/index.rst | 2 +-
> Documentation/x86/intel-iommu.rst | 115 ------------------------
> Documentation/x86/iommu.rst | 143 ++++++++++++++++++++++++++++++
> 3 files changed, 144 insertions(+), 116 deletions(-)
> delete mode 100644 Documentation/x86/intel-iommu.rst
> create mode 100644 Documentation/x86/iommu.rst
Acked-by: Joerg Roedel <jroedel@suse.de>
Jonathan, will you merge that through the documentation tree?
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
2022-04-28 8:16 ` Joerg Roedel
@ 2022-04-28 14:06 ` Jonathan Corbet
-1 siblings, 0 replies; 12+ messages in thread
From: Jonathan Corbet @ 2022-04-28 14:06 UTC (permalink / raw)
To: Joerg Roedel, Alex Deucher
Cc: linux-doc, linux-kernel, hpa, x86, dave.hansen, bp, mingo, tglx,
Suravee.Suthikulpanit, will, iommu, robin.murphy, Vasant.Hegde
Joerg Roedel <joro@8bytes.org> writes:
> Acked-by: Joerg Roedel <jroedel@suse.de>
>
> Jonathan, will you merge that through the documentation tree?
Done.
Thanks,
jon
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] Documentation: x86: rework IOMMU documentation
@ 2022-04-28 14:06 ` Jonathan Corbet
0 siblings, 0 replies; 12+ messages in thread
From: Jonathan Corbet @ 2022-04-28 14:06 UTC (permalink / raw)
To: Joerg Roedel, Alex Deucher
Cc: dave.hansen, linux-doc, robin.murphy, x86, linux-kernel,
Vasant.Hegde, iommu, mingo, bp, hpa, tglx, will
Joerg Roedel <joro@8bytes.org> writes:
> Acked-by: Joerg Roedel <jroedel@suse.de>
>
> Jonathan, will you merge that through the documentation tree?
Done.
Thanks,
jon
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-04-28 14:12 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-22 20:06 [PATCH] Documentation: x86: rework IOMMU documentation Alex Deucher via iommu
2022-04-22 20:06 ` Alex Deucher
2022-04-22 21:26 ` Robin Murphy
2022-04-22 21:26 ` Robin Murphy
2022-04-26 16:44 ` Jacob Pan
2022-04-26 16:44 ` Jacob Pan
2022-04-27 17:12 ` Deucher, Alexander
2022-04-27 17:12 ` Deucher, Alexander via iommu
2022-04-28 8:16 ` Joerg Roedel
2022-04-28 8:16 ` Joerg Roedel
2022-04-28 14:06 ` Jonathan Corbet
2022-04-28 14:06 ` Jonathan Corbet
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.