[git pull] IOMMU Updates for v3.7-rc1

[git pull] IOMMU Updates for v3.7-rc1

[Joerg Roedel]

[-- Attachment #1: Type: text/plain, Size: 3746 bytes --]

Hi Linus,

The following changes since commit a0d271cbfed1dd50278c6b06bead3d00ba0a88f9:

  Linux 3.6 (2012-09-30 16:47:46 -0700)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git tags/iommu-updates-v3.7-rc1

for you to fetch changes up to 009487258399cb4f431992919fa0f386d1b74ceb:

  Merge branches 'dma-debug', 'iommu/fixes', 'arm/tegra', 'arm/exynos', 'x86/amd', 'x86/vt-d' and 'x86/amd-irq-remapping' into next (2012-10-02 14:40:03 +0200)

----------------------------------------------------------------

IOMMU Updates for Linux v3.7-rc1

This time the IOMMU updates contain a bunch of fixes and cleanups to
various IOMMU drivers and the DMA debug code. New features are the
code for IRQ remapping support with the AMD IOMMU (preperation for that
was already merged in the last release) and a debugfs interface to
export some statistics in the NVidia Tegra IOMMU driver.

----------------------------------------------------------------
Alex Williamson (1):
      intel-iommu: Default to non-coherent for domains unattached to iommus

Borislav Petkov (1):
      iommu/amd: Fix features reporting

Dan Carpenter (1):
      iommu/amd: Fix possible use after free in get_irq_table()

Frank Arnold (1):
      iommu/amd: Fix some typos

Hiroshi Doyu (2):
      iommu/tegra: smmu: debugfs for TLB/PTC statistics
      iommu/tegra: smmu: Use debugfs_create_dir for directory

Joerg Roedel (18):
      iommu/amd: Keep track of HPET and IOAPIC device ids
      iommu/amd: Add slab-cache for irq remapping tables
      iommu/amd: Allocate data structures to keep track of irq remapping tables
      iommu/amd: Check if IOAPIC information is correct
      iommu/amd: Split device table initialization into irq and dma part
      iommu/amd: Make sure IOMMU is not considered to translate itself
      iommu/amd: Add IRTE invalidation routine
      iommu/amd: Add routines to manage irq remapping tables
      iommu/amd: Add IOAPIC remapping routines
      iommu/amd: Implement MSI routines for interrupt remapping
      iommu/amd: Add call-back routine for HPET MSI
      iommu/amd: Add initialization routines for AMD interrupt remapping
      iommu/amd: Make sure irq remapping still works on dma init failure
      iommu/irq: Use amd_iommu_irq_ops if supported
      iommu/amd: Print message to system log when irq remapping is enabled
      iommu/amd: Report irq remapping through IOMMU-API
      iommu/amd: Remove obsolete comment line
      Merge branches 'dma-debug', 'iommu/fixes', 'arm/tegra', 'arm/exynos', 'x86/amd', 'x86/vt-d' and 'x86/amd-irq-remapping' into next

Shuah Khan (1):
      dma-debug: Remove local BUS_NOTIFY_UNBOUND_DRIVER define

Stephen Warren (1):
      dma: tegra: move smmu.h into SMMU driver

Wei Yongjun (2):
      iommu/amd: remove duplicated include from amd_iommu_init.c
      iommu/exynos: use list_del_init instead of list_del/INIT_LIST_HEAD

 arch/arm/mach-tegra/include/mach/smmu.h |   63 ----
 drivers/iommu/Kconfig                   |    2 +-
 drivers/iommu/amd_iommu.c               |  514 ++++++++++++++++++++++++++++++-
 drivers/iommu/amd_iommu_init.c          |  253 +++++++++++++--
 drivers/iommu/amd_iommu_proto.h         |    8 +
 drivers/iommu/amd_iommu_types.h         |   59 +++-
 drivers/iommu/exynos-iommu.c            |    3 +-
 drivers/iommu/intel-iommu.c             |    4 +-
 drivers/iommu/irq_remapping.c           |    5 +
 drivers/iommu/irq_remapping.h           |    6 +
 drivers/iommu/tegra-smmu.c              |  261 +++++++++++++++-
 lib/dma-debug.c                         |    5 -
 12 files changed, 1071 insertions(+), 112 deletions(-)
 delete mode 100644 arch/arm/mach-tegra/include/mach/smmu.h

Please pull.

Regards,

	Joerg


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [git pull] IOMMU Updates for v3.7-rc1

[Andrew Oakley]

On Fri, 5 Oct 2012 17:23:12 +0200
Joerg Roedel <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org> wrote:
> IOMMU Updates for Linux v3.7-rc1
> 
> This time the IOMMU updates contain a bunch of fixes and cleanups to
> various IOMMU drivers and the DMA debug code. New features are the
> code for IRQ remapping support with the AMD IOMMU (preperation for
> that was already merged in the last release) and a debugfs interface
> to export some statistics in the NVidia Tegra IOMMU driver.

I'm getting a kernel panic here when I try to boot from master
(9aadf84) in in your IOMMU tree.  I'll try booting the tagged commit in
case it's the merge that caused the problem but I'm not particularly
hopeful.

I have got interrupt remapping support enabled and the IOMMU enabled,
the chipset is an RD890 (890FX).

I think the relevant bit of the kernel output is here:

ACPI: Core revision 20120711
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
Pid: 1, comm: swapper/0 Not tainted 3.6.0+ #2
Call Trace:
 [<ffffffff8163f617>] ? panic+0xc5/0x1d0
 [<ffffffff8163f76e>] ? printk+0x4c/0x51
 [<ffffffff81c96b52>] ? setup_IO_APIC+0x325/0x710
 [<ffffffff81c947b5>] ? native_smp_prepare_cpus+0x2b9/0x334
 [<ffffffff81c87b53>] ? kernel_init+0x59/0x190
 [<ffffffff81643b84>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81c87afa>] ? start_kernel+0x2c7/0x2c7
 [<ffffffff81643b80>] ? gs_change+0x13/0x13

There is a patch to fix a similar problem (32ab31e) for the Intel IRQ
remapping code but I don't know if this is likely to be the same issue
or not.

I've posted the full kernel log, kernel config and lspci output in case
they are useful:
http://adoakley.name/amd-iommu/dmesg.txt
http://adoakley.name/amd-iommu/kernel_config.txt
http://adoakley.name/amd-iommu/lspci.txt

Thanks

-- 
Andrew Oakley

Re: [git pull] IOMMU Updates for v3.7-rc1

[Joerg Roedel]

Hi Andrew,

On Tue, Oct 09, 2012 at 10:45:32PM +0100, Andrew Oakley wrote:
> On Fri, 5 Oct 2012 17:23:12 +0200
> Joerg Roedel <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org> wrote:

> ACPI: Core revision 20120711
> ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
> Pid: 1, comm: swapper/0 Not tainted 3.6.0+ #2
> Call Trace:
>  [<ffffffff8163f617>] ? panic+0xc5/0x1d0
>  [<ffffffff8163f76e>] ? printk+0x4c/0x51
>  [<ffffffff81c96b52>] ? setup_IO_APIC+0x325/0x710
>  [<ffffffff81c947b5>] ? native_smp_prepare_cpus+0x2b9/0x334
>  [<ffffffff81c87b53>] ? kernel_init+0x59/0x190
>  [<ffffffff81643b84>] ? kernel_thread_helper+0x4/0x10
>  [<ffffffff81c87afa>] ? start_kernel+0x2c7/0x2c7
>  [<ffffffff81643b80>] ? gs_change+0x13/0x13
> 
> There is a patch to fix a similar problem (32ab31e) for the Intel IRQ
> remapping code but I don't know if this is likely to be the same issue
> or not.

Can you please boot with 'amd_iommu_dump nointremap' on your kernel
command line and post dmesg? I guess your ACPI table is broken in some
way.

The problem seen on VT-d exists on AMD too, but I have a workaround for
that in the code already. This must be something else...

Thanks,

	Joerg

Re: [git pull] IOMMU Updates for v3.7-rc1

[Andrew Oakley]

On Wed, 10 Oct 2012 09:30:22 +0200
Joerg Roedel <joro@8bytes.org> wrote:

> Hi Andrew,
> 
> On Tue, Oct 09, 2012 at 10:45:32PM +0100, Andrew Oakley wrote:
> > On Fri, 5 Oct 2012 17:23:12 +0200
> > Joerg Roedel <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org> wrote:
> 
> > ACPI: Core revision 20120711
> > ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> > Kernel panic - not syncing: timer doesn't work through
> > Interrupt-remapped IO-APIC Pid: 1, comm: swapper/0 Not tainted
> > 3.6.0+ #2 Call Trace:
> >  [<ffffffff8163f617>] ? panic+0xc5/0x1d0
> >  [<ffffffff8163f76e>] ? printk+0x4c/0x51
> >  [<ffffffff81c96b52>] ? setup_IO_APIC+0x325/0x710
> >  [<ffffffff81c947b5>] ? native_smp_prepare_cpus+0x2b9/0x334
> >  [<ffffffff81c87b53>] ? kernel_init+0x59/0x190
> >  [<ffffffff81643b84>] ? kernel_thread_helper+0x4/0x10
> >  [<ffffffff81c87afa>] ? start_kernel+0x2c7/0x2c7
> >  [<ffffffff81643b80>] ? gs_change+0x13/0x13
> > 
> > There is a patch to fix a similar problem (32ab31e) for the Intel
> > IRQ remapping code but I don't know if this is likely to be the
> > same issue or not.
> 
> Can you please boot with 'amd_iommu_dump nointremap' on your kernel
> command line and post dmesg? I guess your ACPI table is broken in some
> way.

http://adoakley.name/amd-iommu/dmesg_amd_iommu_dump.txt

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
AMD-Vi:        mmio-addr: 00000000fec30000
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:00.2
AMD-Vi:   DEV_SELECT			 devid: 00:02.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 01:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 01:00.1
AMD-Vi:   DEV_SELECT			 devid: 00:04.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 02:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:05.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:06.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:07.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 05:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:09.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 06:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:0b.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 07:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:11.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:12.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:12.2
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:13.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:13.2
AMD-Vi:   DEV_SELECT			 devid: 00:14.0 flags: d7
AMD-Vi:   DEV_SELECT			 devid: 00:14.2 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:14.3 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:14.4 flags: 00
AMD-Vi:   DEV_ALIAS_RANGE		 devid: 08:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END		 devid: 08:1f.7
AMD-Vi:   DEV_SELECT			 devid: 00:14.5 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:15.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 09:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 09:1f.7
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:16.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:16.2
AMD-Vi:   DEV_SPECIAL(HPET[0])		devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(IOAPIC[2])		devid: 00:00.1


-- 
Andrew Oakley

Re: [git pull] IOMMU Updates for v3.7-rc1

[Joerg Roedel]

On Wed, Oct 10, 2012 at 09:58:23AM +0100, Andrew Oakley wrote:
> On Wed, 10 Oct 2012 09:30:22 +0200
> Joerg Roedel <joro@8bytes.org> wrote:

> AMD-Vi:   DEV_SPECIAL(IOAPIC[2])		devid: 00:00.1

Okay, got it. This is the problem. The device id seems to be wrong. The
BIOS should configure your IOAPIC as device id 00:14.0 and not 00:00.1.
The current checking code does not check for this particular situation.
But the attached diff should change that. Can you please test it and
reports if the machine boots for you? Interrupt remapping will be
disabled automatically, though.

diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index 18b0d99..81837b0 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -1599,21 +1599,46 @@ static void __init free_on_init_error(void)
 #endif
 }
 
+/* SB IOAPIC is always on this device in AMD systems */
+#define IOAPIC_SB_DEVID		((0x00 << 8) | PCI_DEVFN(0x14, 0))
+
 static bool __init check_ioapic_information(void)
 {
+	bool ret, has_sb_ioapic;
 	int idx;
 
-	for (idx = 0; idx < nr_ioapics; idx++) {
-		int id = mpc_ioapic_id(idx);
+	has_sb_ioapic = false;
+	ret           = false;
 
-		if (get_ioapic_devid(id) < 0) {
-			pr_err(FW_BUG "AMD-Vi: IO-APIC[%d] not in IVRS table\n", id);
-			pr_err("AMD-Vi: Disabling interrupt remapping due to BIOS Bug\n");
-			return false;
+	for (idx = 0; idx < nr_ioapics; idx++) {
+		int devid, id = mpc_ioapic_id(idx);
+
+		devid = get_ioapic_devid(id);
+		if (devid < 0) {
+			pr_err(FW_BUG "AMD-Vi: IOAPIC[%d] not in IVRS table\n", id);
+			ret = false;
+		} else if (devid == IOAPIC_SB_DEVID) {
+			has_sb_ioapic = true;
+			ret           = true;
 		}
 	}
 
-	return true;
+	if (!has_sb_ioapic) {
+		/*
+		 * We expect the SB IOAPIC to be listed in the IVRS
+		 * table. The system timer is connected to the SB IOAPIC
+		 * and if we don't have it in the list the system will
+		 * panic at boot time.  This situation usually happens
+		 * when the BIOS is buggy and provides us the wrong
+		 * device id for the IOAPIC in the system.
+		 */
+		pr_err(FW_BUG "AMD-Vi: No southbridge IOAPIC found in IVRS table\n");
+	}
+
+	if (!ret)
+		pr_err("AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)\n");
+
+	return ret;
 }
 
 static void __init free_dma_resources(void)

Re: [git pull] IOMMU Updates for v3.7-rc1

[Andrew Oakley]

On Wed, 10 Oct 2012 15:32:49 +0200
Joerg Roedel <joro@8bytes.org> wrote:

> On Wed, Oct 10, 2012 at 09:58:23AM +0100, Andrew Oakley wrote:
> > On Wed, 10 Oct 2012 09:30:22 +0200
> > Joerg Roedel <joro@8bytes.org> wrote:
> 
> > AMD-Vi:   DEV_SPECIAL(IOAPIC[2])		devid: 00:00.1
> 
> Okay, got it. This is the problem. The device id seems to be wrong.
> The BIOS should configure your IOAPIC as device id 00:14.0 and not
> 00:00.1. The current checking code does not check for this particular
> situation. But the attached diff should change that. Can you please
> test it and reports if the machine boots for you? Interrupt remapping
> will be disabled automatically, though.

Yes, that detects the problem and disables the interrupt remapping (the
kernel boots).

If I modify the devid for the IOAPIC in init_iommu_from_acpi then it
also seems to work.  I haven't managed to get vfio/qemu working yet but
it seems to be unrelated ("Invalid ROM contents", same thing happens
when I try to dump the rom with stable kernels).

Thanks

-- 
Andrew Oakley

Re: [git pull] IOMMU Updates for v3.7-rc1

[Andrew Oakley]

On Wed, 10 Oct 2012 20:11:31 +0100
Andrew Oakley <andrew@ado.is-a-geek.net> wrote:
> If I modify the devid for the IOAPIC in init_iommu_from_acpi then it
> also seems to work.  I haven't managed to get vfio/qemu working yet
> but it seems to be unrelated ("Invalid ROM contents", same thing
> happens when I try to dump the rom with stable kernels).

I've now managed to get a network card passed through and working
properly in a guest.  The bug has been reported to the vendor so
hopefully it will get fixed in a BIOS update.

I'm not sure if the fix I've got is safe or if it's really a good idea
to enable interrupt remapping with a broken BIOS.

Thanks

-- 
Andrew Oakley

[PATCH] iommu/amd: Workaround wrong IOAPIC devid in IVRS.

[andrew]

From: Andrew Oakley <andrew@ado.is-a-geek.net>

The IOAPIC always has the same device id but is sometimes reported in
the IVRS table with a different id.  This patch allows interrupt
remapping to work on devices with this bug.
---
 drivers/iommu/amd_iommu_init.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index 18b0d99..ca266b9 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -751,6 +751,9 @@ static void __init set_device_exclusion_range(u16 devid, struct ivmd_header *m)
 	}
 }
 
+/* SB IOAPIC is always on this device in AMD systems */
+#define IOAPIC_SB_DEVID		((0x00 << 8) | PCI_DEVFN(0x14, 0))
+
 /*
  * Takes a pointer to an AMD IOMMU entry in the ACPI table and
  * initializes the hardware and our data structures with it.
@@ -928,6 +931,12 @@ static int __init init_iommu_from_acpi(struct amd_iommu *iommu,
 				    PCI_SLOT(devid),
 				    PCI_FUNC(devid));
 
+			if (type == IVHD_SPECIAL_IOAPIC &&
+			    devid != IOAPIC_SB_DEVID) {
+				pr_err(FW_BUG "AMD-Vi: wrong IOAPIC devid\n");
+				devid = IOAPIC_SB_DEVID;
+			}
+
 			set_dev_entry_from_acpi(iommu, devid, e->flags, 0);
 			ret = add_special_device(type, handle, devid);
 			if (ret)
-- 
1.7.8.6

Re: [git pull] IOMMU Updates for v3.7-rc1

[Joerg Roedel]

On Wed, Oct 10, 2012 at 11:02:31PM +0100, Andrew Oakley wrote:
> On Wed, 10 Oct 2012 20:11:31 +0100
> Andrew Oakley <andrew@ado.is-a-geek.net> wrote:

> I'm not sure if the fix I've got is safe or if it's really a good idea
> to enable interrupt remapping with a broken BIOS.

Sadly it is not. Your patch works for your particular case but is not
generally applicable. There are indeed systems with 2 or more IOAPICs
and they all use different device-ids.

So, typically, there is one IOAPIC per SB, like in your case. But on
RD890 the chipset NB also includes an IOAPIC (which uses
device-id 00:00.1). This IOAPIC exists on your system too but is
disabled by the BIOS because the board does not use it.

So, in the end, device-id 00:00.1 might also be correct and the IVRS
entry could refer to the NB IOAPIC. We don't have any reliable
indication which IOAPIC is described in the IVRS table.

Cheer,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632