All of lore.kernel.org
 help / color / mirror / Atom feed
* Failure to shutdown/reboot with intel_iommu=on
@ 2020-05-06 14:46 Uwe Kleine-König
  2020-05-08 15:07 ` Joerg Roedel
  0 siblings, 1 reply; 10+ messages in thread
From: Uwe Kleine-König @ 2020-05-06 14:46 UTC (permalink / raw)
  To: linux-kernel; +Cc: rafael.j.wysocki, jroedel, tglx, x86

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

Hello,

On my Lenovo T460p I cannot shutdown and reboot when the iommu is
enabled. This is using linux 5.2.7 as provided by Debian, 5.6.4 has the
same problem. Suspend/resume also fails; I suspect this is the same
issue.

When requesting power off the kernel messages just end with:

	sd 0:0:0:0: [sda] Synchronizing SCSI cache
	sd 0:0:0:0: [sda] Stopping disk
	e1000e: EEE TX LPI TIMER: 00000011
	ACPI: Preparing to enter system sleep state S5
	reboot: Power down
	acpi_power_off called

(photo at https://www.kleine-koenig.org/tmp/uklsiommu.jpg in case I
mistyped something. Full dmesg and lspci -vvv at
https://www.kleine-koenig.org/tmp/uklsiommu.tar.gz with and without
iommu enabled.)

With the iommu disabled (CONFIG_INTEL_IOMMU_DEFAULT_ON unset or
intel_iommu=off on cmdline) the machine just works as expected
(including working suspend/resume).

I already talked to tglx on irc but unfortunately no new insights
resulted from that.

Any ideas how to fix or continue debugging the issue?

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-06 14:46 Failure to shutdown/reboot with intel_iommu=on Uwe Kleine-König
@ 2020-05-08 15:07 ` Joerg Roedel
  2020-05-09  1:58   ` Lu Baolu
  2020-05-11 13:43   ` Lenny Szubowicz
  0 siblings, 2 replies; 10+ messages in thread
From: Joerg Roedel @ 2020-05-08 15:07 UTC (permalink / raw)
  To: Uwe Kleine-König; +Cc: linux-kernel, rafael.j.wysocki, tglx, x86, Lu Baolu

+ Baolu, Maintainer of Intel IOMMU

Baolu, does that ring any bells?

On Wed, May 06, 2020 at 04:46:02PM +0200, Uwe Kleine-König wrote:
> Hello,
> 
> On my Lenovo T460p I cannot shutdown and reboot when the iommu is
> enabled. This is using linux 5.2.7 as provided by Debian, 5.6.4 has the
> same problem. Suspend/resume also fails; I suspect this is the same
> issue.
> 
> When requesting power off the kernel messages just end with:
> 
> 	sd 0:0:0:0: [sda] Synchronizing SCSI cache
> 	sd 0:0:0:0: [sda] Stopping disk
> 	e1000e: EEE TX LPI TIMER: 00000011
> 	ACPI: Preparing to enter system sleep state S5
> 	reboot: Power down
> 	acpi_power_off called
> 
> (photo at https://www.kleine-koenig.org/tmp/uklsiommu.jpg in case I
> mistyped something. Full dmesg and lspci -vvv at
> https://www.kleine-koenig.org/tmp/uklsiommu.tar.gz with and without
> iommu enabled.)
> 
> With the iommu disabled (CONFIG_INTEL_IOMMU_DEFAULT_ON unset or
> intel_iommu=off on cmdline) the machine just works as expected
> (including working suspend/resume).
> 
> I already talked to tglx on irc but unfortunately no new insights
> resulted from that.
> 
> Any ideas how to fix or continue debugging the issue?
> 
> Best regards
> Uwe



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-08 15:07 ` Joerg Roedel
@ 2020-05-09  1:58   ` Lu Baolu
  2020-05-11 19:59     ` Uwe Kleine-König
  2020-05-11 13:43   ` Lenny Szubowicz
  1 sibling, 1 reply; 10+ messages in thread
From: Lu Baolu @ 2020-05-09  1:58 UTC (permalink / raw)
  To: Joerg Roedel, Uwe Kleine-König
  Cc: baolu.lu, linux-kernel, rafael.j.wysocki, tglx, x86

Hi Uwe,

Have you tried commenting out intel_disable_iommus() in
intel_iommu_shutdowan()?

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 0182cff2c7ac..532e62600f95 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4928,8 +4928,10 @@ void intel_iommu_shutdown(void)
         for_each_iommu(iommu, drhd)
                 iommu_disable_protect_mem_regions(iommu);

+#if 0
         /* Make sure the IOMMUs are switched off */
         intel_disable_iommus();
+#endif

         up_write(&dmar_global_lock);
  }

Best regards,
baolu

On 5/8/20 11:07 PM, Joerg Roedel wrote:
> + Baolu, Maintainer of Intel IOMMU
> 
> Baolu, does that ring any bells?
> 
> On Wed, May 06, 2020 at 04:46:02PM +0200, Uwe Kleine-König wrote:
>> Hello,
>>
>> On my Lenovo T460p I cannot shutdown and reboot when the iommu is
>> enabled. This is using linux 5.2.7 as provided by Debian, 5.6.4 has the
>> same problem. Suspend/resume also fails; I suspect this is the same
>> issue.
>>
>> When requesting power off the kernel messages just end with:
>>
>> 	sd 0:0:0:0: [sda] Synchronizing SCSI cache
>> 	sd 0:0:0:0: [sda] Stopping disk
>> 	e1000e: EEE TX LPI TIMER: 00000011
>> 	ACPI: Preparing to enter system sleep state S5
>> 	reboot: Power down
>> 	acpi_power_off called
>>
>> (photo at https://www.kleine-koenig.org/tmp/uklsiommu.jpg in case I
>> mistyped something. Full dmesg and lspci -vvv at
>> https://www.kleine-koenig.org/tmp/uklsiommu.tar.gz with and without
>> iommu enabled.)
>>
>> With the iommu disabled (CONFIG_INTEL_IOMMU_DEFAULT_ON unset or
>> intel_iommu=off on cmdline) the machine just works as expected
>> (including working suspend/resume).
>>
>> I already talked to tglx on irc but unfortunately no new insights
>> resulted from that.
>>
>> Any ideas how to fix or continue debugging the issue?
>>
>> Best regards
>> Uwe
> 
> 

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-08 15:07 ` Joerg Roedel
  2020-05-09  1:58   ` Lu Baolu
@ 2020-05-11 13:43   ` Lenny Szubowicz
  2020-05-11 19:58     ` Uwe Kleine-König
  2020-05-12 13:34     ` Joerg Roedel
  1 sibling, 2 replies; 10+ messages in thread
From: Lenny Szubowicz @ 2020-05-11 13:43 UTC (permalink / raw)
  To: Joerg Roedel, Uwe Kleine-König
  Cc: linux-kernel, rafael.j.wysocki, tglx, x86, Lu Baolu

On 5/8/20 11:07 AM, Joerg Roedel wrote:
> + Baolu, Maintainer of Intel IOMMU
> 
> Baolu, does that ring any bells?
> 
> On Wed, May 06, 2020 at 04:46:02PM +0200, Uwe Kleine-König wrote:
>> Hello,
>>
>> On my Lenovo T460p I cannot shutdown and reboot when the iommu is
>> enabled. This is using linux 5.2.7 as provided by Debian, 5.6.4 has the
>> same problem. Suspend/resume also fails; I suspect this is the same
>> issue.
>>
>> When requesting power off the kernel messages just end with:
>>
>> 	sd 0:0:0:0: [sda] Synchronizing SCSI cache
>> 	sd 0:0:0:0: [sda] Stopping disk
>> 	e1000e: EEE TX LPI TIMER: 00000011
>> 	ACPI: Preparing to enter system sleep state S5
>> 	reboot: Power down
>> 	acpi_power_off called
>>
>> (photo at https://www.kleine-koenig.org/tmp/uklsiommu.jpg in case I
>> mistyped something. Full dmesg and lspci -vvv at
>> https://www.kleine-koenig.org/tmp/uklsiommu.tar.gz with and without
>> iommu enabled.)
>>
>> With the iommu disabled (CONFIG_INTEL_IOMMU_DEFAULT_ON unset or
>> intel_iommu=off on cmdline) the machine just works as expected
>> (including working suspend/resume).
>>
>> I already talked to tglx on irc but unfortunately no new insights
>> resulted from that.
>>> Any ideas how to fix or continue debugging the issue?
>>
>> Best regards
>> Uwe
> 
> 

I suspect that you have TPM 2.x functionality enabled in the BIOS/firmware.

Unless you are actually using the TPM, try setting it to TPM 1.2 mode.
I've seen an incompatiblity on other Lenovo laptops between using the
IOMMU, TPM 2.x implementation in firmware, and shutdown/suspend.

                        -Lenny.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-11 13:43   ` Lenny Szubowicz
@ 2020-05-11 19:58     ` Uwe Kleine-König
  2020-05-12 13:34     ` Joerg Roedel
  1 sibling, 0 replies; 10+ messages in thread
From: Uwe Kleine-König @ 2020-05-11 19:58 UTC (permalink / raw)
  To: Lenny Szubowicz, Joerg Roedel
  Cc: linux-kernel, rafael.j.wysocki, tglx, x86, Lu Baolu


[-- Attachment #1.1: Type: text/plain, Size: 486 bytes --]

Hello Lenny,

On 5/11/20 3:43 PM, Lenny Szubowicz wrote:
> I suspect that you have TPM 2.x functionality enabled in the BIOS/firmware.

Indeed.

> Unless you are actually using the TPM, try setting it to TPM 1.2 mode.
> I've seen an incompatiblity on other Lenovo laptops between using the
> IOMMU, TPM 2.x implementation in firmware, and shutdown/suspend.

When setting it to TPM 1.2 reboot works again. Didn't test poweroff and
suspend/resume yet.

Best regards
Uwe


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-09  1:58   ` Lu Baolu
@ 2020-05-11 19:59     ` Uwe Kleine-König
  2020-05-11 20:16       ` Uwe Kleine-König
  0 siblings, 1 reply; 10+ messages in thread
From: Uwe Kleine-König @ 2020-05-11 19:59 UTC (permalink / raw)
  To: Lu Baolu, Joerg Roedel; +Cc: linux-kernel, rafael.j.wysocki, tglx, x86


[-- Attachment #1.1: Type: text/plain, Size: 833 bytes --]

On 5/9/20 3:58 AM, Lu Baolu wrote:
> Hi Uwe,
> 
> Have you tried commenting out intel_disable_iommus() in
> intel_iommu_shutdowan()?
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 0182cff2c7ac..532e62600f95 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -4928,8 +4928,10 @@ void intel_iommu_shutdown(void)
>         for_each_iommu(iommu, drhd)
>                 iommu_disable_protect_mem_regions(iommu);
> 
> +#if 0
>         /* Make sure the IOMMUs are switched off */
>         intel_disable_iommus();
> +#endif
> 
>         up_write(&dmar_global_lock);
>  }

I just tested that and it didn't help. The machine still hangs with the
same symptoms as reported before.

Best regards
Uwe


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-11 19:59     ` Uwe Kleine-König
@ 2020-05-11 20:16       ` Uwe Kleine-König
  0 siblings, 0 replies; 10+ messages in thread
From: Uwe Kleine-König @ 2020-05-11 20:16 UTC (permalink / raw)
  To: Lu Baolu, Joerg Roedel; +Cc: linux-kernel, rafael.j.wysocki, tglx, x86

[-- Attachment #1: Type: text/plain, Size: 2210 bytes --]

Hello again,

On Mon, May 11, 2020 at 09:59:31PM +0200, Uwe Kleine-König wrote:
> On 5/9/20 3:58 AM, Lu Baolu wrote:
> > Hi Uwe,
> > 
> > Have you tried commenting out intel_disable_iommus() in
> > intel_iommu_shutdowan()?
> > 
> > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> > index 0182cff2c7ac..532e62600f95 100644
> > --- a/drivers/iommu/intel-iommu.c
> > +++ b/drivers/iommu/intel-iommu.c
> > @@ -4928,8 +4928,10 @@ void intel_iommu_shutdown(void)
> >         for_each_iommu(iommu, drhd)
> >                 iommu_disable_protect_mem_regions(iommu);
> > 
> > +#if 0
> >         /* Make sure the IOMMUs are switched off */
> >         intel_disable_iommus();
> > +#endif
> > 
> >         up_write(&dmar_global_lock);
> >  }
> 
> I just tested that and it didn't help. The machine still hangs with the
> same symptoms as reported before.

I patched the file a bit differently:

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index ef0a5246700e..b76acae6a6ac 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4922,16 +4922,24 @@ void intel_iommu_shutdown(void)
 	if (no_iommu || dmar_disabled)
 		return;
 
+	pr_warn("%s:%d\n", __func__, __LINE__);
 	down_write(&dmar_global_lock);
 
+	pr_warn("%s:%d\n", __func__, __LINE__);
 	/* Disable PMRs explicitly here. */
-	for_each_iommu(iommu, drhd)
+	for_each_iommu(iommu, drhd) {
+		pr_warn("%s:%d\n", __func__, __LINE__);
 		iommu_disable_protect_mem_regions(iommu);
+		pr_warn("%s:%d\n", __func__, __LINE__);
+	}
 
+	pr_warn("%s:%d\n", __func__, __LINE__);
 	/* Make sure the IOMMUs are switched off */
 	intel_disable_iommus();
 
+	pr_warn("%s:%d\n", __func__, __LINE__);
 	up_write(&dmar_global_lock);
+	pr_warn("%s:%d\n", __func__, __LINE__);
 }
 
 static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev)

and the output shows that the for_each_iommu loop runs twice and the
last pr_warn is reached, too. So the hang doesn't occur in
intel_iommu_shutdown() but later.

I don't know enough about x86 and iommus to judge what that means or
even if this was a useful test.

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-11 13:43   ` Lenny Szubowicz
  2020-05-11 19:58     ` Uwe Kleine-König
@ 2020-05-12 13:34     ` Joerg Roedel
  2020-05-12 20:00       ` Lenny Szubowicz
  1 sibling, 1 reply; 10+ messages in thread
From: Joerg Roedel @ 2020-05-12 13:34 UTC (permalink / raw)
  To: Lenny Szubowicz
  Cc: Uwe Kleine-König, linux-kernel, rafael.j.wysocki, tglx, x86,
	Lu Baolu

On Mon, May 11, 2020 at 09:43:11AM -0400, Lenny Szubowicz wrote:
> I suspect that you have TPM 2.x functionality enabled in the BIOS/firmware.
> 
> Unless you are actually using the TPM, try setting it to TPM 1.2 mode.
> I've seen an incompatiblity on other Lenovo laptops between using the
> IOMMU, TPM 2.x implementation in firmware, and shutdown/suspend.

Interesting, has this been debugged further into the TPM code?


	Joerg

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-12 13:34     ` Joerg Roedel
@ 2020-05-12 20:00       ` Lenny Szubowicz
  2020-05-12 21:13         ` Joerg Roedel
  0 siblings, 1 reply; 10+ messages in thread
From: Lenny Szubowicz @ 2020-05-12 20:00 UTC (permalink / raw)
  To: Joerg Roedel
  Cc: Uwe Kleine-König, linux-kernel, rafael.j.wysocki, tglx, x86,
	Lu Baolu

On 5/12/20 9:34 AM, Joerg Roedel wrote:
> On Mon, May 11, 2020 at 09:43:11AM -0400, Lenny Szubowicz wrote:
>> I suspect that you have TPM 2.x functionality enabled in the BIOS/firmware.
>>
>> Unless you are actually using the TPM, try setting it to TPM 1.2 mode.
>> I've seen an incompatiblity on other Lenovo laptops between using the
>> IOMMU, TPM 2.x implementation in firmware, and shutdown/suspend.
> 
> Interesting, has this been debugged further into the TPM code?
> 
> 
> 	Joerg
> 

I believe the problem is in the Lenovo firmware and not in the kernel.

There are essentially two problems:
  1. TPM 2.0 doesn't work when the IOMMU is enabled
  2. Suspend/shutdown hangs when problem 1 is encountered on boot

Lenovo's firmware implementation of TPM 2.0 functionality on some of their
laptops uses DMA. When you ask the kernel to enable the IOMMU, this DMA
access is correctly blocked by the IOMMU hardware. If you look at your
dmesg log from when you have TPM 2.0 and the IOMMU enabled, there are
TPM timeout messages that indicate the inability to initialize and use
the TPM capability.

The hang on shutdown or S3 suspend appears to be in firmware, i.e.
after the kernel has transferred control back to the firmware.
It makes no difference if the kernel actively shuts down the IOMMU
before transferring control to the firmware on a suspend or shutdown.
The hang still occurs.

My guess is that the firmware wants to do some TPM related processing
on shutdown and suspend and can't handle the TPM state that exists
due to the startup failure. But that's just a guess. I don't know
what the firmware is actually doing.

Some Lenovo laptops provide an ACPI DMAR RMRR that identifies the memory
range that the kernel should open up for permissable DMA access
for this purpose. Unfortunately, the PCI device that performs these
DMA operations is hidden from the kernel by the BIOS. Given that the
associated PCI device is hidden, the Linux kernel does not act upon
the associated DMAR RMRR.

                        -Lenny.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Failure to shutdown/reboot with intel_iommu=on
  2020-05-12 20:00       ` Lenny Szubowicz
@ 2020-05-12 21:13         ` Joerg Roedel
  0 siblings, 0 replies; 10+ messages in thread
From: Joerg Roedel @ 2020-05-12 21:13 UTC (permalink / raw)
  To: Lenny Szubowicz
  Cc: Uwe Kleine-König, linux-kernel, rafael.j.wysocki, tglx, x86,
	Lu Baolu

Hi Lenny,

On Tue, May 12, 2020 at 04:00:26PM -0400, Lenny Szubowicz wrote:
> Some Lenovo laptops provide an ACPI DMAR RMRR that identifies the memory
> range that the kernel should open up for permissable DMA access
> for this purpose. Unfortunately, the PCI device that performs these
> DMA operations is hidden from the kernel by the BIOS. Given that the
> associated PCI device is hidden, the Linux kernel does not act upon
> the associated DMAR RMRR.

That sounds aweful. We should add to the VT-d driver that it sets up
RMRR mappings for request-ids which are not present as a PCI device, to
fix the Laptops which have it.

For the others, is the region the TPM talks to via DMA known so that we
can add a quirk?


	Joerg

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2020-05-12 21:13 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-06 14:46 Failure to shutdown/reboot with intel_iommu=on Uwe Kleine-König
2020-05-08 15:07 ` Joerg Roedel
2020-05-09  1:58   ` Lu Baolu
2020-05-11 19:59     ` Uwe Kleine-König
2020-05-11 20:16       ` Uwe Kleine-König
2020-05-11 13:43   ` Lenny Szubowicz
2020-05-11 19:58     ` Uwe Kleine-König
2020-05-12 13:34     ` Joerg Roedel
2020-05-12 20:00       ` Lenny Szubowicz
2020-05-12 21:13         ` Joerg Roedel

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.