Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Paul Menzel]

Dear Linux folks,


Currently, TPM support is added to the coreboot project for the Lenovo 
X60 [1].

Unfortunately, there seems to have been a regression between Linux 3.16 
and 4.8 and 4.9, so that the Linux kernel doesn’t create the TPM device.

```
$ more /proc/version Linux version 3.16.0-4-686-pae 
(debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) 
#1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07)
$ journalctl -k -o cat | grep -i tpm
tpm_tis 00:08: 1.2 TPM (device-id 0x3202, rev-id 5)
tpm_tis 00:08: Issuing TPM_STARTUP
```

The device `/dev/tpm0` is present.

Here are the results with Linux 4.8.15.

```
kernel: [    0.000000] Linux version 4.8.0-0.bpo.2-686-pae 
(debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10) ) 
#1 SMP Debian 4.8.15-2~bpo8+2 (2017-01-17)
[…]
kernel: [   11.302522] tpm tpm0: tpm_transmit: tpm_send: error -5
kernel: [   11.302598] tpm_tis 00:08: Could not get TPM timeouts and 
durations
```

Remove module and run `sudo modprobe tpm_tis itpm=1 force=1`.

```
kernel: [   35.024386] tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
kernel: [   49.160752] tpm_tis 00:08: can't request region for resource 
[mem 0xfed40000-0xfed44fff]
kernel: [   49.160865] tpm_tis: probe of 00:08 failed with error -16
```

And here with Linux 4.9.13.

```
kernel: [ 7.813547] tpm_tis 00:08: 1.2 TPM (device-id 0x3202, rev-id 5)
kernel: [ 7.891026] tpm tpm0: Issuing TPM_STARTUP
kernel: [ 8.042783] tpm tpm0: Unable to read burstcount
kernel: [ 8.042893] tpm tpm0: tpm_transmit: tpm_send: error -16
kernel: [ 8.042900] tpm_tis 00:08: Could not get TPM timeouts and duration
```

If I am not mistaken, the error code 16 is for the macro `EBUSY`.

It’d be great, if the TPM would also be created in the newer Linux 
Kernel versions, so that Debian users can upgrade to the upcoming stable 
release, which ships Linux 4.9.

Unfortunately, I won’t be able to bisect the commit introducing the 
regression. Could you please tell me how to debug, why the burstcount 
cannot be read.


Kind regards,

Paul


[1] https://review.coreboot.org/13410/

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 06:29:06PM +0200, Paul Menzel wrote:

> Unfortunately, there seems to have been a regression between Linux 3.16 
> and 4.8 and 4.9, so that the Linux kernel doesn’t create the TPM
> device.

That old kernel did not check error codes when reading burst count,
the new one does. It is quite possible the older kernel is failing to
read burst count and subtly malfunctioning.

It would be helpful if you could prove/disprove this statement by
adding some debugging to 3.16.

As to why the burst count cannot be read.. I don't know, check if
increasing the retries helps, it could be this particular TPM is out
of spec for the cold startup process.

> Remove module and run `sudo modprobe tpm_tis itpm=1 force=1`.

Is this really an 'itpm'? It doesn't look like it..

Jason

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Paul Menzel]

Dear Jason,


Thank you for your reply.


On 2017-04-04 19:15, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 06:29:06PM +0200, Paul Menzel wrote:
> 
>> Unfortunately, there seems to have been a regression between Linux 
>> 3.16
>> and 4.8 and 4.9, so that the Linux kernel doesn’t create the TPM
>> device.
> 
> That old kernel did not check error codes when reading burst count,
> the new one does. It is quite possible the older kernel is failing to
> read burst count and subtly malfunctioning.

GNUtoo got the messages below. Unfortunately, I don’t know, what Linux 
Kernel that is.

```
tpm_tis 00:07: 1.2 TPM (device-id 0x3202, rev-id 5)
tpm tpm0: A TPM error (38) occurred attempting to determine the timeouts
tpm tpm0: Issuing TPM_STARTUP
tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 
10000->10000us B 10000->10000us C 0->750000us D 0->750000us
```

> It would be helpful if you could prove/disprove this statement by
> adding some debugging to 3.16.

Could you please help me with that? Can I enable certain options to get 
more debugging information?

> As to why the burst count cannot be read.. I don't know, check if
> increasing the retries helps, it could be this particular TPM is out
> of spec for the cold startup process.

How can I increase the retry count?

>> Remove module and run `sudo modprobe tpm_tis itpm=1 force=1`.
> 
> Is this really an 'itpm'? It doesn't look like it..

Sorry, I don’t know. Leaving that out, gives the same messages though.

```
tpm tpm0: Unable to read burstcount
tpm tpm0: tpm_transmit: tpm_send: error -16
tpm_tis tpm_tis: Could not get TPM timeouts and durations
```


Kind regards,

Paul Menzel

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Maciej S. Szmigiero]

Hi Paul,

> tpm tpm0: Unable to read burstcount
> tpm tpm0: tpm_transmit: tpm_send: error -16
> tpm_tis tpm_tis: Could not get TPM timeouts and durations

This looks like a regression I had on ThinkPad X61S.

You can try with a patch from the following commit
which fixed it for me:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1d70fe9d9c3a4c627f9757cbba5d628687b121c1

It is included in mainline kernel 4.10.2.

Maciej

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jarkko Sakkinen]

On Wed, Apr 05, 2017 at 01:03:42PM +0200, Maciej S. Szmigiero wrote:
> Hi Paul,
> 
> > tpm tpm0: Unable to read burstcount
> > tpm tpm0: tpm_transmit: tpm_send: error -16
> > tpm_tis tpm_tis: Could not get TPM timeouts and durations
> 
> This looks like a regression I had on ThinkPad X61S.
> 
> You can try with a patch from the following commit
> which fixed it for me:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1d70fe9d9c3a4c627f9757cbba5d628687b121c1
> 
> It is included in mainline kernel 4.10.2.
> 
> Maciej

Paul, Maciej, thank you. Please keep us updated.

/Jarkko

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Paul Menzel]

Dear Maciej,


On 2017-04-05 13:03, Maciej S. Szmigiero wrote:

>> tpm tpm0: Unable to read burstcount
>> tpm tpm0: tpm_transmit: tpm_send: error -16
>> tpm_tis tpm_tis: Could not get TPM timeouts and durations
> 
> This looks like a regression I had on ThinkPad X61S.
> 
> You can try with a patch from the following commit
> which fixed it for me:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1d70fe9d9c3a4c627f9757cbba5d628687b121c1
> 
> It is included in mainline kernel 4.10.2.

Indeed, that improves the situation. I still need to pass `force=1` to 
the module to get `/dev/tpm0`. No idea, why it’s not in included in 
Linux 4.9 yet.

```
$ more /proc/version
Linux version 4.11.0-rc5+ (root@d44d03a193d6) (gcc version 6.3.0 
20170321 (Debian 6.3.0-10) ) #2 SMP Tue Apr 4 21:07:08 UTC 2017
$ journalctl -k -o cat | grep tpm
tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
tpm tpm0: Issuing TPM_STARTUP
tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 
10000->10000us B 10000->10000us C 0->752000us D 0->752000us
$ more /sys/class/tpm/tpm0/device/caps
Manufacturer: 0x41544d4c
TCG version: 1.2
Firmware version: 11.5
$ sudo tpm_version
Tspi_Context_Connect failed: 0x00003011 - layer=tsp, code=0011 (17), 
Communication failure
```

So do you have an idea, what “ACPI requirements” changed in the `tpm` 
module since Linux 3.16 to be loaded automatically?


Kind regards,

Paul

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Maciej S. Szmigiero]

On 04/06/17 08:18, Paul Menzel wrote:
> Dear Maciej,
> 
> On 2017-04-05 13:03, Maciej S. Szmigiero wrote:
> 
>>> tpm tpm0: Unable to read burstcount
>>> tpm tpm0: tpm_transmit: tpm_send: error -16
>>> tpm_tis tpm_tis: Could not get TPM timeouts and durations
>>
>> This looks like a regression I had on ThinkPad X61S.
>>
>> You can try with a patch from the following commit
>> which fixed it for me:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1d70fe9d9c3a4c627f9757cbba5d628687b121c1
>>
>> It is included in mainline kernel 4.10.2.
> 
> Indeed, that improves the situation. I still need to pass `force=1` to the module to get `/dev/tpm0`. No idea, why it’s not in included in Linux 4.9 yet.
> 
> ```
> $ more /proc/version
> Linux version 4.11.0-rc5+ (root@d44d03a193d6) (gcc version 6.3.0 20170321 (Debian 6.3.0-10) ) #2 SMP Tue Apr 4 21:07:08 UTC 2017
> $ journalctl -k -o cat | grep tpm
> tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
> tpm tpm0: Issuing TPM_STARTUP
> tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 10000->10000us B 10000->10000us C 0->752000us D 0->752000us
> $ more /sys/class/tpm/tpm0/device/caps
> Manufacturer: 0x41544d4c
> TCG version: 1.2
> Firmware version: 11.5
> $ sudo tpm_version
> Tspi_Context_Connect failed: 0x00003011 - layer=tsp, code=0011 (17), Communication failure
> ```
> 
> So do you have an idea, what “ACPI requirements” changed in the `tpm` module since Linux 3.16 to be loaded automatically?

Unfortunately I can't help here as I am just a normal TPM user who
happened to hit and fix that timeouts regression, but maybe others
on ML will know something more about it.

> Kind regards,
> 
> Paul

Best regards,
Maciej

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jason Gunthorpe]

On Thu, Apr 06, 2017 at 08:18:33AM +0200, Paul Menzel wrote:

> Indeed, that improves the situation. I still need to pass `force=1` to the
> module to get `/dev/tpm0`. No idea, why it’s not in included in Linux 4.9
> yet.

Fair point.. Jarkko - could you forward that patch to -stable?

> $ journalctl -k -o cat | grep tpm
> tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
> tpm tpm0: Issuing TPM_STARTUP
> tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 10000->10000us B
> 10000->10000us C 0->752000us D 0->752000us

There could be more relavent kernel messages than this, try
  grep -i tpm

?

> So do you have an idea, what “ACPI requirements” changed in the `tpm` module
> since Linux 3.16 to be loaded automatically?

We added direct ACPI binding to the driver in addition to PNP, so if
you have an ACPI table it goes down that path and does some additional
validation of what is in the TPM. The BIOS must provide a
acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
minimum.

There are FW_BUG prints if those fail, so check your dmesg closely?

http://lxr.free-electrons.com/source/drivers/char/tpm/tpm_tis.c#L250

If not, can you annotate in some printk's into tpm_tis_acpi_init and
recompile/re-test?

Jason

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Paul Menzel]

On 2017-04-06 18:55, Jason Gunthorpe wrote:
> On Thu, Apr 06, 2017 at 08:18:33AM +0200, Paul Menzel wrote:
> 
>> Indeed, that improves the situation. I still need to pass `force=1` to 
>> the
>> module to get `/dev/tpm0`. No idea, why it’s not in included in Linux 
>> 4.9
>> yet.
> 
> Fair point.. Jarkko - could you forward that patch to -stable?
> 
>> $ journalctl -k -o cat | grep tpm
>> tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
>> tpm tpm0: Issuing TPM_STARTUP
>> tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 
>> 10000->10000us B
>> 10000->10000us C 0->752000us D 0->752000us
> 
> There could be more relavent kernel messages than this, try
>   grep -i tpm
> 
> ?

At least I don’t spot more messages.

```
$ journalctl -b -3 -o cat | grep -i -e "Linux version" -e tpm
Linux version 4.11.0-rc5+ (root@d44d03a193d6) (gcc version 6.3.0 
20170321 (Debian 6.3.0-10) ) #2 SMP Tue Apr 4 21:07:08 UTC 2017
joey : TTY=pts/6 ; PWD=/home/joey ; USER=root ; COMMAND=/usr/bin/apt 
install tpm-tools
new user: name=tss, UID=123, GID=137, home=/var/lib/tpm, 
shell=/bin/false
joey : TTY=pts/6 ; PWD=/home/joey ; USER=root ; COMMAND=/sbin/modprobe 
-r tpm_tis tpm
joey : TTY=pts/6 ; PWD=/home/joey ; USER=root ; COMMAND=/sbin/modprobe 
tpm_tis force=1
tpm_tis tpm_tis: 1.2 TPM (device-id 0x3202, rev-id 5)
tpm tpm0: Issuing TPM_STARTUP
tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 
10000->10000us B 10000->10000us C 0->752000us D 0->752000us
```

>> So do you have an idea, what “ACPI requirements” changed in the `tpm` 
>> module
>> since Linux 3.16 to be loaded automatically?
> 
> We added direct ACPI binding to the driver in addition to PNP, so if
> you have an ACPI table it goes down that path and does some additional
> validation of what is in the TPM. The BIOS must provide a
> acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
> minimum.

Is it correct, that this is added in/for 4.11, so just recently? Testing 
with Linux 4.10.8, everything is detected just fine.

```
$ journalctl -k -o cat | grep -i tpm
tpm_tis 00:08: 1.2 TPM (device-id 0x3202, rev-id 5)
tpm tpm0: A TPM error (38) occurred attempting to determine the timeouts
tpm tpm0: Issuing TPM_STARTUP
tpm tpm0: [Hardware Error]: Adjusting reported timeouts: A 
10000->10000us B 10000->10000us C 0->752000us D 0->752000us
```

> There are FW_BUG prints if those fail, so check your dmesg closely?
> 
> http://lxr.free-electrons.com/source/drivers/char/tpm/tpm_tis.c#L250
> 
> If not, can you annotate in some printk's into tpm_tis_acpi_init and
> recompile/re-test?

If nothing comes to mind with the new information above, then I could do 
that. But I don’t know, if I’ll have time in the next two weeks for 
that.


Kind regards,

Paul

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jason Gunthorpe]

On Thu, Apr 06, 2017 at 08:26:22PM +0200, Paul Menzel wrote:
> >We added direct ACPI binding to the driver in addition to PNP, so if
> >you have an ACPI table it goes down that path and does some additional
> >validation of what is in the TPM. The BIOS must provide a
> >acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
> >minimum.
> 
> Is it correct, that this is added in/for 4.11, so just recently? Testing
> with Linux 4.10.8, everything is detected just fine.

No, it is quite a bit older.. And it should only go for TPM2, which I
don't think you have??

Maybe Jarkko has a guess, but sure sounds like something is recently
broken in 4.11

Jason

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jarkko Sakkinen]

On Thu, Apr 06, 2017 at 01:10:13PM -0600, Jason Gunthorpe wrote:
> On Thu, Apr 06, 2017 at 08:26:22PM +0200, Paul Menzel wrote:
> > >We added direct ACPI binding to the driver in addition to PNP, so if
> > >you have an ACPI table it goes down that path and does some additional
> > >validation of what is in the TPM. The BIOS must provide a
> > >acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
> > >minimum.
> > 
> > Is it correct, that this is added in/for 4.11, so just recently? Testing
> > with Linux 4.10.8, everything is detected just fine.
> 
> No, it is quite a bit older.. And it should only go for TPM2, which I
> don't think you have??
> 
> Maybe Jarkko has a guess, but sure sounds like something is recently
> broken in 4.11
> 
> Jason

I'll come back to this. I have to re-read the whole mail thread to
get back into the context. Lots of multitasking because of release
and so forth. Sorry for the latency!

/Jarkko

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Paul Menzel]

Dear Jarkko,


On 2017-04-07 22:13, Jarkko Sakkinen wrote:
> On Thu, Apr 06, 2017 at 01:10:13PM -0600, Jason Gunthorpe wrote:
>> On Thu, Apr 06, 2017 at 08:26:22PM +0200, Paul Menzel wrote:
>> > >We added direct ACPI binding to the driver in addition to PNP, so if
>> > >you have an ACPI table it goes down that path and does some additional
>> > >validation of what is in the TPM. The BIOS must provide a
>> > >acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
>> > >minimum.
>> >
>> > Is it correct, that this is added in/for 4.11, so just recently? Testing
>> > with Linux 4.10.8, everything is detected just fine.
>> 
>> No, it is quite a bit older.. And it should only go for TPM2, which I
>> don't think you have??
>> 
>> Maybe Jarkko has a guess, but sure sounds like something is recently
>> broken in 4.11
>> 
>> Jason
> 
> I'll come back to this. I have to re-read the whole mail thread to
> get back into the context. Lots of multitasking because of release
> and so forth. Sorry for the latency!

I started bisecting this issue. This is the current state.

```
# git bisect log
# bad: [7a771ceac771d009f7203c40b256b0608d7ea2f8] Merge tag 
'dm-4.11-changes' of 
git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
# good: [c470abd4fde40ea6a0846a2beab642a578c0b8cd] Linux 4.10
git bisect start 'HEAD' 'v4.10'
# good: [b3de5ad688f0f52457e73767f95a640ab4158d0d] Merge tag 
'regmap-v4.11' of 
git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
git bisect good b3de5ad688f0f52457e73767f95a640ab4158d0d
```


Kind regards,

Paul

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jarkko Sakkinen]

On Thu, Apr 06, 2017 at 10:55:57AM -0600, Jason Gunthorpe wrote:
> On Thu, Apr 06, 2017 at 08:18:33AM +0200, Paul Menzel wrote:
> 
> > Indeed, that improves the situation. I still need to pass `force=1` to the
> > module to get `/dev/tpm0`. No idea, why it’s not in included in Linux 4.9
> > yet.
> 
> Fair point.. Jarkko - could you forward that patch to -stable?

Yes.

/Jarkko

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

Hi Paul,

On Thu, 6 Apr 2017 10:55:57 -0600
Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote:

> We added direct ACPI binding to the driver in addition to PNP, so if
> you have an ACPI table it goes down that path and does some additional
> validation of what is in the TPM. The BIOS must provide a
> acpi_dev_resource_memory and a ACPI_SIG_TPM2 for the ACPI entry at a
> minimum.

I am working on it[1]. This commit has not been merged and is a work in
progress. It is however available in coreboot's gerrit.

So far with it and a recent kernel and the patch mentioned above:
- The linux driver finds the TPM automatically and doesn't require
  force=1
- The driver however still require itpm=1 to fully work: without it the
  tpm is found, but I wasn't able to read the PCRs.

Since I use a rolling release distribution, the kenrel version I use
tend to change quite fast, I can re-test if needed.

I need to improve the following in my patch:
- The IDs readings obviously need to be fixed.
- I want to make it work without itpm=1 if possible. I'll test with
  INTC0102 as it is in the driver in the is_itpm function, and it was
  also suggested to me by someone on IRC.

I also still need to investigate more why the itpm workaround is needed.
Does the TPM shipped in such laptops respect the specifications?
Is the wiring bad?

References:
-----------
[1]https://review.coreboot.org/cgit/coreboot.git/commit/?id=060cf4e0f50f765f85e3ecedd836eed85d1571fe

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [tpmdd-devel] Regression between Linux 3.16 and 4.8/4.9 on Lenovo X60 with coreboot

[Jason Gunthorpe]

On Sat, Apr 08, 2017 at 12:40:25PM +0200, Denis 'GNUtoo' Carikli wrote:

> I am working on it[1]. This commit has not been merged and is a work in
> progress. It is however available in coreboot's gerrit.

How did this work on any kernels if there was no PNP or ACPI entry?

FWIW, I wonder if coreboot is missing the code for wait_startup
in drivers/char/tpm/tpm_tis_core.c. If you don't do that step then
the DID/VID might not read properly.

> So far with it and a recent kernel and the patch mentioned above:
> - The linux driver finds the TPM automatically and doesn't require
>   force=1
> - The driver however still require itpm=1 to fully work: without it the
>   tpm is found, but I wasn't able to read the PCRs.

Unless it is an actual broken intel TPM you should never use
itpm=1. Juding by the log messages, it certainly is not.

Intel mode breaks the driver and removes certain error detection, eg
it may erronously succeed.

It is more likely that the failure to read PCRs reflects reality and
itpm=1 just supresses that error detection.

The TPM may need to be setup (eg cleared, EK generated, p/vflags setup
etc, etc) before it will support PCRs.

Usually the BIOS would do these steps when the TPM is first enabled,
you probably need to do them in userspace instead.

WARNING: Failure to provision the TPM properly before first use
could leave it in 'factory test' mode, which basically means it
doesn't work properly at all.

> - I want to make it work without itpm=1 if possible. I'll test with
>   INTC0102 as it is in the driver in the is_itpm function, and it was
>   also suggested to me by someone on IRC.

Don't do this unless it is actually that TPM.

Jason