linux-integrity.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Recent tpm_tis IRQ handling changes are causing kernel backtraces
@ 2021-02-11 13:09 Hans de Goede
  2021-03-16 15:34 ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-02-11 13:09 UTC (permalink / raw)
  To: Jerry Snitselaar
  Cc: Jarkko Sakkinen, Matthew Garrett, James Bottomley, linux-integrity

Hi Jerry,

It looks like there still is an issue with the recent changes to the tpm_tis IRQ
handling. At least I think those are the cause I did not dive any deeper,
I just noticed that we (Fedora) have been receiving an aweful lot of
kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...

See for example:
https://bugzilla.redhat.com/show_bug.cgi?id=1912167
https://bugzilla.redhat.com/show_bug.cgi?id=1927610

Those are just the 3 which landed in my inbox today, for much more see:
https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
(this shows 18 bugs atm).

These were reported through the Fedora ABRT tools which automatically
collects backtraces, the bugs have links to the ABRT reports, e.g. :
https://retrace.fedoraproject.org/faf/reports/28155/
https://retrace.fedoraproject.org/faf/reports/37107/

The 28155 report says that so far there have been 308,412 (ouch) automatic
uploads of that particular variant of these backtraces

Note the second (37107) retrace report is about this happening
on resume, rather then on probe/tpm_tis_probe_irq_... time.

Did your work on  this work land in 5.10 ? Or could it be that the
issue is an incomplete backport to the 5.10.y stable series ?

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-02-11 13:09 Recent tpm_tis IRQ handling changes are causing kernel backtraces Hans de Goede
@ 2021-03-16 15:34 ` Hans de Goede
  2021-03-16 19:18   ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-03-16 15:34 UTC (permalink / raw)
  To: Jerry Snitselaar
  Cc: Jarkko Sakkinen, Matthew Garrett, James Bottomley, linux-integrity

Hi,

On 2/11/21 2:09 PM, Hans de Goede wrote:
> Hi Jerry,
> 
> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
> handling. At least I think those are the cause I did not dive any deeper,
> I just noticed that we (Fedora) have been receiving an aweful lot of
> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
> 
> See for example:
> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
> 
> Those are just the 3 which landed in my inbox today, for much more see:
> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> (this shows 18 bugs atm).
> 
> These were reported through the Fedora ABRT tools which automatically
> collects backtraces, the bugs have links to the ABRT reports, e.g. :
> https://retrace.fedoraproject.org/faf/reports/28155/
> https://retrace.fedoraproject.org/faf/reports/37107/
> 
> The 28155 report says that so far there have been 308,412 (ouch) automatic
> uploads of that particular variant of these backtraces
> 
> Note the second (37107) retrace report is about this happening
> on resume, rather then on probe/tpm_tis_probe_irq_... time.
> 
> Did your work on this work land in 5.10 ? Or could it be that the
> issue is an incomplete backport to the 5.10.y stable series ?

Ping ?

It is raining bug-reports about this:

https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data

Currently lists 25 bugs and that is excluding bugs which have already
been marked as a duplicate.

Can someone involved in the patch-series which is causing this regression
please take a look at these kernel backtraces ?

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-03-16 15:34 ` Hans de Goede
@ 2021-03-16 19:18   ` Jarkko Sakkinen
  2021-05-08  9:07     ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-03-16 19:18 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
> Hi,
> 
> On 2/11/21 2:09 PM, Hans de Goede wrote:
> > Hi Jerry,
> > 
> > It looks like there still is an issue with the recent changes to the tpm_tis IRQ
> > handling. At least I think those are the cause I did not dive any deeper,
> > I just noticed that we (Fedora) have been receiving an aweful lot of
> > kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
> > 
> > See for example:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1912167
> > https://bugzilla.redhat.com/show_bug.cgi?id=1927610
> > 
> > Those are just the 3 which landed in my inbox today, for much more see:
> > https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> > (this shows 18 bugs atm).
> > 
> > These were reported through the Fedora ABRT tools which automatically
> > collects backtraces, the bugs have links to the ABRT reports, e.g. :
> > https://retrace.fedoraproject.org/faf/reports/28155/
> > https://retrace.fedoraproject.org/faf/reports/37107/
> > 
> > The 28155 report says that so far there have been 308,412 (ouch) automatic
> > uploads of that particular variant of these backtraces
> > 
> > Note the second (37107) retrace report is about this happening
> > on resume, rather then on probe/tpm_tis_probe_irq_... time.
> > 
> > Did your work on this work land in 5.10 ? Or could it be that the
> > issue is an incomplete backport to the 5.10.y stable series ?
> 
> Ping ?
> 
> It is raining bug-reports about this:
> 
> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> 
> Currently lists 25 bugs and that is excluding bugs which have already
> been marked as a duplicate.
> 
> Can someone involved in the patch-series which is causing this regression
> please take a look at these kernel backtraces ?
> 
> Regards,
> 
> Hans

I incorporated two fixes to this issue to my last PR, which were taken
to the mainline. What is the situation with the mainline?

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-03-16 19:18   ` Jarkko Sakkinen
@ 2021-05-08  9:07     ` Hans de Goede
  2021-05-10 17:25       ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-05-08  9:07 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

Hi Jarko,

On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
>> Hi,
>>
>> On 2/11/21 2:09 PM, Hans de Goede wrote:
>>> Hi Jerry,
>>>
>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
>>> handling. At least I think those are the cause I did not dive any deeper,
>>> I just noticed that we (Fedora) have been receiving an aweful lot of
>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
>>>
>>> See for example:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
>>>
>>> Those are just the 3 which landed in my inbox today, for much more see:
>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>> (this shows 18 bugs atm).
>>>
>>> These were reported through the Fedora ABRT tools which automatically
>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
>>> https://retrace.fedoraproject.org/faf/reports/28155/
>>> https://retrace.fedoraproject.org/faf/reports/37107/
>>>
>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
>>> uploads of that particular variant of these backtraces
>>>
>>> Note the second (37107) retrace report is about this happening
>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
>>>
>>> Did your work on this work land in 5.10 ? Or could it be that the
>>> issue is an incomplete backport to the 5.10.y stable series ?
>>
>> Ping ?
>>
>> It is raining bug-reports about this:
>>
>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>
>> Currently lists 25 bugs and that is excluding bugs which have already
>> been marked as a duplicate.
>>
>> Can someone involved in the patch-series which is causing this regression
>> please take a look at these kernel backtraces ?
>>
>> Regards,
>>
>> Hans
> 
> I incorporated two fixes to this issue to my last PR, which were taken
> to the mainline. What is the situation with the mainline?

Thank you for your reply and sorry for being slow to respond.

Is this expected to be fixed in 5.11, or when you say mainline you
main Linus' master branch / so the fixes are in 5.12 only ?

The reason I'm asking is because we just received another bugreport
about this against 5.11.17. The bug is marked private (our tool to
automatically file bugs for kernel backtraces does this) so let me
just copy and paste the trace here:

WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
tpm_tis_status+0x66/0x70

CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
05/27/2016
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
d0 90 66 66 66 66 90 41 57 41 56 41 55 41
RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
Call Trace:
 tpm_tis_send_data+0x2b/0x230
 tpm_tis_send_main+0x1e/0xe0
 tpm_transmit+0xd8/0x3d0
 tpm_transmit_cmd+0x25/0x90
 tpm1_do_selftest+0x88/0x130
 ? _cond_resched+0x16/0x40
 tpm_tis_resume+0x4d/0x120
 ? pnpacpi_resume+0x1b/0xa0
 ? pnp_bus_suspend+0x10/0x10
 pnp_bus_resume+0x63/0x90
 dpm_run_callback+0x4c/0x120
 device_resume+0xa7/0x200
 dpm_resume+0xce/0x2c0
 dpm_resume_end+0xd/0x20
 suspend_devices_and_enter+0x195/0x750
 pm_suspend.cold+0x329/0x374
 state_store+0x71/0xd0
 kernfs_fop_write_iter+0x124/0x1b0
 new_sync_write+0x108/0x180
 vfs_write+0x1bc/0x270
 ksys_write+0x4f/0xc0
 do_syscall_64+0x33/0x40
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f8f102ec4e7
Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-08  9:07     ` Hans de Goede
@ 2021-05-10 17:25       ` Jarkko Sakkinen
  2021-05-11  8:37         ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-05-10 17:25 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
> Hi Jarko,
> 
> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
> > On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 2/11/21 2:09 PM, Hans de Goede wrote:
> >>> Hi Jerry,
> >>>
> >>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
> >>> handling. At least I think those are the cause I did not dive any deeper,
> >>> I just noticed that we (Fedora) have been receiving an aweful lot of
> >>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
> >>>
> >>> See for example:
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
> >>>
> >>> Those are just the 3 which landed in my inbox today, for much more see:
> >>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>> (this shows 18 bugs atm).
> >>>
> >>> These were reported through the Fedora ABRT tools which automatically
> >>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
> >>> https://retrace.fedoraproject.org/faf/reports/28155/
> >>> https://retrace.fedoraproject.org/faf/reports/37107/
> >>>
> >>> The 28155 report says that so far there have been 308,412 (ouch) automatic
> >>> uploads of that particular variant of these backtraces
> >>>
> >>> Note the second (37107) retrace report is about this happening
> >>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
> >>>
> >>> Did your work on this work land in 5.10 ? Or could it be that the
> >>> issue is an incomplete backport to the 5.10.y stable series ?
> >>
> >> Ping ?
> >>
> >> It is raining bug-reports about this:
> >>
> >> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>
> >> Currently lists 25 bugs and that is excluding bugs which have already
> >> been marked as a duplicate.
> >>
> >> Can someone involved in the patch-series which is causing this regression
> >> please take a look at these kernel backtraces ?
> >>
> >> Regards,
> >>
> >> Hans
> > 
> > I incorporated two fixes to this issue to my last PR, which were taken
> > to the mainline. What is the situation with the mainline?
> 
> Thank you for your reply and sorry for being slow to respond.
> 
> Is this expected to be fixed in 5.11, or when you say mainline you
> main Linus' master branch / so the fixes are in 5.12 only ?
> 
> The reason I'm asking is because we just received another bugreport
> about this against 5.11.17. The bug is marked private (our tool to
> automatically file bugs for kernel backtraces does this) so let me
> just copy and paste the trace here:
> 
> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
> tpm_tis_status+0x66/0x70
> 
> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
> 05/27/2016
> RIP: 0010:tpm_tis_status+0x66/0x70
> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
> FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
> Call Trace:
>  tpm_tis_send_data+0x2b/0x230
>  tpm_tis_send_main+0x1e/0xe0
>  tpm_transmit+0xd8/0x3d0
>  tpm_transmit_cmd+0x25/0x90
>  tpm1_do_selftest+0x88/0x130
>  ? _cond_resched+0x16/0x40
>  tpm_tis_resume+0x4d/0x120
>  ? pnpacpi_resume+0x1b/0xa0
>  ? pnp_bus_suspend+0x10/0x10
>  pnp_bus_resume+0x63/0x90
>  dpm_run_callback+0x4c/0x120
>  device_resume+0xa7/0x200
>  dpm_resume+0xce/0x2c0
>  dpm_resume_end+0xd/0x20
>  suspend_devices_and_enter+0x195/0x750
>  pm_suspend.cold+0x329/0x374
>  state_store+0x71/0xd0
>  kernfs_fop_write_iter+0x124/0x1b0
>  new_sync_write+0x108/0x180
>  vfs_write+0x1bc/0x270
>  ksys_write+0x4f/0xc0
>  do_syscall_64+0x33/0x40
>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f8f102ec4e7
> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
> 
> Regards,
> 
> Hans

I sent a couple fixes (cc'd to you).

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-10 17:25       ` Jarkko Sakkinen
@ 2021-05-11  8:37         ` Hans de Goede
  2021-05-11 23:48           ` Jarkko Sakkinen
  2021-05-26 19:03           ` Hans de Goede
  0 siblings, 2 replies; 29+ messages in thread
From: Hans de Goede @ 2021-05-11  8:37 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

Hi,

On 5/10/21 7:25 PM, Jarkko Sakkinen wrote:
> On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
>> Hi Jarko,
>>
>> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
>>> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 2/11/21 2:09 PM, Hans de Goede wrote:
>>>>> Hi Jerry,
>>>>>
>>>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
>>>>> handling. At least I think those are the cause I did not dive any deeper,
>>>>> I just noticed that we (Fedora) have been receiving an aweful lot of
>>>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
>>>>>
>>>>> See for example:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
>>>>>
>>>>> Those are just the 3 which landed in my inbox today, for much more see:
>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>> (this shows 18 bugs atm).
>>>>>
>>>>> These were reported through the Fedora ABRT tools which automatically
>>>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
>>>>> https://retrace.fedoraproject.org/faf/reports/28155/
>>>>> https://retrace.fedoraproject.org/faf/reports/37107/
>>>>>
>>>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
>>>>> uploads of that particular variant of these backtraces
>>>>>
>>>>> Note the second (37107) retrace report is about this happening
>>>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
>>>>>
>>>>> Did your work on this work land in 5.10 ? Or could it be that the
>>>>> issue is an incomplete backport to the 5.10.y stable series ?
>>>>
>>>> Ping ?
>>>>
>>>> It is raining bug-reports about this:
>>>>
>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>
>>>> Currently lists 25 bugs and that is excluding bugs which have already
>>>> been marked as a duplicate.
>>>>
>>>> Can someone involved in the patch-series which is causing this regression
>>>> please take a look at these kernel backtraces ?
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>
>>> I incorporated two fixes to this issue to my last PR, which were taken
>>> to the mainline. What is the situation with the mainline?
>>
>> Thank you for your reply and sorry for being slow to respond.
>>
>> Is this expected to be fixed in 5.11, or when you say mainline you
>> main Linus' master branch / so the fixes are in 5.12 only ?
>>
>> The reason I'm asking is because we just received another bugreport
>> about this against 5.11.17. The bug is marked private (our tool to
>> automatically file bugs for kernel backtraces does this) so let me
>> just copy and paste the trace here:
>>
>> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
>> tpm_tis_status+0x66/0x70
>>
>> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
>> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
>> 05/27/2016
>> RIP: 0010:tpm_tis_status+0x66/0x70
>> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
>> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
>> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
>> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
>> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
>> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
>> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
>> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
>> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
>> FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
>> Call Trace:
>>  tpm_tis_send_data+0x2b/0x230
>>  tpm_tis_send_main+0x1e/0xe0
>>  tpm_transmit+0xd8/0x3d0
>>  tpm_transmit_cmd+0x25/0x90
>>  tpm1_do_selftest+0x88/0x130
>>  ? _cond_resched+0x16/0x40
>>  tpm_tis_resume+0x4d/0x120
>>  ? pnpacpi_resume+0x1b/0xa0
>>  ? pnp_bus_suspend+0x10/0x10
>>  pnp_bus_resume+0x63/0x90
>>  dpm_run_callback+0x4c/0x120
>>  device_resume+0xa7/0x200
>>  dpm_resume+0xce/0x2c0
>>  dpm_resume_end+0xd/0x20
>>  suspend_devices_and_enter+0x195/0x750
>>  pm_suspend.cold+0x329/0x374
>>  state_store+0x71/0xd0
>>  kernfs_fop_write_iter+0x124/0x1b0
>>  new_sync_write+0x108/0x180
>>  vfs_write+0x1bc/0x270
>>  ksys_write+0x4f/0xc0
>>  do_syscall_64+0x33/0x40
>>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>> RIP: 0033:0x7f8f102ec4e7
>> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
>> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
>> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
>> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
>> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
>> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
>> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
>> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
>>
>> Regards,
>>
>> Hans
> 
> I sent a couple fixes (cc'd to you).

I've seen the fixes, thank you.

I'll probably add these as downstream patches to the Fedora 5.12 kernels for now
and see if that helps.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-11  8:37         ` Hans de Goede
@ 2021-05-11 23:48           ` Jarkko Sakkinen
  2021-05-26 19:03           ` Hans de Goede
  1 sibling, 0 replies; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-05-11 23:48 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

On Tue, May 11, 2021 at 10:37:40AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 5/10/21 7:25 PM, Jarkko Sakkinen wrote:
> > On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
> >> Hi Jarko,
> >>
> >> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
> >>> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
> >>>> Hi,
> >>>>
> >>>> On 2/11/21 2:09 PM, Hans de Goede wrote:
> >>>>> Hi Jerry,
> >>>>>
> >>>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
> >>>>> handling. At least I think those are the cause I did not dive any deeper,
> >>>>> I just noticed that we (Fedora) have been receiving an aweful lot of
> >>>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
> >>>>>
> >>>>> See for example:
> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
> >>>>>
> >>>>> Those are just the 3 which landed in my inbox today, for much more see:
> >>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>>>> (this shows 18 bugs atm).
> >>>>>
> >>>>> These were reported through the Fedora ABRT tools which automatically
> >>>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
> >>>>> https://retrace.fedoraproject.org/faf/reports/28155/
> >>>>> https://retrace.fedoraproject.org/faf/reports/37107/
> >>>>>
> >>>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
> >>>>> uploads of that particular variant of these backtraces
> >>>>>
> >>>>> Note the second (37107) retrace report is about this happening
> >>>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
> >>>>>
> >>>>> Did your work on this work land in 5.10 ? Or could it be that the
> >>>>> issue is an incomplete backport to the 5.10.y stable series ?
> >>>>
> >>>> Ping ?
> >>>>
> >>>> It is raining bug-reports about this:
> >>>>
> >>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>>>
> >>>> Currently lists 25 bugs and that is excluding bugs which have already
> >>>> been marked as a duplicate.
> >>>>
> >>>> Can someone involved in the patch-series which is causing this regression
> >>>> please take a look at these kernel backtraces ?
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>
> >>> I incorporated two fixes to this issue to my last PR, which were taken
> >>> to the mainline. What is the situation with the mainline?
> >>
> >> Thank you for your reply and sorry for being slow to respond.
> >>
> >> Is this expected to be fixed in 5.11, or when you say mainline you
> >> main Linus' master branch / so the fixes are in 5.12 only ?
> >>
> >> The reason I'm asking is because we just received another bugreport
> >> about this against 5.11.17. The bug is marked private (our tool to
> >> automatically file bugs for kernel backtraces does this) so let me
> >> just copy and paste the trace here:
> >>
> >> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
> >> tpm_tis_status+0x66/0x70
> >>
> >> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
> >> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
> >> 05/27/2016
> >> RIP: 0010:tpm_tis_status+0x66/0x70
> >> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
> >> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
> >> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
> >> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
> >> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
> >> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
> >> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
> >> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
> >> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
> >> FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
> >> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
> >> Call Trace:
> >>  tpm_tis_send_data+0x2b/0x230
> >>  tpm_tis_send_main+0x1e/0xe0
> >>  tpm_transmit+0xd8/0x3d0
> >>  tpm_transmit_cmd+0x25/0x90
> >>  tpm1_do_selftest+0x88/0x130
> >>  ? _cond_resched+0x16/0x40
> >>  tpm_tis_resume+0x4d/0x120
> >>  ? pnpacpi_resume+0x1b/0xa0
> >>  ? pnp_bus_suspend+0x10/0x10
> >>  pnp_bus_resume+0x63/0x90
> >>  dpm_run_callback+0x4c/0x120
> >>  device_resume+0xa7/0x200
> >>  dpm_resume+0xce/0x2c0
> >>  dpm_resume_end+0xd/0x20
> >>  suspend_devices_and_enter+0x195/0x750
> >>  pm_suspend.cold+0x329/0x374
> >>  state_store+0x71/0xd0
> >>  kernfs_fop_write_iter+0x124/0x1b0
> >>  new_sync_write+0x108/0x180
> >>  vfs_write+0x1bc/0x270
> >>  ksys_write+0x4f/0xc0
> >>  do_syscall_64+0x33/0x40
> >>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >> RIP: 0033:0x7f8f102ec4e7
> >> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
> >> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
> >> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> >> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> >> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
> >> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
> >> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
> >> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
> >> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
> >>
> >> Regards,
> >>
> >> Hans
> > 
> > I sent a couple fixes (cc'd to you).
> 
> I've seen the fixes, thank you.
> 
> I'll probably add these as downstream patches to the Fedora 5.12 kernels for now
> and see if that helps.
> 
> Regards,
> 
> Hans

They should soon mirror to 'linux-next'. I'm going to send an additional PR
at some point to Linus for v5.13, with these fixes included.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-11  8:37         ` Hans de Goede
  2021-05-11 23:48           ` Jarkko Sakkinen
@ 2021-05-26 19:03           ` Hans de Goede
  2021-05-27 14:00             ` Jarkko Sakkinen
  1 sibling, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-05-26 19:03 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

Hi,

On 5/11/21 10:37 AM, Hans de Goede wrote:
> Hi,
> 
> On 5/10/21 7:25 PM, Jarkko Sakkinen wrote:
>> On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
>>> Hi Jarko,
>>>
>>> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
>>>> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
>>>>> Hi,
>>>>>
>>>>> On 2/11/21 2:09 PM, Hans de Goede wrote:
>>>>>> Hi Jerry,
>>>>>>
>>>>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
>>>>>> handling. At least I think those are the cause I did not dive any deeper,
>>>>>> I just noticed that we (Fedora) have been receiving an aweful lot of
>>>>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
>>>>>>
>>>>>> See for example:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
>>>>>>
>>>>>> Those are just the 3 which landed in my inbox today, for much more see:
>>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>>> (this shows 18 bugs atm).
>>>>>>
>>>>>> These were reported through the Fedora ABRT tools which automatically
>>>>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
>>>>>> https://retrace.fedoraproject.org/faf/reports/28155/
>>>>>> https://retrace.fedoraproject.org/faf/reports/37107/
>>>>>>
>>>>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
>>>>>> uploads of that particular variant of these backtraces
>>>>>>
>>>>>> Note the second (37107) retrace report is about this happening
>>>>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
>>>>>>
>>>>>> Did your work on this work land in 5.10 ? Or could it be that the
>>>>>> issue is an incomplete backport to the 5.10.y stable series ?
>>>>>
>>>>> Ping ?
>>>>>
>>>>> It is raining bug-reports about this:
>>>>>
>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
>>>>>
>>>>> Currently lists 25 bugs and that is excluding bugs which have already
>>>>> been marked as a duplicate.
>>>>>
>>>>> Can someone involved in the patch-series which is causing this regression
>>>>> please take a look at these kernel backtraces ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Hans
>>>>
>>>> I incorporated two fixes to this issue to my last PR, which were taken
>>>> to the mainline. What is the situation with the mainline?
>>>
>>> Thank you for your reply and sorry for being slow to respond.
>>>
>>> Is this expected to be fixed in 5.11, or when you say mainline you
>>> main Linus' master branch / so the fixes are in 5.12 only ?
>>>
>>> The reason I'm asking is because we just received another bugreport
>>> about this against 5.11.17. The bug is marked private (our tool to
>>> automatically file bugs for kernel backtraces does this) so let me
>>> just copy and paste the trace here:
>>>
>>> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
>>> tpm_tis_status+0x66/0x70
>>>
>>> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
>>> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
>>> 05/27/2016
>>> RIP: 0010:tpm_tis_status+0x66/0x70
>>> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
>>> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
>>> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
>>> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
>>> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
>>> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
>>> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
>>> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
>>> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
>>> FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
>>> Call Trace:
>>>  tpm_tis_send_data+0x2b/0x230
>>>  tpm_tis_send_main+0x1e/0xe0
>>>  tpm_transmit+0xd8/0x3d0
>>>  tpm_transmit_cmd+0x25/0x90
>>>  tpm1_do_selftest+0x88/0x130
>>>  ? _cond_resched+0x16/0x40
>>>  tpm_tis_resume+0x4d/0x120
>>>  ? pnpacpi_resume+0x1b/0xa0
>>>  ? pnp_bus_suspend+0x10/0x10
>>>  pnp_bus_resume+0x63/0x90
>>>  dpm_run_callback+0x4c/0x120
>>>  device_resume+0xa7/0x200
>>>  dpm_resume+0xce/0x2c0
>>>  dpm_resume_end+0xd/0x20
>>>  suspend_devices_and_enter+0x195/0x750
>>>  pm_suspend.cold+0x329/0x374
>>>  state_store+0x71/0xd0
>>>  kernfs_fop_write_iter+0x124/0x1b0
>>>  new_sync_write+0x108/0x180
>>>  vfs_write+0x1bc/0x270
>>>  ksys_write+0x4f/0xc0
>>>  do_syscall_64+0x33/0x40
>>>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> RIP: 0033:0x7f8f102ec4e7
>>> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
>>> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
>>> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
>>> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>>> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
>>> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
>>> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
>>> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
>>> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
>>>
>>> Regards,
>>>
>>> Hans
>>
>> I sent a couple fixes (cc'd to you).
> 
> I've seen the fixes, thank you.
> 
> I'll probably add these as downstream patches to the Fedora 5.12 kernels for now
> and see if that helps.

I'm afraid that we are still getting tpm irq kernel-backtrace reports with
5.12.6 which has the 2 fixes (AFAICT), here are 2 of them:

WARNING: CPU: 5 PID: 142 at drivers/char/tpm/tpm_tis_core.c:205
tpm_tis_status+0x66/0x70
Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer xt_CHECKSUM
xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp
llc ccm michael_mic nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct cmac nft_chain_nat ip6table_nat
ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security
ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter
snd_soc_skl_hda_dsp snd_soc_hdac_hdmi qrtr_mhi bnep snd_hda_codec_hdmi
snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl
snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation
soundwire_cadence sunrpc snd_sof_intel_hda snd_sof_pci iTCO_wdt snd_sof
intel_pmc_bxt iTCO_vendor_support snd_sof_xtensa_dsp snd_soc_hdac_hda
snd_hda_ext_core
 snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core mei_hdcp
x86_pkg_temp_thermal qrtr snd_compress intel_pmt_telemetry intel_powerclamp
intel_rapl_msr ns snd_pcm_dmaengine intel_pmt_class ac97_bus dell_laptop
ath11k_pci coretemp ledtrig_audio ath11k dell_smm_hwmon kvm_intel snd_hda_intel
snd_intel_dspcfg snd_intel_sdw_acpi qmi_helpers kvm snd_hda_codec mac80211
snd_hda_core snd_hwdep snd_seq snd_seq_device irqbypass intel_cstate dell_wmi
intel_uncore snd_pcm dell_smbios dcdbas cfg80211 snd_timer pcspkr snd wmi_bmof
dell_wmi_sysman dell_wmi_descriptor i2c_i801 soundcore i2c_smbus mhi uvcvideo
libarc4 hci_uart videobuf2_vmalloc videobuf2_memops mei_me videobuf2_v4l2 vfat
mei videobuf2_common fat btqca joydev hid_sensor_als btrtl idma64 btbcm
hid_sensor_trigger videodev hid_sensor_iio_common processor_thermal_device
btintel industrialio_triggered_buffer processor_thermal_rfim kfifo_buf
processor_thermal_mbox mc industrialio processor_thermal_rapl bluetooth
thunderbolt
 intel_rapl_common intel_pmt intel_soc_dts_iosf ecdh_generic ucsi_acpi rfkill
typec_ucsi typec ecc int3403_thermal intel_hid int340x_thermal_zone
sparse_keymap int3400_thermal acpi_pad acpi_tad acpi_thermal_rel zram ip_tables
dm_crypt trusted hid_sensor_hub intel_ishtp_loader intel_ishtp_hid
hid_multitouch i915 i2c_algo_bit rtsx_pci_sdmmc nvme drm_kms_helper mmc_core
nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel cec drm
ghash_clmulni_intel rtsx_pci serio_raw intel_ish_ipc intel_ishtp vmd
i2c_hid_acpi i2c_hid wmi video pinctrl_tigerlake fuse
CPU: 5 PID: 142 Comm: kworker/5:1 Not tainted 5.12.6-300.fc34.x86_64 #1
Hardware name: Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021
Workqueue: tpm_dev_wq tpm_dev_async_work
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
RSP: 0018:ffffafc80037bd40 EFLAGS: 00010286
RAX: 000000000000001b RBX: ffff9c8c47cff000 RCX: 0000000000000027
RDX: ffff9c93af7585c8 RSI: 0000000000000001 RDI: ffff9c93af7585c0
RBP: ffff9c8c47cff000 R08: 0000000000000000 R09: ffffafc80037bb70
R10: ffffafc80037bb68 R11: ffffffffa6b45f28 R12: ffff9c8c47df5aa8
R13: ffff9c8c4d14e0ba R14: 0000000000000000 R15: ffffafc80037bdf2
FS:  0000000000000000(0000) GS:ffff9c93af740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fadaa0bd490 CR3: 0000000170c42005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 tpm_tis_send_data+0x2b/0x230
 ? tpm_tcg_read_bytes+0x30/0x50
 tpm_tis_send_main+0x1e/0xe0
 tpm_transmit+0xd6/0x3d0WARNING: CPU: 2 PID: 1 at drivers/char/tpm/tpm_tis_core.c:205
tpm_tis_status+0x66/0x70
Modules linked in:
CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.12.6-300.fc34.x86_64 #1
Hardware name: Dell Inc. XPS 13 9310/0GG9PT, BIOS 2.2.0 04/06/2021
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
RSP: 0018:ffffad554006bae0 EFLAGS: 00010282
RAX: 000000000000001b RBX: ffff96bf471d5000 RCX: ffffffffa6b15ee8
RDX: c0000000ffffdfff RSI: 00000000ffffdfff RDI: ffffffffa752ec6c
RBP: ffff96bf471d5000 R08: 0000000000000000 R09: ffffad554006b910
R10: ffffad554006b908 R11: ffffffffa6b45f28 R12: ffff96bf472f61a8
R13: ffff96bf47d87000 R14: 0000000000000000 R15: ffffad554006bb92
FS:  0000000000000000(0000) GS:ffff96c2bf680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff201c82958 CR3: 0000000010a10001 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 tpm_tis_send_data+0x2b/0x230
 tpm_tis_send_main+0x1e/0xe0
 tpm_transmit+0xd6/0x3d0
 tpm_transmit_cmd+0x25/0x90
 tpm2_pcr_extend+0x1f9/0x240
 tpm_pcr_extend+0xa1/0xb0
 ima_add_template_entry+0x16e/0x220
 ? ima_store_template+0x3a/0xb0
 ? hash_setup+0xc5/0xc5
 ima_add_boot_aggregate+0xd4/0x13e
 ima_init+0x51/0x94
 init_ima+0x23/0xb5
 ? hash_setup+0xc5/0xc5
 do_one_initcall+0x44/0x1d0
 kernel_init_freeable+0x1da/0x221
 ? rest_init+0xb4/0xb4
 kernel_init+0xa/0x11c
 ret_from_fork+0x1f/0x30

 tpm_dev_transmit.constprop.0+0x47/0xa0
 tpm_dev_async_work+0x62/0x90
 process_one_work+0x1ec/0x380
 worker_thread+0x53/0x3e0
 ? process_one_work+0x380/0x380
 kthread+0x11b/0x140
 ? kthread_associate_blkcg+0xa0/0xa0
 ret_from_fork+0x1f/0x30


Regards,

Hans





^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-26 19:03           ` Hans de Goede
@ 2021-05-27 14:00             ` Jarkko Sakkinen
  2021-05-27 15:27               ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-05-27 14:00 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

On Wed, May 26, 2021 at 09:03:26PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 5/11/21 10:37 AM, Hans de Goede wrote:
> > Hi,
> > 
> > On 5/10/21 7:25 PM, Jarkko Sakkinen wrote:
> >> On Sat, May 08, 2021 at 11:07:43AM +0200, Hans de Goede wrote:
> >>> Hi Jarko,
> >>>
> >>> On 3/16/21 8:18 PM, Jarkko Sakkinen wrote:
> >>>> On Tue, Mar 16, 2021 at 04:34:01PM +0100, Hans de Goede wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 2/11/21 2:09 PM, Hans de Goede wrote:
> >>>>>> Hi Jerry,
> >>>>>>
> >>>>>> It looks like there still is an issue with the recent changes to the tpm_tis IRQ
> >>>>>> handling. At least I think those are the cause I did not dive any deeper,
> >>>>>> I just noticed that we (Fedora) have been receiving an aweful lot of
> >>>>>> kernel tpm_tis_send_data backtraces with most starting with tpm_tis_probe_irq_...
> >>>>>>
> >>>>>> See for example:
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1912167
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1927610
> >>>>>>
> >>>>>> Those are just the 3 which landed in my inbox today, for much more see:
> >>>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>>>>> (this shows 18 bugs atm).
> >>>>>>
> >>>>>> These were reported through the Fedora ABRT tools which automatically
> >>>>>> collects backtraces, the bugs have links to the ABRT reports, e.g. :
> >>>>>> https://retrace.fedoraproject.org/faf/reports/28155/
> >>>>>> https://retrace.fedoraproject.org/faf/reports/37107/
> >>>>>>
> >>>>>> The 28155 report says that so far there have been 308,412 (ouch) automatic
> >>>>>> uploads of that particular variant of these backtraces
> >>>>>>
> >>>>>> Note the second (37107) retrace report is about this happening
> >>>>>> on resume, rather then on probe/tpm_tis_probe_irq_... time.
> >>>>>>
> >>>>>> Did your work on this work land in 5.10 ? Or could it be that the
> >>>>>> issue is an incomplete backport to the 5.10.y stable series ?
> >>>>>
> >>>>> Ping ?
> >>>>>
> >>>>> It is raining bug-reports about this:
> >>>>>
> >>>>> https://bugzilla.redhat.com/buglist.cgi?quicksearch=tpm_tis_send_data
> >>>>>
> >>>>> Currently lists 25 bugs and that is excluding bugs which have already
> >>>>> been marked as a duplicate.
> >>>>>
> >>>>> Can someone involved in the patch-series which is causing this regression
> >>>>> please take a look at these kernel backtraces ?
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Hans
> >>>>
> >>>> I incorporated two fixes to this issue to my last PR, which were taken
> >>>> to the mainline. What is the situation with the mainline?
> >>>
> >>> Thank you for your reply and sorry for being slow to respond.
> >>>
> >>> Is this expected to be fixed in 5.11, or when you say mainline you
> >>> main Linus' master branch / so the fixes are in 5.12 only ?
> >>>
> >>> The reason I'm asking is because we just received another bugreport
> >>> about this against 5.11.17. The bug is marked private (our tool to
> >>> automatically file bugs for kernel backtraces does this) so let me
> >>> just copy and paste the trace here:
> >>>
> >>> WARNING: CPU: 0 PID: 3060 at drivers/char/tpm/tpm_tis_core.c:205
> >>> tpm_tis_status+0x66/0x70
> >>>
> >>> CPU: 0 PID: 3060 Comm: systemd-sleep Not tainted 5.11.17-200.fc33.x86_64 #1
> >>> Hardware name: Hewlett-Packard HP ProBook 6460b/161D, BIOS 68SCE Ver. F.63
> >>> 05/27/2016
> >>> RIP: 0010:tpm_tis_status+0x66/0x70
> >>> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d 38 02 56 01 00 75 f0 48 c7 c7 94 67
> >>> 43 96 88 44 24 07 c6 05 24 02 56 01 01 e8 4a 53 3c 00 <0f> 0b 0f b6 44 24 07 eb
> >>> d0 90 66 66 66 66 90 41 57 41 56 41 55 41
> >>> RSP: 0018:ffffaac581427b10 EFLAGS: 00010282
> >>> RAX: 000000000000001b RBX: ffff9dc800b93000 RCX: ffff9dc83b418ac8
> >>> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9dc83b418ac0
> >>> RBP: ffff9dc800b93000 R08: ffffffff96a64ec0 R09: ffffaac581427ab0
> >>> R10: 0000000000000001 R11: 000000002d2d2d2d R12: ffff9dc80667c268
> >>> R13: ffff9dc801fd1000 R14: 0000000000000000 R15: ffffaac581427bca
> >>> FS:  00007f8f0f32c000(0000) GS:ffff9dc83b400000(0000) knlGS:0000000000000000
> >>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>> CR2: 0000557044ec6c86 CR3: 0000000062e92001 CR4: 00000000000606f0
> >>> Call Trace:
> >>>  tpm_tis_send_data+0x2b/0x230
> >>>  tpm_tis_send_main+0x1e/0xe0
> >>>  tpm_transmit+0xd8/0x3d0
> >>>  tpm_transmit_cmd+0x25/0x90
> >>>  tpm1_do_selftest+0x88/0x130
> >>>  ? _cond_resched+0x16/0x40
> >>>  tpm_tis_resume+0x4d/0x120
> >>>  ? pnpacpi_resume+0x1b/0xa0
> >>>  ? pnp_bus_suspend+0x10/0x10
> >>>  pnp_bus_resume+0x63/0x90
> >>>  dpm_run_callback+0x4c/0x120
> >>>  device_resume+0xa7/0x200
> >>>  dpm_resume+0xce/0x2c0
> >>>  dpm_resume_end+0xd/0x20
> >>>  suspend_devices_and_enter+0x195/0x750
> >>>  pm_suspend.cold+0x329/0x374
> >>>  state_store+0x71/0xd0
> >>>  kernfs_fop_write_iter+0x124/0x1b0
> >>>  new_sync_write+0x108/0x180
> >>>  vfs_write+0x1bc/0x270
> >>>  ksys_write+0x4f/0xc0
> >>>  do_syscall_64+0x33/0x40
> >>>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >>> RIP: 0033:0x7f8f102ec4e7
> >>> Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64
> >>> 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51
> >>> c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> >>> RSP: 002b:00007ffe87216bf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> >>> RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f8f102ec4e7
> >>> RDX: 0000000000000004 RSI: 00007ffe87216ce0 RDI: 0000000000000004
> >>> RBP: 00007ffe87216ce0 R08: 000055c485d835e0 R09: 00007f8f103830c0
> >>> R10: 00007f8f10382fc0 R11: 0000000000000246 R12: 0000000000000004
> >>> R13: 000055c485d7f650 R14: 0000000000000004 R15: 00007f8f103bf720
> >>>
> >>> Regards,
> >>>
> >>> Hans
> >>
> >> I sent a couple fixes (cc'd to you).
> > 
> > I've seen the fixes, thank you.
> > 
> > I'll probably add these as downstream patches to the Fedora 5.12 kernels for now
> > and see if that helps.
> 
> I'm afraid that we are still getting tpm irq kernel-backtrace reports with
> 5.12.6 which has the 2 fixes (AFAICT), here are 2 of them:
> 
> WARNING: CPU: 5 PID: 142 at drivers/char/tpm/tpm_tis_core.c:205
> tpm_tis_status+0x66/0x70
> Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer xt_CHECKSUM
> xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp
> llc ccm michael_mic nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
> nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
> nf_reject_ipv6 nft_reject nft_ct cmac nft_chain_nat ip6table_nat
> ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack
> nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security
> ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter
> snd_soc_skl_hda_dsp snd_soc_hdac_hdmi qrtr_mhi bnep snd_hda_codec_hdmi
> snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl
> snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation
> soundwire_cadence sunrpc snd_sof_intel_hda snd_sof_pci iTCO_wdt snd_sof
> intel_pmc_bxt iTCO_vendor_support snd_sof_xtensa_dsp snd_soc_hdac_hda
> snd_hda_ext_core
>  snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core mei_hdcp
> x86_pkg_temp_thermal qrtr snd_compress intel_pmt_telemetry intel_powerclamp
> intel_rapl_msr ns snd_pcm_dmaengine intel_pmt_class ac97_bus dell_laptop
> ath11k_pci coretemp ledtrig_audio ath11k dell_smm_hwmon kvm_intel snd_hda_intel
> snd_intel_dspcfg snd_intel_sdw_acpi qmi_helpers kvm snd_hda_codec mac80211
> snd_hda_core snd_hwdep snd_seq snd_seq_device irqbypass intel_cstate dell_wmi
> intel_uncore snd_pcm dell_smbios dcdbas cfg80211 snd_timer pcspkr snd wmi_bmof
> dell_wmi_sysman dell_wmi_descriptor i2c_i801 soundcore i2c_smbus mhi uvcvideo
> libarc4 hci_uart videobuf2_vmalloc videobuf2_memops mei_me videobuf2_v4l2 vfat
> mei videobuf2_common fat btqca joydev hid_sensor_als btrtl idma64 btbcm
> hid_sensor_trigger videodev hid_sensor_iio_common processor_thermal_device
> btintel industrialio_triggered_buffer processor_thermal_rfim kfifo_buf
> processor_thermal_mbox mc industrialio processor_thermal_rapl bluetooth
> thunderbolt
>  intel_rapl_common intel_pmt intel_soc_dts_iosf ecdh_generic ucsi_acpi rfkill
> typec_ucsi typec ecc int3403_thermal intel_hid int340x_thermal_zone
> sparse_keymap int3400_thermal acpi_pad acpi_tad acpi_thermal_rel zram ip_tables
> dm_crypt trusted hid_sensor_hub intel_ishtp_loader intel_ishtp_hid
> hid_multitouch i915 i2c_algo_bit rtsx_pci_sdmmc nvme drm_kms_helper mmc_core
> nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel cec drm
> ghash_clmulni_intel rtsx_pci serio_raw intel_ish_ipc intel_ishtp vmd
> i2c_hid_acpi i2c_hid wmi video pinctrl_tigerlake fuse
> CPU: 5 PID: 142 Comm: kworker/5:1 Not tainted 5.12.6-300.fc34.x86_64 #1
> Hardware name: Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021
> Workqueue: tpm_dev_wq tpm_dev_async_work
> RIP: 0010:tpm_tis_status+0x66/0x70
> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
> 44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
> d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
> RSP: 0018:ffffafc80037bd40 EFLAGS: 00010286
> RAX: 000000000000001b RBX: ffff9c8c47cff000 RCX: 0000000000000027
> RDX: ffff9c93af7585c8 RSI: 0000000000000001 RDI: ffff9c93af7585c0
> RBP: ffff9c8c47cff000 R08: 0000000000000000 R09: ffffafc80037bb70
> R10: ffffafc80037bb68 R11: ffffffffa6b45f28 R12: ffff9c8c47df5aa8
> R13: ffff9c8c4d14e0ba R14: 0000000000000000 R15: ffffafc80037bdf2
> FS:  0000000000000000(0000) GS:ffff9c93af740000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007fadaa0bd490 CR3: 0000000170c42005 CR4: 0000000000770ee0
> PKRU: 55555554
> Call Trace:
>  tpm_tis_send_data+0x2b/0x230
>  ? tpm_tcg_read_bytes+0x30/0x50
>  tpm_tis_send_main+0x1e/0xe0
>  tpm_transmit+0xd6/0x3d0WARNING: CPU: 2 PID: 1 at drivers/char/tpm/tpm_tis_core.c:205
> tpm_tis_status+0x66/0x70

Does the stack trace stop here for the first one?

> Modules linked in:
> CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.12.6-300.fc34.x86_64 #1
> Hardware name: Dell Inc. XPS 13 9310/0GG9PT, BIOS 2.2.0 04/06/2021
> RIP: 0010:tpm_tis_status+0x66/0x70
> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
> 44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
> d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
> RSP: 0018:ffffad554006bae0 EFLAGS: 00010282
> RAX: 000000000000001b RBX: ffff96bf471d5000 RCX: ffffffffa6b15ee8
> RDX: c0000000ffffdfff RSI: 00000000ffffdfff RDI: ffffffffa752ec6c
> RBP: ffff96bf471d5000 R08: 0000000000000000 R09: ffffad554006b910
> R10: ffffad554006b908 R11: ffffffffa6b45f28 R12: ffff96bf472f61a8
> R13: ffff96bf47d87000 R14: 0000000000000000 R15: ffffad554006bb92
> FS:  0000000000000000(0000) GS:ffff96c2bf680000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007ff201c82958 CR3: 0000000010a10001 CR4: 0000000000770ee0
> PKRU: 55555554
> Call Trace:
>  tpm_tis_send_data+0x2b/0x230
>  tpm_tis_send_main+0x1e/0xe0
>  tpm_transmit+0xd6/0x3d0
>  tpm_transmit_cmd+0x25/0x90
>  tpm2_pcr_extend+0x1f9/0x240
>  tpm_pcr_extend+0xa1/0xb0
>  ima_add_template_entry+0x16e/0x220
>  ? ima_store_template+0x3a/0xb0
>  ? hash_setup+0xc5/0xc5
>  ima_add_boot_aggregate+0xd4/0x13e
>  ima_init+0x51/0x94
>  init_ima+0x23/0xb5
>  ? hash_setup+0xc5/0xc5
>  do_one_initcall+0x44/0x1d0
>  kernel_init_freeable+0x1da/0x221
>  ? rest_init+0xb4/0xb4
>  kernel_init+0xa/0x11c
>  ret_from_fork+0x1f/0x30
> 
>  tpm_dev_transmit.constprop.0+0x47/0xa0
>  tpm_dev_async_work+0x62/0x90
>  process_one_work+0x1ec/0x380
>  worker_thread+0x53/0x3e0
>  ? process_one_work+0x380/0x380
>  kthread+0x11b/0x140
>  ? kthread_associate_blkcg+0xa0/0xa0
>  ret_from_fork+0x1f/0x30
> 
> 
> Regards,
> 
> Hans

OK, this is a weird one, and *might* be something unrelated, even though
it triggers the warning. tpm_pcr_extend() does pin the TPM chip and request
the locality.

For the 2nd one I'd be interested about the hardware specifics.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-27 14:00             ` Jarkko Sakkinen
@ 2021-05-27 15:27               ` Hans de Goede
       [not found]                 ` <20210531043616.u3v25qzkkrik5apq@kernel.org>
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-05-27 15:27 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, James Bottomley, linux-integrity

Hi,

On 5/27/21 4:00 PM, Jarkko Sakkinen wrote:
> On Wed, May 26, 2021 at 09:03:26PM +0200, Hans de Goede wrote:

<snip>

>> Call Trace:
>>  tpm_tis_send_data+0x2b/0x230
>>  ? tpm_tcg_read_bytes+0x30/0x50
>>  tpm_tis_send_main+0x1e/0xe0
>>  tpm_transmit+0xd6/0x3d0WARNING: CPU: 2 PID: 1 at drivers/char/tpm/tpm_tis_core.c:205
>> tpm_tis_status+0x66/0x70
> 
> Does the stack trace stop here for the first one?

No, it goes on below the second one which I copy and pasted, it looks like my cursor was
not at the end when I pasted the second one, sorry, let me paste the first one again:

WARNING: CPU: 5 PID: 142 at drivers/char/tpm/tpm_tis_core.c:205 tpm_tis_status+0x66/0x70
Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp bridge stp llc ccm michael_mic nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct cmac nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter snd_soc_skl_hda_dsp snd_soc_hdac_hdmi qrtr_mhi bnep snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence sunrpc snd_sof_intel_hda snd_sof_pci iTCO_wdt snd_sof intel_pmc_bxt iTCO_vendor_support snd_sof_xtensa_dsp snd_soc_hdac_hda snd_hda_ext_core
 snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core mei_hdcp x86_pkg_temp_thermal qrtr snd_compress intel_pmt_telemetry intel_powerclamp intel_rapl_msr ns snd_pcm_dmaengine intel_pmt_class ac97_bus dell_laptop ath11k_pci coretemp ledtrig_audio ath11k dell_smm_hwmon kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi qmi_helpers kvm snd_hda_codec mac80211 snd_hda_core snd_hwdep snd_seq snd_seq_device irqbypass intel_cstate dell_wmi intel_uncore snd_pcm dell_smbios dcdbas cfg80211 snd_timer pcspkr snd wmi_bmof dell_wmi_sysman dell_wmi_descriptor i2c_i801 soundcore i2c_smbus mhi uvcvideo libarc4 hci_uart videobuf2_vmalloc videobuf2_memops mei_me videobuf2_v4l2 vfat mei videobuf2_common fat btqca joydev hid_sensor_als btrtl idma64 btbcm hid_sensor_trigger videodev hid_sensor_iio_common processor_thermal_device btintel industrialio_triggered_buffer processor_thermal_rfim kfifo_buf processor_thermal_mbox mc industrialio processor_thermal_rapl bluetooth thunderbolt
 intel_rapl_common intel_pmt intel_soc_dts_iosf ecdh_generic ucsi_acpi rfkill typec_ucsi typec ecc int3403_thermal intel_hid int340x_thermal_zone sparse_keymap int3400_thermal acpi_pad acpi_tad acpi_thermal_rel zram ip_tables dm_crypt trusted hid_sensor_hub intel_ishtp_loader intel_ishtp_hid hid_multitouch i915 i2c_algo_bit rtsx_pci_sdmmc nvme drm_kms_helper mmc_core nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel rtsx_pci serio_raw intel_ish_ipc intel_ishtp vmd i2c_hid_acpi i2c_hid wmi video pinctrl_tigerlake fuse
CPU: 5 PID: 142 Comm: kworker/5:1 Not tainted 5.12.6-300.fc34.x86_64 #1
Hardware name: Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021
Workqueue: tpm_dev_wq tpm_dev_async_work
RIP: 0010:tpm_tis_status+0x66/0x70
Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c 44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
RSP: 0018:ffffafc80037bd40 EFLAGS: 00010286
RAX: 000000000000001b RBX: ffff9c8c47cff000 RCX: 0000000000000027
RDX: ffff9c93af7585c8 RSI: 0000000000000001 RDI: ffff9c93af7585c0
RBP: ffff9c8c47cff000 R08: 0000000000000000 R09: ffffafc80037bb70
R10: ffffafc80037bb68 R11: ffffffffa6b45f28 R12: ffff9c8c47df5aa8
R13: ffff9c8c4d14e0ba R14: 0000000000000000 R15: ffffafc80037bdf2
FS:  0000000000000000(0000) GS:ffff9c93af740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fadaa0bd490 CR3: 0000000170c42005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 tpm_tis_send_data+0x2b/0x230
 ? tpm_tcg_read_bytes+0x30/0x50
 tpm_tis_send_main+0x1e/0xe0
 tpm_transmit+0xd6/0x3d0
 tpm_dev_transmit.constprop.0+0x47/0xa0
 tpm_dev_async_work+0x62/0x90
 process_one_work+0x1ec/0x380
 worker_thread+0x53/0x3e0
 ? process_one_work+0x380/0x380
 kthread+0x11b/0x140
 ? kthread_associate_blkcg+0xa0/0xa0
 ret_from_fork+0x1f/0x30

This is from:
https://bugzilla.redhat.com/show_bug.cgi?id=1964974   (private)
https://retrace.fedoraproject.org/faf/reports/74723/  (public)



>> Modules linked in:
>> CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.12.6-300.fc34.x86_64 #1
>> Hardware name: Dell Inc. XPS 13 9310/0GG9PT, BIOS 2.2.0 04/06/2021
>> RIP: 0010:tpm_tis_status+0x66/0x70
>> Code: 23 75 05 48 83 c4 10 c3 31 c0 80 3d ca a0 55 01 00 75 f0 48 c7 c7 b4 1c
>> 44 a6 88 44 24 07 c6 05 b6 a0 55 01 01 e8 6b f3 3c 00 <0f> 0b 0f b6 44 24 07 eb
>> d0 90 0f 1f 44 00 00 41 57 41 56 41 55 41
>> RSP: 0018:ffffad554006bae0 EFLAGS: 00010282
>> RAX: 000000000000001b RBX: ffff96bf471d5000 RCX: ffffffffa6b15ee8
>> RDX: c0000000ffffdfff RSI: 00000000ffffdfff RDI: ffffffffa752ec6c
>> RBP: ffff96bf471d5000 R08: 0000000000000000 R09: ffffad554006b910
>> R10: ffffad554006b908 R11: ffffffffa6b45f28 R12: ffff96bf472f61a8
>> R13: ffff96bf47d87000 R14: 0000000000000000 R15: ffffad554006bb92
>> FS:  0000000000000000(0000) GS:ffff96c2bf680000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00007ff201c82958 CR3: 0000000010a10001 CR4: 0000000000770ee0
>> PKRU: 55555554
>> Call Trace:
>>  tpm_tis_send_data+0x2b/0x230
>>  tpm_tis_send_main+0x1e/0xe0
>>  tpm_transmit+0xd6/0x3d0
>>  tpm_transmit_cmd+0x25/0x90
>>  tpm2_pcr_extend+0x1f9/0x240
>>  tpm_pcr_extend+0xa1/0xb0
>>  ima_add_template_entry+0x16e/0x220
>>  ? ima_store_template+0x3a/0xb0
>>  ? hash_setup+0xc5/0xc5
>>  ima_add_boot_aggregate+0xd4/0x13e
>>  ima_init+0x51/0x94
>>  init_ima+0x23/0xb5
>>  ? hash_setup+0xc5/0xc5
>>  do_one_initcall+0x44/0x1d0
>>  kernel_init_freeable+0x1da/0x221
>>  ? rest_init+0xb4/0xb4
>>  kernel_init+0xa/0x11c
>>  ret_from_fork+0x1f/0x30
>>
>>  tpm_dev_transmit.constprop.0+0x47/0xa0
>>  tpm_dev_async_work+0x62/0x90
>>  process_one_work+0x1ec/0x380
>>  worker_thread+0x53/0x3e0
>>  ? process_one_work+0x380/0x380
>>  kthread+0x11b/0x140
>>  ? kthread_associate_blkcg+0xa0/0xa0
>>  ret_from_fork+0x1f/0x30
>>
>>
>> Regards,
>>
>> Hans
> 
> OK, this is a weird one, and *might* be something unrelated, even though
> it triggers the warning. tpm_pcr_extend() does pin the TPM chip and request
> the locality.
> 
> For the 2nd one I'd be interested about the hardware specifics.

Actually I just checked and both backtrace happen on a:

"Dell Inc. XPS 13 9310"

The second backtrace is from:

https://bugzilla.redhat.com/show_bug.cgi?id=1964735  (private)
https://retrace.fedoraproject.org/faf/reports/38209/ (public)

Note there is public bugzilla, with dmesg with the same backtrace
(on the same laptop), but then with 5.12.5 here:

https://bugzilla.redhat.com/show_bug.cgi?id=1963712

There are also 2 interesting comments on the public bugzilla:

"updated to linux kernel 5.12.5 performed 
sudo shutdown -r now"

"I installed Fedora 34 UEFI from USB on a Dell XPS 13 Developer Edition"

So it seems this is happening on the "Dell XPS 13 Developer
Edition".

I've also checked the BIOS versions involved in the 2 different
bugs and 1964735 has "BIOS 1.2.5 12/10/2020" where as
1963712 has "BIOS 2.2.0 04/06/2021" so this seems to be
independent of the BIOS version.

###

Interestingly enough the first backtrace is also happening on a:
"Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"

So it seems that at least with 5.12.6 (which has the last 2 fixes)
all reports are about the XPS 13 9310. I wonder if there is an
issue with the TPM interrupt line on the XPS 13 9310; I've asked the
reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
       [not found]                 ` <20210531043616.u3v25qzkkrik5apq@kernel.org>
@ 2021-05-31  8:24                   ` Hans de Goede
  2021-06-01 18:02                     ` Jarkko Sakkinen
  2021-06-01 16:04                   ` Hans de Goede
  1 sibling, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-05-31  8:24 UTC (permalink / raw)
  To: Jarkko Sakkinen, Jerry Snitselaar, Matthew Garrett,
	linux-integrity, James Bottomley

Hi,

On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> On Thu, May 27, 2021 at 05:27:49PM +0200, Hans de Goede wrote:
>> This is from:
>> https://retrace.fedoraproject.org/faf/reports/74723/  (public)
> 
> I wonder if this occurs only with O_NONBLOCK.
> 
> Any chances to get the output of
> 
>   sudo tools/testing/selftests/tpm2/test_smoke.sh
> 
> ?
> 
> It's obvious that there is some sort of bug, but it's not yet obvious that
> this bug is connected to the locality issue yet, as in this case locality
> is successfully reserved by tpm_try_get_ops() in tpm_dev_async_work()
> (driver/chars/tpm/tpm-dev-common.c).

As mentioned I've asked the user to try with tpm_tis.interrupts=0 and see
if that makes a difference. I got a reply that the user only hit this
once and that this is not (easily) reproducible :|

"looks like a spurious problem that may already be solved."

I did get permission to open up the bug (make it public) so if you want
more info it is probably easiest if you interact directly with the
reporter here:

https://bugzilla.redhat.com/show_bug.cgi?id=1964974

If you don't already have a bugzilla.redhat.com account, creating one
is super easy, you only need to enter your email address and pick a
password.


>> The second backtrace is from:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1964735  (private)
>> https://retrace.fedoraproject.org/faf/reports/38209/ (public)
> 
> This must have a successful tpm_try_get_ops() for ima_tpm_chip
> (map_tpm_chip == tpm_default_chip()).
> 
> Would also be interesting to see the status code, i.e. TPM_STS_0
> register value, but these are completely lacking the warning
> message, and the warning message apparently does not have the
> information printed.
> 
> I'll send a patch that updates the warning, and also retract
> of using WARN() given panic-on-warn is commonly set [*]. It's
> not right thing to do to torn down the machine because of invalid
> status code.
> 
> So, I'll fix that by instead:
> 
>   dev_err_once(&chip->dev, "invalid status: 0x%02x\n", status);
>   dump_stack();
> 
> Should be visible enough to get notified, and provides the important
> stack dump to quickly find possible impairment of tpm_try_get_ops(),
> and also contains the missing status code printout.

Sounds good, thanks.

>> Note there is public bugzilla, with dmesg with the same backtrace
>> (on the same laptop), but then with 5.12.5 here:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>
>> There are also 2 interesting comments on the public bugzilla:
>>
>> "updated to linux kernel 5.12.5 performed 
>> sudo shutdown -r now"
>>
>> "I installed Fedora 34 UEFI from USB on a Dell XPS 13 Developer Edition"
>>
>> So it seems this is happening on the "Dell XPS 13 Developer
>> Edition".
>>
>> I've also checked the BIOS versions involved in the 2 different
>> bugs and 1964735 has "BIOS 1.2.5 12/10/2020" where as
>> 1963712 has "BIOS 2.2.0 04/06/2021" so this seems to be
>> independent of the BIOS version.
>>
>> ###
>>
>> Interestingly enough the first backtrace is also happening on a:
>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>
>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>> all reports are about the XPS 13 9310. I wonder if there is an
>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> 
> This is helpful for sure that these all are happening on matching hardware.

Ack, I'm still waiting to hear back from reporters of the second
backtrace, wrt tpm_tis.interrupts=0.

I've marked a couple of reports of the second backtrace as duplicates of:
https://bugzilla.redhat.com/show_bug.cgi?id=1963712

So all reporters of this are now following this public bug, so if you want
to reach out to them with questions that is probably the easiest way.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
       [not found]                 ` <20210531043616.u3v25qzkkrik5apq@kernel.org>
  2021-05-31  8:24                   ` Hans de Goede
@ 2021-06-01 16:04                   ` Hans de Goede
  2021-06-01 18:03                     ` Jarkko Sakkinen
  2021-06-14 13:33                     ` Hans de Goede
  1 sibling, 2 replies; 29+ messages in thread
From: Hans de Goede @ 2021-06-01 16:04 UTC (permalink / raw)
  To: Jarkko Sakkinen, Jerry Snitselaar, Matthew Garrett,
	linux-integrity, James Bottomley

Hi,

On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>> Interestingly enough the first backtrace is also happening on a:
>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>
>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>> all reports are about the XPS 13 9310. I wonder if there is an
>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> 
> This is helpful for sure that these all are happening on matching hardware.

So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:

1. Backtrace starting with a call to ima_add_boot_aggregate
https://bugzilla.redhat.com/show_bug.cgi?id=1963712

2. Backtrace starting with a call to tpm_dev_async_work:
https://bugzilla.redhat.com/show_bug.cgi?id=1964974
(note this one is not easily reproducible)

3. Backtrace starting with a call to rng_dev_read:
https://bugzilla.redhat.com/show_bug.cgi?id=1920510

3. is the new one. All bugs linked above are public, all 3 backtraces
so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-05-31  8:24                   ` Hans de Goede
@ 2021-06-01 18:02                     ` Jarkko Sakkinen
  0 siblings, 0 replies; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-01 18:02 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Mon, May 31, 2021 at 10:24:05AM +0200, Hans de Goede wrote:
> Hi,
> 
> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> > On Thu, May 27, 2021 at 05:27:49PM +0200, Hans de Goede wrote:
> >> This is from:
> >> https://retrace.fedoraproject.org/faf/reports/74723/  (public)
> > 
> > I wonder if this occurs only with O_NONBLOCK.
> > 
> > Any chances to get the output of
> > 
> >   sudo tools/testing/selftests/tpm2/test_smoke.sh
> > 
> > ?
> > 
> > It's obvious that there is some sort of bug, but it's not yet obvious that
> > this bug is connected to the locality issue yet, as in this case locality
> > is successfully reserved by tpm_try_get_ops() in tpm_dev_async_work()
> > (driver/chars/tpm/tpm-dev-common.c).
> 
> As mentioned I've asked the user to try with tpm_tis.interrupts=0 and see
> if that makes a difference. I got a reply that the user only hit this
> once and that this is not (easily) reproducible :|

This is not about interrupts. It's about opening the file with O_NONBLOCK.
I.e. the function returns and defers sending the command.

> "looks like a spurious problem that may already be solved."
> 
> I did get permission to open up the bug (make it public) so if you want
> more info it is probably easiest if you interact directly with the
> reporter here:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> 
> If you don't already have a bugzilla.redhat.com account, creating one
> is super easy, you only need to enter your email address and pick a
> password.

I'll create one. I don't actually remember if I have one or not :-)

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-01 16:04                   ` Hans de Goede
@ 2021-06-01 18:03                     ` Jarkko Sakkinen
  2021-06-14 13:33                     ` Hans de Goede
  1 sibling, 0 replies; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-01 18:03 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Tue, Jun 01, 2021 at 06:04:40PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >> Interestingly enough the first backtrace is also happening on a:
> >> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>
> >> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >> all reports are about the XPS 13 9310. I wonder if there is an
> >> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> > 
> > This is helpful for sure that these all are happening on matching hardware.
> 
> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> 
> 1. Backtrace starting with a call to ima_add_boot_aggregate
> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> 
> 2. Backtrace starting with a call to tpm_dev_async_work:
> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> (note this one is not easily reproducible)
> 
> 3. Backtrace starting with a call to rng_dev_read:
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> 
> 3. is the new one. All bugs linked above are public, all 3 backtraces
> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> 
> Regards,
> 
> Hans

I'll put myself watcher for each of these once I have the account, thank
you (will take tomorrow, it's 9PM atm).


/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-01 16:04                   ` Hans de Goede
  2021-06-01 18:03                     ` Jarkko Sakkinen
@ 2021-06-14 13:33                     ` Hans de Goede
  2021-06-15 13:01                       ` Jarkko Sakkinen
  2021-06-21 12:04                       ` Hans de Goede
  1 sibling, 2 replies; 29+ messages in thread
From: Hans de Goede @ 2021-06-14 13:33 UTC (permalink / raw)
  To: Jarkko Sakkinen, Jerry Snitselaar, Matthew Garrett,
	linux-integrity, James Bottomley

Hi,

On 6/1/21 6:04 PM, Hans de Goede wrote:
> Hi,
> 
> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>> Interestingly enough the first backtrace is also happening on a:
>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>
>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>> all reports are about the XPS 13 9310. I wonder if there is an
>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>
>> This is helpful for sure that these all are happening on matching hardware.
> 
> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> 
> 1. Backtrace starting with a call to ima_add_boot_aggregate
> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> 
> 2. Backtrace starting with a call to tpm_dev_async_work:
> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> (note this one is not easily reproducible)
> 
> 3. Backtrace starting with a call to rng_dev_read:
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> 
> 3. is the new one. All bugs linked above are public, all 3 backtraces
> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.

Quick status update, I've got a response from a XPS 13 9310 user in:

https://bugzilla.redhat.com/show_bug.cgi?id=1920510

Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
(I expected this because all the bug-reports started when the interrupt
code got fixed/re-enabled a while ago).

Si I think that there just is something broken wrt the interrupt setup
on the XPS 13 9310 and that we should probably add an antry for the
XPS 13 9310 to the already existing tpm_tis_dmi_table pointing to the
also already existing tpm_tis_disable_irq callback.

If other people agree that that is probably the best way forward ?
then I can prepare a patch and ask the user to test this.

Regards,

Hans




^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-14 13:33                     ` Hans de Goede
@ 2021-06-15 13:01                       ` Jarkko Sakkinen
  2021-06-15 13:59                         ` Hans de Goede
  2021-06-21 12:04                       ` Hans de Goede
  1 sibling, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-15 13:01 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Mon, Jun 14, 2021 at 03:33:33PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/1/21 6:04 PM, Hans de Goede wrote:
> > Hi,
> > 
> > On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>> Interestingly enough the first backtrace is also happening on a:
> >>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>
> >>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>> all reports are about the XPS 13 9310. I wonder if there is an
> >>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>
> >> This is helpful for sure that these all are happening on matching hardware.
> > 
> > So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> > with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> > 
> > 1. Backtrace starting with a call to ima_add_boot_aggregate
> > https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> > 
> > 2. Backtrace starting with a call to tpm_dev_async_work:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> > (note this one is not easily reproducible)
> > 
> > 3. Backtrace starting with a call to rng_dev_read:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> > 
> > 3. is the new one. All bugs linked above are public, all 3 backtraces
> > so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> > and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> 
> Quick status update, I've got a response from a XPS 13 9310 user in:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> 
> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> (I expected this because all the bug-reports started when the interrupt
> code got fixed/re-enabled a while ago).
> 
> Si I think that there just is something broken wrt the interrupt setup
> on the XPS 13 9310 and that we should probably add an antry for the
> XPS 13 9310 to the already existing tpm_tis_dmi_table pointing to the
> also already existing tpm_tis_disable_irq callback.
> 
> If other people agree that that is probably the best way forward ?
> then I can prepare a patch and ask the user to test this.
> 
> Regards,
> 
> Hans

I think it all roots down to the use of TPM before tpm2_probe(), i.e.
TPM is not in expected state when tpm_chip_start() is called. I
suggested to try out adding tpm_chip_stop() right before
tpm_chip_start() in another response.

That's only thing that makes logically sense to me at least.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-15 13:01                       ` Jarkko Sakkinen
@ 2021-06-15 13:59                         ` Hans de Goede
  2021-06-23 13:37                           ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-15 13:59 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 6/15/21 3:01 PM, Jarkko Sakkinen wrote:
> On Mon, Jun 14, 2021 at 03:33:33PM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>
>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>
>>>> This is helpful for sure that these all are happening on matching hardware.
>>>
>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>
>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>
>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>> (note this one is not easily reproducible)
>>>
>>> 3. Backtrace starting with a call to rng_dev_read:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>
>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>
>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>
>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>> (I expected this because all the bug-reports started when the interrupt
>> code got fixed/re-enabled a while ago).
>>
>> Si I think that there just is something broken wrt the interrupt setup
>> on the XPS 13 9310 and that we should probably add an antry for the
>> XPS 13 9310 to the already existing tpm_tis_dmi_table pointing to the
>> also already existing tpm_tis_disable_irq callback.
>>
>> If other people agree that that is probably the best way forward ?
>> then I can prepare a patch and ask the user to test this.
>>
>> Regards,
>>
>> Hans
> 
> I think it all roots down to the use of TPM before tpm2_probe(), i.e.
> TPM is not in expected state when tpm_chip_start() is called. I
> suggested to try out adding tpm_chip_stop() right before
> tpm_chip_start() in another response.
> 
> That's only thing that makes logically sense to me at least.

Shouldn't there the be some completion somewhere with get completed
at the end of probe(), with the read-path which is involved in this
case blocking on this?

If you can prepare a kernel-patch to test, then I can build a Fedora
kernel rpm with the patch added for the user to test.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-14 13:33                     ` Hans de Goede
  2021-06-15 13:01                       ` Jarkko Sakkinen
@ 2021-06-21 12:04                       ` Hans de Goede
  2021-06-23 13:40                         ` Jarkko Sakkinen
  1 sibling, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-21 12:04 UTC (permalink / raw)
  To: Jarkko Sakkinen, Jerry Snitselaar, Matthew Garrett,
	linux-integrity, James Bottomley

Hi,

On 6/14/21 3:33 PM, Hans de Goede wrote:
> Hi,
> 
> On 6/1/21 6:04 PM, Hans de Goede wrote:
>> Hi,
>>
>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>> Interestingly enough the first backtrace is also happening on a:
>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>
>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>
>>> This is helpful for sure that these all are happening on matching hardware.
>>
>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>
>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>
>> 2. Backtrace starting with a call to tpm_dev_async_work:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>> (note this one is not easily reproducible)
>>
>> 3. Backtrace starting with a call to rng_dev_read:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>
>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> 
> Quick status update, I've got a response from a XPS 13 9310 user in:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> 
> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> (I expected this because all the bug-reports started when the interrupt
> code got fixed/re-enabled a while ago).

One more status update.

- A new 4th variant of the backtrace has been spotted, where the problem hits
when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1958381

- So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
models. But the new variant is happening on a Dell XPS 15 9500 and the
backtrace starting at ima_add_boot_aggregate is also being reported on
a Dell XPS 15 9500 (as well as on the XPS 13 9310).

Regards,

Hans



^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-15 13:59                         ` Hans de Goede
@ 2021-06-23 13:37                           ` Jarkko Sakkinen
  0 siblings, 0 replies; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-23 13:37 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Tue, Jun 15, 2021 at 03:59:06PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/15/21 3:01 PM, Jarkko Sakkinen wrote:
> > On Mon, Jun 14, 2021 at 03:33:33PM +0200, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 6/1/21 6:04 PM, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>>> Interestingly enough the first backtrace is also happening on a:
> >>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>>
> >>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>>
> >>>> This is helpful for sure that these all are happening on matching hardware.
> >>>
> >>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>>
> >>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>
> >>> 2. Backtrace starting with a call to tpm_dev_async_work:
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>> (note this one is not easily reproducible)
> >>>
> >>> 3. Backtrace starting with a call to rng_dev_read:
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>
> >>> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> >>
> >> Quick status update, I've got a response from a XPS 13 9310 user in:
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>
> >> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> >> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> >> (I expected this because all the bug-reports started when the interrupt
> >> code got fixed/re-enabled a while ago).
> >>
> >> Si I think that there just is something broken wrt the interrupt setup
> >> on the XPS 13 9310 and that we should probably add an antry for the
> >> XPS 13 9310 to the already existing tpm_tis_dmi_table pointing to the
> >> also already existing tpm_tis_disable_irq callback.
> >>
> >> If other people agree that that is probably the best way forward ?
> >> then I can prepare a patch and ask the user to test this.
> >>
> >> Regards,
> >>
> >> Hans
> > 
> > I think it all roots down to the use of TPM before tpm2_probe(), i.e.
> > TPM is not in expected state when tpm_chip_start() is called. I
> > suggested to try out adding tpm_chip_stop() right before
> > tpm_chip_start() in another response.
> > 
> > That's only thing that makes logically sense to me at least.
> 
> Shouldn't there the be some completion somewhere with get completed
> at the end of probe(), with the read-path which is involved in this
> case blocking on this?
> 
> If you can prepare a kernel-patch to test, then I can build a Fedora
> kernel rpm with the patch added for the user to test.
> 
> Regards,
> 
> Hans

After rc1 PR I'll have to check through the hardware that I have if
something would reproduce the issue locally. This is too blind shooting
ATM.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-21 12:04                       ` Hans de Goede
@ 2021-06-23 13:40                         ` Jarkko Sakkinen
  2021-06-23 13:54                           ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-23 13:40 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/14/21 3:33 PM, Hans de Goede wrote:
> > Hi,
> > 
> > On 6/1/21 6:04 PM, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>> Interestingly enough the first backtrace is also happening on a:
> >>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>
> >>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>
> >>> This is helpful for sure that these all are happening on matching hardware.
> >>
> >> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>
> >> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>
> >> 2. Backtrace starting with a call to tpm_dev_async_work:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >> (note this one is not easily reproducible)
> >>
> >> 3. Backtrace starting with a call to rng_dev_read:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>
> >> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> > 
> > Quick status update, I've got a response from a XPS 13 9310 user in:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> > 
> > Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> > and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> > (I expected this because all the bug-reports started when the interrupt
> > code got fixed/re-enabled a while ago).
> 
> One more status update.
> 
> - A new 4th variant of the backtrace has been spotted, where the problem hits
> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
> 
> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
> models. But the new variant is happening on a Dell XPS 15 9500 and the
> backtrace starting at ima_add_boot_aggregate is also being reported on
> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
> 
> Regards,
> 
> Hans

OK, I'll have to query if I could borrow that laptop from someone. It's
fairly common laptop, i.e. might be possible.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces
  2021-06-23 13:40                         ` Jarkko Sakkinen
@ 2021-06-23 13:54                           ` Hans de Goede
  2021-06-29 18:04                             ` Recent tpm_tis IRQ handling changes are causing kernel backtraces] Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-23 13:54 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 6/14/21 3:33 PM, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>>
>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>>
>>>>> This is helpful for sure that these all are happening on matching hardware.
>>>>
>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>>
>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>
>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>>> (note this one is not easily reproducible)
>>>>
>>>> 3. Backtrace starting with a call to rng_dev_read:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>
>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>>
>>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>
>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>>> (I expected this because all the bug-reports started when the interrupt
>>> code got fixed/re-enabled a while ago).
>>
>> One more status update.
>>
>> - A new 4th variant of the backtrace has been spotted, where the problem hits
>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
>>
>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
>> models. But the new variant is happening on a Dell XPS 15 9500 and the
>> backtrace starting at ima_add_boot_aggregate is also being reported on
>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
>>
>> Regards,
>>
>> Hans
> 
> OK, I'll have to query if I could borrow that laptop from someone. It's
> fairly common laptop, i.e. might be possible.

In the mean time I've also got a report that this variant of the backtrace:

1. Backtrace starting with a call to ima_add_boot_aggregate
https://bugzilla.redhat.com/show_bug.cgi?id=1963712

Is also still happening with recent 5.12.y kernels on
Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
an icelake processor. So the common denominator seems to be that they are
all 2020 Dell laptop models using the latest Intel CPUs.

FYI the complete list of models on which some of the 4 backtrace variants
are still seen on recent 5.12.y kernels is now:

Dell XPS 13 9310
Dell XPS 15 9500
Dell Precision 7750

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-23 13:54                           ` Hans de Goede
@ 2021-06-29 18:04                             ` Jarkko Sakkinen
  2021-06-29 19:14                               ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-29 18:04 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
> > On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 6/14/21 3:33 PM, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 6/1/21 6:04 PM, Hans de Goede wrote:
> >>>> Hi,
> >>>>
> >>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>>>> Interestingly enough the first backtrace is also happening on a:
> >>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>>>
> >>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>>>
> >>>>> This is helpful for sure that these all are happening on matching hardware.
> >>>>
> >>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>>>
> >>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>
> >>>> 2. Backtrace starting with a call to tpm_dev_async_work:
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>>> (note this one is not easily reproducible)
> >>>>
> >>>> 3. Backtrace starting with a call to rng_dev_read:
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>
> >>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> >>>
> >>> Quick status update, I've got a response from a XPS 13 9310 user in:
> >>>
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>
> >>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> >>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> >>> (I expected this because all the bug-reports started when the interrupt
> >>> code got fixed/re-enabled a while ago).
> >>
> >> One more status update.
> >>
> >> - A new 4th variant of the backtrace has been spotted, where the problem hits
> >> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
> >>
> >> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
> >> models. But the new variant is happening on a Dell XPS 15 9500 and the
> >> backtrace starting at ima_add_boot_aggregate is also being reported on
> >> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
> >>
> >> Regards,
> >>
> >> Hans
> > 
> > OK, I'll have to query if I could borrow that laptop from someone. It's
> > fairly common laptop, i.e. might be possible.
> 
> In the mean time I've also got a report that this variant of the backtrace:
> 
> 1. Backtrace starting with a call to ima_add_boot_aggregate
> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> 
> Is also still happening with recent 5.12.y kernels on
> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
> an icelake processor. So the common denominator seems to be that they are
> all 2020 Dell laptop models using the latest Intel CPUs.
> 
> FYI the complete list of models on which some of the 4 backtrace variants
> are still seen on recent 5.12.y kernels is now:
> 
> Dell XPS 13 9310
> Dell XPS 15 9500
> Dell Precision 7750
> 
> Regards,
> 
> Hans

Does "tpm_tis.interrupts=0" uniformly workaround the issue?

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-29 18:04                             ` Recent tpm_tis IRQ handling changes are causing kernel backtraces] Jarkko Sakkinen
@ 2021-06-29 19:14                               ` Hans de Goede
  2021-06-29 22:05                                 ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-29 19:14 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
>>>>> Hi,
>>>>>
>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>>>>
>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>>>>
>>>>>>> This is helpful for sure that these all are happening on matching hardware.
>>>>>>
>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>>>>
>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>>
>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>>>>> (note this one is not easily reproducible)
>>>>>>
>>>>>> 3. Backtrace starting with a call to rng_dev_read:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>
>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>>>>
>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>>>>
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>
>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>>>>> (I expected this because all the bug-reports started when the interrupt
>>>>> code got fixed/re-enabled a while ago).
>>>>
>>>> One more status update.
>>>>
>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
>>>>
>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>
>>> OK, I'll have to query if I could borrow that laptop from someone. It's
>>> fairly common laptop, i.e. might be possible.
>>
>> In the mean time I've also got a report that this variant of the backtrace:
>>
>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>
>> Is also still happening with recent 5.12.y kernels on
>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
>> an icelake processor. So the common denominator seems to be that they are
>> all 2020 Dell laptop models using the latest Intel CPUs.
>>
>> FYI the complete list of models on which some of the 4 backtrace variants
>> are still seen on recent 5.12.y kernels is now:
>>
>> Dell XPS 13 9310
>> Dell XPS 15 9500
>> Dell Precision 7750
>>
>> Regards,
>>
>> Hans
> 
> Does "tpm_tis.interrupts=0" uniformly workaround the issue?

I unfortunately have not gotten much replies to my request to test with
tpm_tis.interrupts=0, but for those people who have bothered to test
(2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-29 19:14                               ` Hans de Goede
@ 2021-06-29 22:05                                 ` Jarkko Sakkinen
  2021-06-30 12:47                                   ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-06-29 22:05 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
> > On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
> >>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
> >>>> Hi,
> >>>>
> >>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>>>>>> Interestingly enough the first backtrace is also happening on a:
> >>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>>>>>
> >>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>>>>>
> >>>>>>> This is helpful for sure that these all are happening on matching hardware.
> >>>>>>
> >>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>>>>>
> >>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>>>
> >>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>>>>> (note this one is not easily reproducible)
> >>>>>>
> >>>>>> 3. Backtrace starting with a call to rng_dev_read:
> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>>
> >>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> >>>>>
> >>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
> >>>>>
> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>
> >>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> >>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> >>>>> (I expected this because all the bug-reports started when the interrupt
> >>>>> code got fixed/re-enabled a while ago).
> >>>>
> >>>> One more status update.
> >>>>
> >>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
> >>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
> >>>>
> >>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
> >>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
> >>>> backtrace starting at ima_add_boot_aggregate is also being reported on
> >>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>
> >>> OK, I'll have to query if I could borrow that laptop from someone. It's
> >>> fairly common laptop, i.e. might be possible.
> >>
> >> In the mean time I've also got a report that this variant of the backtrace:
> >>
> >> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>
> >> Is also still happening with recent 5.12.y kernels on
> >> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
> >> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
> >> an icelake processor. So the common denominator seems to be that they are
> >> all 2020 Dell laptop models using the latest Intel CPUs.
> >>
> >> FYI the complete list of models on which some of the 4 backtrace variants
> >> are still seen on recent 5.12.y kernels is now:
> >>
> >> Dell XPS 13 9310
> >> Dell XPS 15 9500
> >> Dell Precision 7750
> >>
> >> Regards,
> >>
> >> Hans
> > 
> > Does "tpm_tis.interrupts=0" uniformly workaround the issue?
> 
> I unfortunately have not gotten much replies to my request to test with
> tpm_tis.interrupts=0, but for those people who have bothered to test
> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.

So we see this in dmesg as first anything from TPM:

[    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)

This means that one command is successfully processed by the TPM, i.e.
tpm2_probe() in tpm_tis_core_init().

My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
this code path would be executed:

        if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
                dev_err(&chip->dev, FW_BUG
                        "TPM interrupt not working, polling instead\n");

                disable_interrupts(chip);
        }

TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
a legit value is given to IRQ by ACPI.  We are probably planning re-enable
IRQ code after these type of issues are fully resolved, but right now you
should not end up having it enabled (see tpm_tis_send() function).

To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
never executed. And early in the same function the interrupt hardware is
*explicitly* disabled.

For me this looks like a hardware bug right now: interrupts stay enabled
for some reason.

ACPI dump would be useful to verify some of the assumptions in this.

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-29 22:05                                 ` Jarkko Sakkinen
@ 2021-06-30 12:47                                   ` Hans de Goede
  2021-06-30 13:36                                     ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-30 12:47 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 6/30/21 12:05 AM, Jarkko Sakkinen wrote:
> On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
>>> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
>>>> Hi,
>>>>
>>>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
>>>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>>>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>>>>>>
>>>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>>>>>>
>>>>>>>>> This is helpful for sure that these all are happening on matching hardware.
>>>>>>>>
>>>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>>>>>>
>>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>>>>
>>>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>>>>>>> (note this one is not easily reproducible)
>>>>>>>>
>>>>>>>> 3. Backtrace starting with a call to rng_dev_read:
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>>
>>>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>>>>>>
>>>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>>>>>>
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>
>>>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>>>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>>>>>>> (I expected this because all the bug-reports started when the interrupt
>>>>>>> code got fixed/re-enabled a while ago).
>>>>>>
>>>>>> One more status update.
>>>>>>
>>>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
>>>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
>>>>>>
>>>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
>>>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
>>>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
>>>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Hans
>>>>>
>>>>> OK, I'll have to query if I could borrow that laptop from someone. It's
>>>>> fairly common laptop, i.e. might be possible.
>>>>
>>>> In the mean time I've also got a report that this variant of the backtrace:
>>>>
>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>
>>>> Is also still happening with recent 5.12.y kernels on
>>>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
>>>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
>>>> an icelake processor. So the common denominator seems to be that they are
>>>> all 2020 Dell laptop models using the latest Intel CPUs.
>>>>
>>>> FYI the complete list of models on which some of the 4 backtrace variants
>>>> are still seen on recent 5.12.y kernels is now:
>>>>
>>>> Dell XPS 13 9310
>>>> Dell XPS 15 9500
>>>> Dell Precision 7750
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>
>>> Does "tpm_tis.interrupts=0" uniformly workaround the issue?
>>
>> I unfortunately have not gotten much replies to my request to test with
>> tpm_tis.interrupts=0, but for those people who have bothered to test
>> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.
> 
> So we see this in dmesg as first anything from TPM:
> 
> [    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)
> 
> This means that one command is successfully processed by the TPM, i.e.
> tpm2_probe() in tpm_tis_core_init().
> 
> My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
> confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
> this code path would be executed:
> 
>         if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
>                 dev_err(&chip->dev, FW_BUG
>                         "TPM interrupt not working, polling instead\n");
> 
>                 disable_interrupts(chip);
>         }
> 
> TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
> a legit value is given to IRQ by ACPI.  We are probably planning re-enable
> IRQ code after these type of issues are fully resolved, but right now you
> should not end up having it enabled (see tpm_tis_send() function).
> 
> To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
> never executed. And early in the same function the interrupt hardware is
> *explicitly* disabled.
> 
> For me this looks like a hardware bug right now: interrupts stay enabled
> for some reason.
> 
> ACPI dump would be useful to verify some of the assumptions in this.

Ok, I've added a comment to the Fedora bugs for the 4 different backtrace
variants asking for acpidumps for the Dell XPS 13 9310, Dell XPS 15 9500
and Dell Precision 7750 laptops.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-30 12:47                                   ` Hans de Goede
@ 2021-06-30 13:36                                     ` Hans de Goede
  2021-07-09 18:44                                       ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-06-30 13:36 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 6/30/21 2:47 PM, Hans de Goede wrote:
> Hi,
> 
> On 6/30/21 12:05 AM, Jarkko Sakkinen wrote:
>> On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
>>>> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
>>>>> Hi,
>>>>>
>>>>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
>>>>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>>>>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>>>>>>>
>>>>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>>>>>>>
>>>>>>>>>> This is helpful for sure that these all are happening on matching hardware.
>>>>>>>>>
>>>>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>>>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>>>>>>>
>>>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>>>>>
>>>>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>>>>>>>> (note this one is not easily reproducible)
>>>>>>>>>
>>>>>>>>> 3. Backtrace starting with a call to rng_dev_read:
>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>>>
>>>>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>>>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>>>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>>>>>>>
>>>>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>>>>>>>
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>>
>>>>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>>>>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>>>>>>>> (I expected this because all the bug-reports started when the interrupt
>>>>>>>> code got fixed/re-enabled a while ago).
>>>>>>>
>>>>>>> One more status update.
>>>>>>>
>>>>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
>>>>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
>>>>>>>
>>>>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
>>>>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
>>>>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
>>>>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Hans
>>>>>>
>>>>>> OK, I'll have to query if I could borrow that laptop from someone. It's
>>>>>> fairly common laptop, i.e. might be possible.
>>>>>
>>>>> In the mean time I've also got a report that this variant of the backtrace:
>>>>>
>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>
>>>>> Is also still happening with recent 5.12.y kernels on
>>>>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
>>>>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
>>>>> an icelake processor. So the common denominator seems to be that they are
>>>>> all 2020 Dell laptop models using the latest Intel CPUs.
>>>>>
>>>>> FYI the complete list of models on which some of the 4 backtrace variants
>>>>> are still seen on recent 5.12.y kernels is now:
>>>>>
>>>>> Dell XPS 13 9310
>>>>> Dell XPS 15 9500
>>>>> Dell Precision 7750
>>>>>
>>>>> Regards,
>>>>>
>>>>> Hans
>>>>
>>>> Does "tpm_tis.interrupts=0" uniformly workaround the issue?
>>>
>>> I unfortunately have not gotten much replies to my request to test with
>>> tpm_tis.interrupts=0, but for those people who have bothered to test
>>> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.
>>
>> So we see this in dmesg as first anything from TPM:
>>
>> [    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)
>>
>> This means that one command is successfully processed by the TPM, i.e.
>> tpm2_probe() in tpm_tis_core_init().
>>
>> My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
>> confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
>> this code path would be executed:
>>
>>         if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
>>                 dev_err(&chip->dev, FW_BUG
>>                         "TPM interrupt not working, polling instead\n");
>>
>>                 disable_interrupts(chip);
>>         }
>>
>> TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
>> a legit value is given to IRQ by ACPI.  We are probably planning re-enable
>> IRQ code after these type of issues are fully resolved, but right now you
>> should not end up having it enabled (see tpm_tis_send() function).
>>
>> To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
>> never executed. And early in the same function the interrupt hardware is
>> *explicitly* disabled.
>>
>> For me this looks like a hardware bug right now: interrupts stay enabled
>> for some reason.
>>
>> ACPI dump would be useful to verify some of the assumptions in this.
> 
> Ok, I've added a comment to the Fedora bugs for the 4 different backtrace
> variants asking for acpidumps for the Dell XPS 13 9310, Dell XPS 15 9500
> and Dell Precision 7750 laptops.

2 XPS 9310 acpidumps have been attached to:

https://bugzilla.redhat.com/show_bug.cgi?id=1920510
https://bugzilla.redhat.com/show_bug.cgi?id=1964974

Note the reporter of the first bug mentions that he is no longer having
this issue, but we are definitely still getting reports for kernel version
> 5.12.6 (which has the last 2 fixes) from XPS 9310 users...

Maybe there are different BIOS versions in play ? It might be interesting
to compare the 2 acpidumps...

Regards,

Hans




^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-06-30 13:36                                     ` Hans de Goede
@ 2021-07-09 18:44                                       ` Jarkko Sakkinen
  2021-07-17 16:10                                         ` Hans de Goede
  0 siblings, 1 reply; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-07-09 18:44 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Wed, Jun 30, 2021 at 03:36:55PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 6/30/21 2:47 PM, Hans de Goede wrote:
> > Hi,
> > 
> > On 6/30/21 12:05 AM, Jarkko Sakkinen wrote:
> >> On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
> >>>> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
> >>>>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>>>>>>>>> Interestingly enough the first backtrace is also happening on a:
> >>>>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>>>>>>>>
> >>>>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>>>>>>>>
> >>>>>>>>>> This is helpful for sure that these all are happening on matching hardware.
> >>>>>>>>>
> >>>>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >>>>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>>>>>>>>
> >>>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>>>>>>
> >>>>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
> >>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>>>>>>>> (note this one is not easily reproducible)
> >>>>>>>>>
> >>>>>>>>> 3. Backtrace starting with a call to rng_dev_read:
> >>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>>>>>
> >>>>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >>>>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >>>>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> >>>>>>>>
> >>>>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
> >>>>>>>>
> >>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>>>>
> >>>>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> >>>>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> >>>>>>>> (I expected this because all the bug-reports started when the interrupt
> >>>>>>>> code got fixed/re-enabled a while ago).
> >>>>>>>
> >>>>>>> One more status update.
> >>>>>>>
> >>>>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
> >>>>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
> >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
> >>>>>>>
> >>>>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
> >>>>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
> >>>>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
> >>>>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Hans
> >>>>>>
> >>>>>> OK, I'll have to query if I could borrow that laptop from someone. It's
> >>>>>> fairly common laptop, i.e. might be possible.
> >>>>>
> >>>>> In the mean time I've also got a report that this variant of the backtrace:
> >>>>>
> >>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>>
> >>>>> Is also still happening with recent 5.12.y kernels on
> >>>>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
> >>>>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
> >>>>> an icelake processor. So the common denominator seems to be that they are
> >>>>> all 2020 Dell laptop models using the latest Intel CPUs.
> >>>>>
> >>>>> FYI the complete list of models on which some of the 4 backtrace variants
> >>>>> are still seen on recent 5.12.y kernels is now:
> >>>>>
> >>>>> Dell XPS 13 9310
> >>>>> Dell XPS 15 9500
> >>>>> Dell Precision 7750
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Hans
> >>>>
> >>>> Does "tpm_tis.interrupts=0" uniformly workaround the issue?
> >>>
> >>> I unfortunately have not gotten much replies to my request to test with
> >>> tpm_tis.interrupts=0, but for those people who have bothered to test
> >>> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.
> >>
> >> So we see this in dmesg as first anything from TPM:
> >>
> >> [    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)
> >>
> >> This means that one command is successfully processed by the TPM, i.e.
> >> tpm2_probe() in tpm_tis_core_init().
> >>
> >> My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
> >> confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
> >> this code path would be executed:
> >>
> >>         if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
> >>                 dev_err(&chip->dev, FW_BUG
> >>                         "TPM interrupt not working, polling instead\n");
> >>
> >>                 disable_interrupts(chip);
> >>         }
> >>
> >> TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
> >> a legit value is given to IRQ by ACPI.  We are probably planning re-enable
> >> IRQ code after these type of issues are fully resolved, but right now you
> >> should not end up having it enabled (see tpm_tis_send() function).
> >>
> >> To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
> >> never executed. And early in the same function the interrupt hardware is
> >> *explicitly* disabled.
> >>
> >> For me this looks like a hardware bug right now: interrupts stay enabled
> >> for some reason.
> >>
> >> ACPI dump would be useful to verify some of the assumptions in this.
> > 
> > Ok, I've added a comment to the Fedora bugs for the 4 different backtrace
> > variants asking for acpidumps for the Dell XPS 13 9310, Dell XPS 15 9500
> > and Dell Precision 7750 laptops.
> 
> 2 XPS 9310 acpidumps have been attached to:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> 
> Note the reporter of the first bug mentions that he is no longer having
> this issue, but we are definitely still getting reports for kernel version
> > 5.12.6 (which has the last 2 fixes) from XPS 9310 users...
> 
> Maybe there are different BIOS versions in play ? It might be interesting
> to compare the 2 acpidumps...

❯ diff -u ../tmp/ssdt7.dsl ../tmp2/ssdt7.dsl
--- ../tmp/ssdt7.dsl	2021-07-09 21:32:06.473166420 +0300
+++ ../tmp2/ssdt7.dsl	2021-07-09 21:33:45.065934469 +0300
@@ -5,7 +5,7 @@
  * 
  * Disassembling to symbolic ASL+ operators
  *
- * Disassembly of ssdt7.dat, Fri Jul  9 21:32:06 2021
+ * Disassembly of ssdt7.dat, Fri Jul  9 21:33:45 2021
  *
  * Original Table Header:
  *     Signature        "SSDT"
@@ -121,7 +121,7 @@
                     0xFED40000,         // Address Base
                     0x00005000,         // Address Length
                     )
-                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y58)
+                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y55)
                 {
                     0x0000000C,
                 }
@@ -141,7 +141,7 @@
                 }
                 Else
                 {
-                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
+                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
                     LIRQ = IRQN /* \_SB_.TPM_.IRQN */
                     Return (RES0) /* \_SB_.TPM_.RES0 */
                 }
@@ -152,14 +152,14 @@
                 If ((IRQN != Zero))
                 {
                     CreateDWordField (Arg0, 0x11, IRQ0)
-                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
+                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
                     LIRQ = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
                     IRQN = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
                     CreateBitField (Arg0, 0x79, ITRG)
-                    CreateBitField (RES0, \_SB.TPM._Y58._HE, LTRG)  // _HE_: High-Edge
+                    CreateBitField (RES0, \_SB.TPM._Y55._HE, LTRG)  // _HE_: High-Edge
                     LTRG = ITRG /* \_SB_.TPM_._SRS.ITRG */
                     CreateBitField (Arg0, 0x7A, ILVL)
-                    CreateBitField (RES0, \_SB.TPM._Y58._LL, LLVL)  // _LL_: Low Level
+                    CreateBitField (RES0, \_SB.TPM._Y55._LL, LLVL)  // _LL_: Low Level
                     LLVL = ILVL /* \_SB_.TPM_._SRS.ILVL */
                     If ((((TID0 & 0x0F) == Zero) || ((TID0 & 0x0F
                         ) == 0x0F)))

This delta from "acpidump for Dell XPS 9310 (with Qualcomm QCA6390)" to
"acpidump output from a Dell XPS 13 9310 that no longer has a problem"
in SSDT7. The bug I'm referring to is

https://bugzilla.redhat.com/show_bug.cgi?id=1920510

Looks to me just using a different label.

What if we just set "interrupts=0" explicitly for STM0125 HID since the
workaround seems to work according to the report?

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-07-09 18:44                                       ` Jarkko Sakkinen
@ 2021-07-17 16:10                                         ` Hans de Goede
  2021-07-27  2:50                                           ` Jarkko Sakkinen
  0 siblings, 1 reply; 29+ messages in thread
From: Hans de Goede @ 2021-07-17 16:10 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

Hi,

On 7/9/21 8:44 PM, Jarkko Sakkinen wrote:
> On Wed, Jun 30, 2021 at 03:36:55PM +0200, Hans de Goede wrote:
>> Hi,
>>
>> On 6/30/21 2:47 PM, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 6/30/21 12:05 AM, Jarkko Sakkinen wrote:
>>>> On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
>>>>> Hi,
>>>>>
>>>>> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
>>>>>> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
>>>>>>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
>>>>>>>>>>>>> Interestingly enough the first backtrace is also happening on a:
>>>>>>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
>>>>>>>>>>>>>
>>>>>>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
>>>>>>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
>>>>>>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
>>>>>>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
>>>>>>>>>>>>
>>>>>>>>>>>> This is helpful for sure that these all are happening on matching hardware.
>>>>>>>>>>>
>>>>>>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
>>>>>>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
>>>>>>>>>>>
>>>>>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>>>>>>>
>>>>>>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>>>>>>>>>> (note this one is not easily reproducible)
>>>>>>>>>>>
>>>>>>>>>>> 3. Backtrace starting with a call to rng_dev_read:
>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>>>>>
>>>>>>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
>>>>>>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
>>>>>>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
>>>>>>>>>>
>>>>>>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
>>>>>>>>>>
>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>>>>>>>>>>
>>>>>>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
>>>>>>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
>>>>>>>>>> (I expected this because all the bug-reports started when the interrupt
>>>>>>>>>> code got fixed/re-enabled a while ago).
>>>>>>>>>
>>>>>>>>> One more status update.
>>>>>>>>>
>>>>>>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
>>>>>>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
>>>>>>>>>
>>>>>>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
>>>>>>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
>>>>>>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
>>>>>>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Hans
>>>>>>>>
>>>>>>>> OK, I'll have to query if I could borrow that laptop from someone. It's
>>>>>>>> fairly common laptop, i.e. might be possible.
>>>>>>>
>>>>>>> In the mean time I've also got a report that this variant of the backtrace:
>>>>>>>
>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
>>>>>>>
>>>>>>> Is also still happening with recent 5.12.y kernels on
>>>>>>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
>>>>>>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
>>>>>>> an icelake processor. So the common denominator seems to be that they are
>>>>>>> all 2020 Dell laptop models using the latest Intel CPUs.
>>>>>>>
>>>>>>> FYI the complete list of models on which some of the 4 backtrace variants
>>>>>>> are still seen on recent 5.12.y kernels is now:
>>>>>>>
>>>>>>> Dell XPS 13 9310
>>>>>>> Dell XPS 15 9500
>>>>>>> Dell Precision 7750
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Hans
>>>>>>
>>>>>> Does "tpm_tis.interrupts=0" uniformly workaround the issue?
>>>>>
>>>>> I unfortunately have not gotten much replies to my request to test with
>>>>> tpm_tis.interrupts=0, but for those people who have bothered to test
>>>>> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.
>>>>
>>>> So we see this in dmesg as first anything from TPM:
>>>>
>>>> [    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)
>>>>
>>>> This means that one command is successfully processed by the TPM, i.e.
>>>> tpm2_probe() in tpm_tis_core_init().
>>>>
>>>> My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
>>>> confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
>>>> this code path would be executed:
>>>>
>>>>         if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
>>>>                 dev_err(&chip->dev, FW_BUG
>>>>                         "TPM interrupt not working, polling instead\n");
>>>>
>>>>                 disable_interrupts(chip);
>>>>         }
>>>>
>>>> TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
>>>> a legit value is given to IRQ by ACPI.  We are probably planning re-enable
>>>> IRQ code after these type of issues are fully resolved, but right now you
>>>> should not end up having it enabled (see tpm_tis_send() function).
>>>>
>>>> To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
>>>> never executed. And early in the same function the interrupt hardware is
>>>> *explicitly* disabled.
>>>>
>>>> For me this looks like a hardware bug right now: interrupts stay enabled
>>>> for some reason.
>>>>
>>>> ACPI dump would be useful to verify some of the assumptions in this.
>>>
>>> Ok, I've added a comment to the Fedora bugs for the 4 different backtrace
>>> variants asking for acpidumps for the Dell XPS 13 9310, Dell XPS 15 9500
>>> and Dell Precision 7750 laptops.
>>
>> 2 XPS 9310 acpidumps have been attached to:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
>>
>> Note the reporter of the first bug mentions that he is no longer having
>> this issue, but we are definitely still getting reports for kernel version
>>> 5.12.6 (which has the last 2 fixes) from XPS 9310 users...
>>
>> Maybe there are different BIOS versions in play ? It might be interesting
>> to compare the 2 acpidumps...
> 
> ❯ diff -u ../tmp/ssdt7.dsl ../tmp2/ssdt7.dsl
> --- ../tmp/ssdt7.dsl	2021-07-09 21:32:06.473166420 +0300
> +++ ../tmp2/ssdt7.dsl	2021-07-09 21:33:45.065934469 +0300
> @@ -5,7 +5,7 @@
>   * 
>   * Disassembling to symbolic ASL+ operators
>   *
> - * Disassembly of ssdt7.dat, Fri Jul  9 21:32:06 2021
> + * Disassembly of ssdt7.dat, Fri Jul  9 21:33:45 2021
>   *
>   * Original Table Header:
>   *     Signature        "SSDT"
> @@ -121,7 +121,7 @@
>                      0xFED40000,         // Address Base
>                      0x00005000,         // Address Length
>                      )
> -                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y58)
> +                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y55)
>                  {
>                      0x0000000C,
>                  }
> @@ -141,7 +141,7 @@
>                  }
>                  Else
>                  {
> -                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
> +                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
>                      LIRQ = IRQN /* \_SB_.TPM_.IRQN */
>                      Return (RES0) /* \_SB_.TPM_.RES0 */
>                  }
> @@ -152,14 +152,14 @@
>                  If ((IRQN != Zero))
>                  {
>                      CreateDWordField (Arg0, 0x11, IRQ0)
> -                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
> +                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
>                      LIRQ = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
>                      IRQN = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
>                      CreateBitField (Arg0, 0x79, ITRG)
> -                    CreateBitField (RES0, \_SB.TPM._Y58._HE, LTRG)  // _HE_: High-Edge
> +                    CreateBitField (RES0, \_SB.TPM._Y55._HE, LTRG)  // _HE_: High-Edge
>                      LTRG = ITRG /* \_SB_.TPM_._SRS.ITRG */
>                      CreateBitField (Arg0, 0x7A, ILVL)
> -                    CreateBitField (RES0, \_SB.TPM._Y58._LL, LLVL)  // _LL_: Low Level
> +                    CreateBitField (RES0, \_SB.TPM._Y55._LL, LLVL)  // _LL_: Low Level
>                      LLVL = ILVL /* \_SB_.TPM_._SRS.ILVL */
>                      If ((((TID0 & 0x0F) == Zero) || ((TID0 & 0x0F
>                          ) == 0x0F)))
> 
> This delta from "acpidump for Dell XPS 9310 (with Qualcomm QCA6390)" to
> "acpidump output from a Dell XPS 13 9310 that no longer has a problem"
> in SSDT7. The bug I'm referring to is
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> 
> Looks to me just using a different label.

Yes, although I have the feeling that does indicate that their are possibly
other changes under the hood. The 0x00000000c interrupt referred to seems to
be an interrupt directly on the APIC, which means that it is a GPIO in direct-irq
mode with level vs edge interrupt mode selection (pullup/down settings is all
directly done by the BIOS.

I'm aware that misconfiguring those settings (which Linux cannot see) was
an issue with the TPM IRQ on some Lenovo models, maybe the same is going on
here; and later BIOS versions contain a fix (and this somehow also has
changed the label in the DSDT ?).

> What if we just set "interrupts=0" explicitly for STM0125 HID since the
> workaround seems to work according to the report?

That sounds like a reasonable workaround.

Regards,

Hans


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Recent tpm_tis IRQ handling changes are causing kernel backtraces]
  2021-07-17 16:10                                         ` Hans de Goede
@ 2021-07-27  2:50                                           ` Jarkko Sakkinen
  0 siblings, 0 replies; 29+ messages in thread
From: Jarkko Sakkinen @ 2021-07-27  2:50 UTC (permalink / raw)
  To: Hans de Goede
  Cc: Jerry Snitselaar, Matthew Garrett, linux-integrity, James Bottomley

On Sat, Jul 17, 2021 at 06:10:04PM +0200, Hans de Goede wrote:
> Hi,
> 
> On 7/9/21 8:44 PM, Jarkko Sakkinen wrote:
> > On Wed, Jun 30, 2021 at 03:36:55PM +0200, Hans de Goede wrote:
> >> Hi,
> >>
> >> On 6/30/21 2:47 PM, Hans de Goede wrote:
> >>> Hi,
> >>>
> >>> On 6/30/21 12:05 AM, Jarkko Sakkinen wrote:
> >>>> On Tue, Jun 29, 2021 at 09:14:39PM +0200, Hans de Goede wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 6/29/21 8:04 PM, Jarkko Sakkinen wrote:
> >>>>>> On Wed, Jun 23, 2021 at 03:54:59PM +0200, Hans de Goede wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On 6/23/21 3:40 PM, Jarkko Sakkinen wrote:
> >>>>>>>> On Mon, Jun 21, 2021 at 02:04:52PM +0200, Hans de Goede wrote:
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> On 6/14/21 3:33 PM, Hans de Goede wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> On 6/1/21 6:04 PM, Hans de Goede wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> On 5/31/21 6:36 AM, Jarkko Sakkinen wrote:
> >>>>>>>>>>>>> Interestingly enough the first backtrace is also happening on a:
> >>>>>>>>>>>>> "Dell Inc. XPS 13 9310/0MRT12, BIOS 2.2.0 04/06/2021"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So it seems that at least with 5.12.6 (which has the last 2 fixes)
> >>>>>>>>>>>>> all reports are about the XPS 13 9310. I wonder if there is an
> >>>>>>>>>>>>> issue with the TPM interrupt line on the XPS 13 9310; I've asked the
> >>>>>>>>>>>>> reporters to try adding tpm_tis.interrupts=0 to their kernel commandline.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is helpful for sure that these all are happening on matching hardware.
> >>>>>>>>>>>
> >>>>>>>>>>> So our kernel-backtrace tracking info (ABRT) just recorded a third backtrace
> >>>>>>>>>>> with a kernel >= 5.12.6, again on the XPS 13 9310, so now we have 3 variants:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>>>>>>>>
> >>>>>>>>>>> 2. Backtrace starting with a call to tpm_dev_async_work:
> >>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>>>>>>>>>> (note this one is not easily reproducible)
> >>>>>>>>>>>
> >>>>>>>>>>> 3. Backtrace starting with a call to rng_dev_read:
> >>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>>>>>>>
> >>>>>>>>>>> 3. is the new one. All bugs linked above are public, all 3 backtraces
> >>>>>>>>>>> so far have only been reported on the XPS 13 9310 (with kernel >= 5.12.6)
> >>>>>>>>>>> and I've asked all the reporters to check if tpm_tis.interrupts=0 helps.
> >>>>>>>>>>
> >>>>>>>>>> Quick status update, I've got a response from a XPS 13 9310 user in:
> >>>>>>>>>>
> >>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >>>>>>>>>>
> >>>>>>>>>> Indicating that a. he can reproduce this with the latest >= 5.12.6 kernels;
> >>>>>>>>>> and b. it goes away when specifying tpm_tis.interrupts=0 as I expected
> >>>>>>>>>> (I expected this because all the bug-reports started when the interrupt
> >>>>>>>>>> code got fixed/re-enabled a while ago).
> >>>>>>>>>
> >>>>>>>>> One more status update.
> >>>>>>>>>
> >>>>>>>>> - A new 4th variant of the backtrace has been spotted, where the problem hits
> >>>>>>>>> when called from probe() -> tpm2_auto_startup -> tpm2_do_selftest, see:
> >>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1958381
> >>>>>>>>>
> >>>>>>>>> - So far all reports with kernel >= 5.12.6 have been on a Dell XPS 13 9310
> >>>>>>>>> models. But the new variant is happening on a Dell XPS 15 9500 and the
> >>>>>>>>> backtrace starting at ima_add_boot_aggregate is also being reported on
> >>>>>>>>> a Dell XPS 15 9500 (as well as on the XPS 13 9310).
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>> Hans
> >>>>>>>>
> >>>>>>>> OK, I'll have to query if I could borrow that laptop from someone. It's
> >>>>>>>> fairly common laptop, i.e. might be possible.
> >>>>>>>
> >>>>>>> In the mean time I've also got a report that this variant of the backtrace:
> >>>>>>>
> >>>>>>> 1. Backtrace starting with a call to ima_add_boot_aggregate
> >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1963712
> >>>>>>>
> >>>>>>> Is also still happening with recent 5.12.y kernels on
> >>>>>>> Dell Precision 7750 laptops. Both the Precision 7750 and the XPS 9500 use
> >>>>>>> 10th gen comet lake processors (i7-10750H), where as the XPS 9310 is using
> >>>>>>> an icelake processor. So the common denominator seems to be that they are
> >>>>>>> all 2020 Dell laptop models using the latest Intel CPUs.
> >>>>>>>
> >>>>>>> FYI the complete list of models on which some of the 4 backtrace variants
> >>>>>>> are still seen on recent 5.12.y kernels is now:
> >>>>>>>
> >>>>>>> Dell XPS 13 9310
> >>>>>>> Dell XPS 15 9500
> >>>>>>> Dell Precision 7750
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Hans
> >>>>>>
> >>>>>> Does "tpm_tis.interrupts=0" uniformly workaround the issue?
> >>>>>
> >>>>> I unfortunately have not gotten much replies to my request to test with
> >>>>> tpm_tis.interrupts=0, but for those people who have bothered to test
> >>>>> (2 reporters IIRC) using tpm_tis.interrupts=0 does avoid the issue.
> >>>>
> >>>> So we see this in dmesg as first anything from TPM:
> >>>>
> >>>> [    0.904572] tpm_tis STM0125:00: 2.0 TPM (device-id 0x0, rev-id 78)
> >>>>
> >>>> This means that one command is successfully processed by the TPM, i.e.
> >>>> tpm2_probe() in tpm_tis_core_init().
> >>>>
> >>>> My first *guess*  was that IRQ is given by ACPI, would need ACPI dump to
> >>>> confirm (e.g. sudo acpidump > acpi.dump). It cannot be so because otherwise
> >>>> this code path would be executed:
> >>>>
> >>>>         if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) {
> >>>>                 dev_err(&chip->dev, FW_BUG
> >>>>                         "TPM interrupt not working, polling instead\n");
> >>>>
> >>>>                 disable_interrupts(chip);
> >>>>         }
> >>>>
> >>>> TPM_CHIP_FLAG_IRQ is never set, so you should see this message in dmesg if
> >>>> a legit value is given to IRQ by ACPI.  We are probably planning re-enable
> >>>> IRQ code after these type of issues are fully resolved, but right now you
> >>>> should not end up having it enabled (see tpm_tis_send() function).
> >>>>
> >>>> To put this together "if (irq != -1) {" path in tpm_tis_core_init() is
> >>>> never executed. And early in the same function the interrupt hardware is
> >>>> *explicitly* disabled.
> >>>>
> >>>> For me this looks like a hardware bug right now: interrupts stay enabled
> >>>> for some reason.
> >>>>
> >>>> ACPI dump would be useful to verify some of the assumptions in this.
> >>>
> >>> Ok, I've added a comment to the Fedora bugs for the 4 different backtrace
> >>> variants asking for acpidumps for the Dell XPS 13 9310, Dell XPS 15 9500
> >>> and Dell Precision 7750 laptops.
> >>
> >> 2 XPS 9310 acpidumps have been attached to:
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1964974
> >>
> >> Note the reporter of the first bug mentions that he is no longer having
> >> this issue, but we are definitely still getting reports for kernel version
> >>> 5.12.6 (which has the last 2 fixes) from XPS 9310 users...
> >>
> >> Maybe there are different BIOS versions in play ? It might be interesting
> >> to compare the 2 acpidumps...
> > 
> > ❯ diff -u ../tmp/ssdt7.dsl ../tmp2/ssdt7.dsl
> > --- ../tmp/ssdt7.dsl	2021-07-09 21:32:06.473166420 +0300
> > +++ ../tmp2/ssdt7.dsl	2021-07-09 21:33:45.065934469 +0300
> > @@ -5,7 +5,7 @@
> >   * 
> >   * Disassembling to symbolic ASL+ operators
> >   *
> > - * Disassembly of ssdt7.dat, Fri Jul  9 21:32:06 2021
> > + * Disassembly of ssdt7.dat, Fri Jul  9 21:33:45 2021
> >   *
> >   * Original Table Header:
> >   *     Signature        "SSDT"
> > @@ -121,7 +121,7 @@
> >                      0xFED40000,         // Address Base
> >                      0x00005000,         // Address Length
> >                      )
> > -                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y58)
> > +                Interrupt (ResourceConsumer, Level, ActiveLow, Shared, ,, _Y55)
> >                  {
> >                      0x0000000C,
> >                  }
> > @@ -141,7 +141,7 @@
> >                  }
> >                  Else
> >                  {
> > -                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
> > +                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
> >                      LIRQ = IRQN /* \_SB_.TPM_.IRQN */
> >                      Return (RES0) /* \_SB_.TPM_.RES0 */
> >                  }
> > @@ -152,14 +152,14 @@
> >                  If ((IRQN != Zero))
> >                  {
> >                      CreateDWordField (Arg0, 0x11, IRQ0)
> > -                    CreateDWordField (RES0, \_SB.TPM._Y58._INT, LIRQ)  // _INT: Interrupts
> > +                    CreateDWordField (RES0, \_SB.TPM._Y55._INT, LIRQ)  // _INT: Interrupts
> >                      LIRQ = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
> >                      IRQN = IRQ0 /* \_SB_.TPM_._SRS.IRQ0 */
> >                      CreateBitField (Arg0, 0x79, ITRG)
> > -                    CreateBitField (RES0, \_SB.TPM._Y58._HE, LTRG)  // _HE_: High-Edge
> > +                    CreateBitField (RES0, \_SB.TPM._Y55._HE, LTRG)  // _HE_: High-Edge
> >                      LTRG = ITRG /* \_SB_.TPM_._SRS.ITRG */
> >                      CreateBitField (Arg0, 0x7A, ILVL)
> > -                    CreateBitField (RES0, \_SB.TPM._Y58._LL, LLVL)  // _LL_: Low Level
> > +                    CreateBitField (RES0, \_SB.TPM._Y55._LL, LLVL)  // _LL_: Low Level
> >                      LLVL = ILVL /* \_SB_.TPM_._SRS.ILVL */
> >                      If ((((TID0 & 0x0F) == Zero) || ((TID0 & 0x0F
> >                          ) == 0x0F)))
> > 
> > This delta from "acpidump for Dell XPS 9310 (with Qualcomm QCA6390)" to
> > "acpidump output from a Dell XPS 13 9310 that no longer has a problem"
> > in SSDT7. The bug I'm referring to is
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920510
> > 
> > Looks to me just using a different label.
> 
> Yes, although I have the feeling that does indicate that their are possibly
> other changes under the hood. The 0x00000000c interrupt referred to seems to
> be an interrupt directly on the APIC, which means that it is a GPIO in direct-irq
> mode with level vs edge interrupt mode selection (pullup/down settings is all
> directly done by the BIOS.
> 
> I'm aware that misconfiguring those settings (which Linux cannot see) was
> an issue with the TPM IRQ on some Lenovo models, maybe the same is going on
> here; and later BIOS versions contain a fix (and this somehow also has
> changed the label in the DSDT ?).
> 
> > What if we just set "interrupts=0" explicitly for STM0125 HID since the
> > workaround seems to work according to the report?
> 
> That sounds like a reasonable workaround.

I just came from two week leave.

I'll create a patch that adds such workaround when I'm done with
purging all the email

/Jarkko

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2021-07-27  2:51 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-11 13:09 Recent tpm_tis IRQ handling changes are causing kernel backtraces Hans de Goede
2021-03-16 15:34 ` Hans de Goede
2021-03-16 19:18   ` Jarkko Sakkinen
2021-05-08  9:07     ` Hans de Goede
2021-05-10 17:25       ` Jarkko Sakkinen
2021-05-11  8:37         ` Hans de Goede
2021-05-11 23:48           ` Jarkko Sakkinen
2021-05-26 19:03           ` Hans de Goede
2021-05-27 14:00             ` Jarkko Sakkinen
2021-05-27 15:27               ` Hans de Goede
     [not found]                 ` <20210531043616.u3v25qzkkrik5apq@kernel.org>
2021-05-31  8:24                   ` Hans de Goede
2021-06-01 18:02                     ` Jarkko Sakkinen
2021-06-01 16:04                   ` Hans de Goede
2021-06-01 18:03                     ` Jarkko Sakkinen
2021-06-14 13:33                     ` Hans de Goede
2021-06-15 13:01                       ` Jarkko Sakkinen
2021-06-15 13:59                         ` Hans de Goede
2021-06-23 13:37                           ` Jarkko Sakkinen
2021-06-21 12:04                       ` Hans de Goede
2021-06-23 13:40                         ` Jarkko Sakkinen
2021-06-23 13:54                           ` Hans de Goede
2021-06-29 18:04                             ` Recent tpm_tis IRQ handling changes are causing kernel backtraces] Jarkko Sakkinen
2021-06-29 19:14                               ` Hans de Goede
2021-06-29 22:05                                 ` Jarkko Sakkinen
2021-06-30 12:47                                   ` Hans de Goede
2021-06-30 13:36                                     ` Hans de Goede
2021-07-09 18:44                                       ` Jarkko Sakkinen
2021-07-17 16:10                                         ` Hans de Goede
2021-07-27  2:50                                           ` Jarkko Sakkinen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).