i8k_smm_func() takes enormous of time to execute

i8k_smm_func() takes enormous of time to execute

[Oleksandr Natalenko]

Hi, Jonathan, Mario et al.

I've noticed that querying Dell Vostro 3360 hwmon sensor freezes the system 
completely (like, really completely, even mouse cursor does not move, and 
sound playback stops) for approx. half of a second. Then, system recovers and 
runs as usual. Also, sensor readings are okay, for instance:

===
$ sensors dell_smm-virtual-0
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 5615 RPM
CPU:            +51.0°C  
Ambient:        +36.0°C  
Other:          +63.0°C  
SODIMM:         +41.0°C 
===

So, I've used trace-cmd to check what takes that much amount of time to 
execute, and got this:

===
# trace-cmd record -p function_graph -l i8k_smm -F sensors dell_smm-virtual-0
$ trace-cmd report
...
sensors-23694 [002] 89099.214369: funcgraph_entry:      # 503440.746 us |  
i8k_smm();
...
===

Clearly, 0.5 s delay.

Looking at i8k_smm(), it calls i8k_smm_func() on 0th CPU:

===
 232     ret = smp_call_on_cpu(0, i8k_smm_func, regs, true);
===

which, in turn, does some asm magic.

I know that SMM is kinda "black box", and kernel has little to do with it, but 
I think that under Windows, for instance, it would work without freezes. So, 
likely, querying SMM might be done differently.

I do not know how to approach this issue, thus asking for help/advice. Also, 
CCing Jonathan since the comment before asm magic says this:

===
 137  * Call the System Management Mode BIOS. Code provided by Jonathan 
Buzzard.
===

This was also reported in various places before, for instance, [1], but 
unfortunately, without any solution.

Thanks.

Regards,
  Oleksandr

[1] https://bugs.launchpad.net/ubuntu/+source/acpi/+bug/10490

Re: i8k_smm_func() takes enormous of time to execute

[Jean Delvare]

Hi Oleksandr,

On jeu., 2017-11-23 at 22:41 +0100, Oleksandr Natalenko wrote:
> I've noticed that querying Dell Vostro 3360 hwmon sensor freezes the system 
> completely (like, really completely, even mouse cursor does not move, and 
> sound playback stops) for approx. half of a second. Then, system recovers and 
> runs as usual. Also, sensor readings are okay, for instance:
> 
> ===
> $ sensors dell_smm-virtual-0
> dell_smm-virtual-0
> Adapter: Virtual device
> Processor Fan: 5615 RPM
> CPU:            +51.0°C  
> Ambient:        +36.0°C  
> Other:          +63.0°C  
> SODIMM:         +41.0°C 
> ===
> 
> So, I've used trace-cmd to check what takes that much amount of time to 
> execute, and got this:
> 
> ===
> # trace-cmd record -p function_graph -l i8k_smm -F sensors dell_smm-virtual-0
> $ trace-cmd report
> ...
> sensors-23694 [002] 89099.214369: funcgraph_entry:      # 503440.746 us |  
> i8k_smm();
> ...
> ===
> 
> Clearly, 0.5 s delay.
> 
> Looking at i8k_smm(), it calls i8k_smm_func() on 0th CPU:
> 
> ===
>  232     ret = smp_call_on_cpu(0, i8k_smm_func, regs, true);
> ===
> 
> which, in turn, does some asm magic.
> 
> I know that SMM is kinda "black box", and kernel has little to do with it, but 
> I think that under Windows, for instance, it would work without freezes. So, 
> likely, querying SMM might be done differently.
> 
> I do not know how to approach this issue, thus asking for help/advice. Also, 
> CCing Jonathan since the comment before asm magic says this:
> 
> ===
>  137  * Call the System Management Mode BIOS. Code provided by Jonathan 
> Buzzard.
> ===
> 
> This was also reported in various places before, for instance, [1], but 
> unfortunately, without any solution.

There are two patches waiting to be tested in
https://bugzilla.kernel.org/show_bug.cgi?id=195751

-- 
Jean Delvare
SUSE L3 Support

Re: i8k_smm_func() takes enormous of time to execute

[Pali Rohár]

On Thursday 23 November 2017 22:41:03 Oleksandr Natalenko wrote:
> Hi, Jonathan, Mario et al.
> 
> I've noticed that querying Dell Vostro 3360 hwmon sensor freezes the system 
> completely (like, really completely, even mouse cursor does not move, and 
> sound playback stops) for approx. half of a second. Then, system recovers and 
> runs as usual. Also, sensor readings are okay, for instance:
> 
> ===
> $ sensors dell_smm-virtual-0
> dell_smm-virtual-0
> Adapter: Virtual device
> Processor Fan: 5615 RPM
> CPU:            +51.0°C  
> Ambient:        +36.0°C  
> Other:          +63.0°C  
> SODIMM:         +41.0°C 
> ===
> 
> So, I've used trace-cmd to check what takes that much amount of time to 
> execute, and got this:
> 
> ===
> # trace-cmd record -p function_graph -l i8k_smm -F sensors dell_smm-virtual-0
> $ trace-cmd report
> ...
> sensors-23694 [002] 89099.214369: funcgraph_entry:      # 503440.746 us |  
> i8k_smm();
> ...
> ===
> 
> Clearly, 0.5 s delay.
> 
> Looking at i8k_smm(), it calls i8k_smm_func() on 0th CPU:
> 
> ===
>  232     ret = smp_call_on_cpu(0, i8k_smm_func, regs, true);
> ===
> 
> which, in turn, does some asm magic.

Hi! If you compile module with DEBUG, then i8k_smm_func function print
into dmes how much time it spend in SMM mode. And also it print which
SMM action it was.

Therefore you would be able to figure out which SMM call (which is just
RPC) cause this freeze.

Basically when CPU is in SMM mode, then execution of kernel is stopped
and kernel does not know that such thing happened. So it is up to the
SMM implementation to return control ASAP back to non-SMM mode.

Kernel has no control over SMM, it can be fixed only by vendor by new
BIOS or firmware update.

What we can do is just identify problematic calls and avoid their usage.

This problem is independent of kernel or whole operating system.

> I know that SMM is kinda "black box", and kernel has little to do with it, but 
> I think that under Windows, for instance, it would work without freezes. So, 
> likely, querying SMM might be done differently.
> 
> I do not know how to approach this issue, thus asking for help/advice. Also, 
> CCing Jonathan since the comment before asm magic says this:
> 
> ===
>  137  * Call the System Management Mode BIOS. Code provided by Jonathan 
> Buzzard.
> ===
> 
> This was also reported in various places before, for instance, [1], but 
> unfortunately, without any solution.
> 
> Thanks.
> 
> Regards,
>   Oleksandr
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/acpi/+bug/10490

-- 
Pali Rohár
pali.rohar@gmail.com

Re: i8k_smm_func() takes enormous of time to execute

[Oleksandr Natalenko]

Hi.

On pátek 24. listopadu 2017 11:22:28 CET Jean Delvare wrote:
> Hi Oleksandr,
> 
> On jeu., 2017-11-23 at 22:41 +0100, Oleksandr Natalenko wrote:
> > I've noticed that querying Dell Vostro 3360 hwmon sensor freezes the
> > system
> > completely (like, really completely, even mouse cursor does not move, and
> > sound playback stops) for approx. half of a second. Then, system recovers
> > and runs as usual. Also, sensor readings are okay, for instance:
> > 
> > ===
> > $ sensors dell_smm-virtual-0
> > dell_smm-virtual-0
> > Adapter: Virtual device
> > Processor Fan: 5615 RPM
> > CPU:            +51.0°C
> > Ambient:        +36.0°C
> > Other:          +63.0°C
> > SODIMM:         +41.0°C
> > ===
> > 
> > So, I've used trace-cmd to check what takes that much amount of time to
> > execute, and got this:
> > 
> > ===
> > # trace-cmd record -p function_graph -l i8k_smm -F sensors
> > dell_smm-virtual-0 $ trace-cmd report
> > ...
> > sensors-23694 [002] 89099.214369: funcgraph_entry:      # 503440.746 us |
> > i8k_smm();
> > ...
> > ===
> > 
> > Clearly, 0.5 s delay.
> > 
> > Looking at i8k_smm(), it calls i8k_smm_func() on 0th CPU:
> > 
> > ===
> > 
> >  232     ret = smp_call_on_cpu(0, i8k_smm_func, regs, true);
> > 
> > ===
> > 
> > which, in turn, does some asm magic.
> > 
> > I know that SMM is kinda "black box", and kernel has little to do with it,
> > but I think that under Windows, for instance, it would work without
> > freezes. So, likely, querying SMM might be done differently.
> > 
> > I do not know how to approach this issue, thus asking for help/advice.
> > Also, CCing Jonathan since the comment before asm magic says this:
> > 
> > ===
> > 
> >  137  * Call the System Management Mode BIOS. Code provided by Jonathan
> > 
> > Buzzard.
> > ===
> > 
> > This was also reported in various places before, for instance, [1], but
> > unfortunately, without any solution.
> 
> There are two patches waiting to be tested in
> https://bugzilla.kernel.org/show_bug.cgi?id=195751

Tested and attached a couple of patches on top of those to the BZ. If 
disabling fan control is the only approach here, I can confirm that this 
works.

Thanks.

Regards,
  Oleksandr

Re: i8k_smm_func() takes enormous of time to execute

[Pali Rohár]

On Friday 24 November 2017 12:17:30 Oleksandr Natalenko wrote:
> > There are two patches waiting to be tested in
> > https://bugzilla.kernel.org/show_bug.cgi?id=195751
> 
> Tested and attached a couple of patches on top of those to the BZ. If 
> disabling fan control is the only approach here, I can confirm that this 
> works.

Hi! Please figure out which SMM call (they are identified by eax and ebx
registers) cause this freeze. So we would now what needs to be
blacklisted.

-- 
Pali Rohár
pali.rohar@gmail.com

Re: i8k_smm_func() takes enormous of time to execute

[Oleksandr Natalenko]

On pátek 24. listopadu 2017 12:25:43 CET Pali Rohár wrote:
> On Friday 24 November 2017 12:17:30 Oleksandr Natalenko wrote:
> > > There are two patches waiting to be tested in
> > > https://bugzilla.kernel.org/show_bug.cgi?id=195751
> > 
> > Tested and attached a couple of patches on top of those to the BZ. If
> > disabling fan control is the only approach here, I can confirm that this
> > works.
> 
> Hi! Please figure out which SMM call (they are identified by eax and ebx
> registers) cause this freeze. So we would now what needs to be
> blacklisted.

Here it goes:

[ 7.191081] dell_smm_hwmon: smm(0x10a3 0x0000) = 0x002e (took     1837 usecs)
[ 7.194007] dell_smm_hwmon: smm(0x00a3 0x0001) = 0xffff (took      151 usecs)
[ 7.198239] dell_smm_hwmon: smm(0x00a3 0x0000) = 0x0002 (took     1411 usecs)
[ 7.199095] dell_smm_hwmon: smm(0x02a3 0x0001) = 0xffff (took       71 usecs)
[ 7.700493] dell_smm_hwmon: smm(0x02a3 0x0000) = 0x157c (took   488912 usecs)
[ 7.701277] dell_smm_hwmon: smm(0x0025 0X0000) = 0xffff (took       71 usecs)

0x02a3 is I8K_SMM_GET_SPEED.

Re: i8k_smm_func() takes enormous of time to execute

[Pali Rohár]

On Friday 24 November 2017 13:28:59 Oleksandr Natalenko wrote:
> On pátek 24. listopadu 2017 12:25:43 CET Pali Rohár wrote:
> > On Friday 24 November 2017 12:17:30 Oleksandr Natalenko wrote:
> > > > There are two patches waiting to be tested in
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=195751
> > > 
> > > Tested and attached a couple of patches on top of those to the BZ. If
> > > disabling fan control is the only approach here, I can confirm that this
> > > works.
> > 
> > Hi! Please figure out which SMM call (they are identified by eax and ebx
> > registers) cause this freeze. So we would now what needs to be
> > blacklisted.
> 
> Here it goes:
> 
> [ 7.191081] dell_smm_hwmon: smm(0x10a3 0x0000) = 0x002e (took     1837 usecs)
> [ 7.194007] dell_smm_hwmon: smm(0x00a3 0x0001) = 0xffff (took      151 usecs)
> [ 7.198239] dell_smm_hwmon: smm(0x00a3 0x0000) = 0x0002 (took     1411 usecs)
> [ 7.199095] dell_smm_hwmon: smm(0x02a3 0x0001) = 0xffff (took       71 usecs)
> [ 7.700493] dell_smm_hwmon: smm(0x02a3 0x0000) = 0x157c (took   488912 usecs)
> [ 7.701277] dell_smm_hwmon: smm(0x0025 0X0000) = 0xffff (took       71 usecs)
> 
> 0x02a3 is I8K_SMM_GET_SPEED.

Ok, so it is really a good idea to disable fan control completely on your machine.

Your last patch in bugzilla looks ok, you add my Reviewed-by: Pali Rohár <pali.rohar@gmail.com>

-- 
Pali Rohár
pali.rohar@gmail.com

Re: i8k_smm_func() takes enormous of time to execute

[Oleksandr Natalenko]

Hi.

04.12.2017 11:35, Pali Rohár wrote:
> On Friday 24 November 2017 13:28:59 Oleksandr Natalenko wrote:
>> On pátek 24. listopadu 2017 12:25:43 CET Pali Rohár wrote:
>> > On Friday 24 November 2017 12:17:30 Oleksandr Natalenko wrote:
>> > > > There are two patches waiting to be tested in
>> > > > https://bugzilla.kernel.org/show_bug.cgi?id=195751
>> > >
>> > > Tested and attached a couple of patches on top of those to the BZ. If
>> > > disabling fan control is the only approach here, I can confirm that this
>> > > works.
>> >
>> > Hi! Please figure out which SMM call (they are identified by eax and ebx
>> > registers) cause this freeze. So we would now what needs to be
>> > blacklisted.
>> 
>> Here it goes:
>> 
>> [ 7.191081] dell_smm_hwmon: smm(0x10a3 0x0000) = 0x002e (took     1837 
>> usecs)
>> [ 7.194007] dell_smm_hwmon: smm(0x00a3 0x0001) = 0xffff (took      151 
>> usecs)
>> [ 7.198239] dell_smm_hwmon: smm(0x00a3 0x0000) = 0x0002 (took     1411 
>> usecs)
>> [ 7.199095] dell_smm_hwmon: smm(0x02a3 0x0001) = 0xffff (took       71 
>> usecs)
>> [ 7.700493] dell_smm_hwmon: smm(0x02a3 0x0000) = 0x157c (took   488912 
>> usecs)
>> [ 7.701277] dell_smm_hwmon: smm(0x0025 0X0000) = 0xffff (took       71 
>> usecs)
>> 
>> 0x02a3 is I8K_SMM_GET_SPEED.
> 
> Ok, so it is really a good idea to disable fan control completely on
> your machine.

This still doesn't explain why things work in non-Linux environment… But 
as a workaround, okay, let it be.

> Your last patch in bugzilla looks ok, you add my Reviewed-by: Pali
> Rohár <pali.rohar@gmail.com>

Could you please advice on how to proceed further? I can submit all 3 
patches (incl. yours two), to a ML.

Thanks.

Oleksandr

Re: i8k_smm_func() takes enormous of time to execute

[Pali Rohár]

On Monday 04 December 2017 13:08:18 Oleksandr Natalenko wrote:
> Hi.
> 
> 04.12.2017 11:35, Pali Rohár wrote:
> > On Friday 24 November 2017 13:28:59 Oleksandr Natalenko wrote:
> > > On pátek 24. listopadu 2017 12:25:43 CET Pali Rohár wrote:
> > > > On Friday 24 November 2017 12:17:30 Oleksandr Natalenko wrote:
> > > > > > There are two patches waiting to be tested in
> > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=195751
> > > > >
> > > > > Tested and attached a couple of patches on top of those to the BZ. If
> > > > > disabling fan control is the only approach here, I can confirm that this
> > > > > works.
> > > >
> > > > Hi! Please figure out which SMM call (they are identified by eax and ebx
> > > > registers) cause this freeze. So we would now what needs to be
> > > > blacklisted.
> > > 
> > > Here it goes:
> > > 
> > > [ 7.191081] dell_smm_hwmon: smm(0x10a3 0x0000) = 0x002e (took
> > > 1837 usecs)
> > > [ 7.194007] dell_smm_hwmon: smm(0x00a3 0x0001) = 0xffff (took
> > > 151 usecs)
> > > [ 7.198239] dell_smm_hwmon: smm(0x00a3 0x0000) = 0x0002 (took
> > > 1411 usecs)
> > > [ 7.199095] dell_smm_hwmon: smm(0x02a3 0x0001) = 0xffff (took
> > > 71 usecs)
> > > [ 7.700493] dell_smm_hwmon: smm(0x02a3 0x0000) = 0x157c (took
> > > 488912 usecs)
> > > [ 7.701277] dell_smm_hwmon: smm(0x0025 0X0000) = 0xffff (took
> > > 71 usecs)
> > > 
> > > 0x02a3 is I8K_SMM_GET_SPEED.
> > 
> > Ok, so it is really a good idea to disable fan control completely on
> > your machine.
> 
> This still doesn't explain why things work in non-Linux environment…

No, they does not work in non-Linux environment too. You can call same
smm request also on Windows and it freeze computer in same way. Just
windows do not have such smm driver out of box. To reproduce this
problem you need to write either own kernel driver in assembler (like
driver for linux) or write that smm code in assembler for classic
(userspace) application and find out how to call that smm code via
WinAPI (probably you would need Administrator rights and study lot of
WinAPI documentation).

> But as a workaround, okay, let it be.

Basically once you issue this 0x02a3 call your computer freeze (more
precisely stay in SMM mode -- it is x86 processor mode) and such
operation is fully independent of what is running on your x86 cpu
(Linux, Windows, DOS, some RTOS system, ...)

> > Your last patch in bugzilla looks ok, you add my Reviewed-by: Pali
> > Rohár <pali.rohar@gmail.com>
> 
> Could you please advice on how to proceed further? I can submit all 3
> patches (incl. yours two), to a ML.

Now it is up to hwmon maintainers (Jean Delvare & Guenter Roeck) to pick
up these patches. I think nothing more is needed from your side. (And if
yes, Jean would write.)

-- 
Pali Rohár
pali.rohar@gmail.com

Re: i8k_smm_func() takes enormous of time to execute

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On Monday 04 December 2017 13:18:29 Pali Rohár wrote:
> > > Your last patch in bugzilla looks ok, you add my Reviewed-by: Pali
> > > Rohár <pali.rohar@gmail.com>
> > 
> > Could you please advice on how to proceed further? I can submit all 3
> > patches (incl. yours two), to a ML.
> 
> Now it is up to hwmon maintainers (Jean Delvare & Guenter Roeck) to pick
> up these patches. I think nothing more is needed from your side. (And if
> yes, Jean would write.)
> 

Jean Delvare & Guenter Roeck: PING

Can you process patches from that bugzilla ticket?
https://bugzilla.kernel.org/show_bug.cgi?id=195751

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: i8k_smm_func() takes enormous of time to execute

[Guenter Roeck]

On 01/27/2018 06:09 AM, Pali Rohár wrote:
> On Monday 04 December 2017 13:18:29 Pali Rohár wrote:
>>>> Your last patch in bugzilla looks ok, you add my Reviewed-by: Pali
>>>> Rohár <pali.rohar@gmail.com>
>>>
>>> Could you please advice on how to proceed further? I can submit all 3
>>> patches (incl. yours two), to a ML.
>>
>> Now it is up to hwmon maintainers (Jean Delvare & Guenter Roeck) to pick
>> up these patches. I think nothing more is needed from your side. (And if
>> yes, Jean would write.)
>>
> 
> Jean Delvare & Guenter Roeck: PING
> 
> Can you process patches from that bugzilla ticket?
> https://bugzilla.kernel.org/show_bug.cgi?id=195751
> 
No. You should know better. Please follow the established process for
submitting patches into the upstream kernel.

Guenter