[Xenomai] [PULL] ipipe 4.4 updates

[Xenomai] [PULL] ipipe 4.4 updates

[Jan Kiszka]

The following changes since commit 5618f90d33b17ab0ee9352705610a5e9f00f9964:

  Revert "ipipe: printk: defer dev_printk() output to safe context" (2017-12-15 18:10:01 +0100)

are available in the git repository at:

  git://git.xenomai.org/ipipe-jki for-upstream/4.4-update

for you to fetch changes up to c6de6878f74e4b079066eddc373f3e8c125fdaa7:

  Merge tag 'v4.4.120' into for-upstream/4.4-update (2018-03-08 18:49:28 +0100)


I-pipe specific commits:

Jan Kiszka (5):
      arm/ipipe: Avoid open-coded __ipipe_call_mayday
      x86/ipipe: Deny JUMP_LABEL over I-pipe
      Merge tag 'v4.4.109' into for-upstream/4.4-update
      x86/ipipe: Disable access_ok context under I-pipe
      Merge tag 'v4.4.120' into for-upstream/4.4-update

Philippe Gerum (1):
      ipipe: printk: enable dev_printk() from the head stage


4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
repo.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates)

[Jan Kiszka]

On 2018-03-09 08:51, Jan Kiszka wrote:
> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
> repo.

I started to patch further on this during my flight (wip/4.9 updated),
noticed that the 4.14-wip queue will need a little bit sysentry tweaking
as well (missing 64-bit syscall dispatching), and then had to find 4.9
in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.

If you should look into 4.9-x86 as you indicated, please check this.

Another catch along that road: maxcpus=1 with >1 CPUs available triggers
a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well.

Jan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180310/1e8dff24/attachment.sig>

Re: [Xenomai] [PULL] ipipe 4.4 updates

[Jan Kiszka]

On 2018-03-09 08:51, Jan Kiszka wrote:
> The following changes since commit 5618f90d33b17ab0ee9352705610a5e9f00f9964:
> 
>   Revert "ipipe: printk: defer dev_printk() output to safe context" (2017-12-15 18:10:01 +0100)
> 
> are available in the git repository at:
> 
>   git://git.xenomai.org/ipipe-jki for-upstream/4.4-update
> 
> for you to fetch changes up to c6de6878f74e4b079066eddc373f3e8c125fdaa7:
> 
>   Merge tag 'v4.4.120' into for-upstream/4.4-update (2018-03-08 18:49:28 +0100)
> 
> 
> I-pipe specific commits:
> 
> Jan Kiszka (5):
>       arm/ipipe: Avoid open-coded __ipipe_call_mayday
>       x86/ipipe: Deny JUMP_LABEL over I-pipe
>       Merge tag 'v4.4.109' into for-upstream/4.4-update
>       x86/ipipe: Disable access_ok context under I-pipe
>       Merge tag 'v4.4.120' into for-upstream/4.4-update
> 
> Philippe Gerum (1):
>       ipipe: printk: enable dev_printk() from the head stage
> 

I've force pushed a new version, now
94e9b41aa1e88636aff5e3e339a44d27fe93b3a6, because Simon spotted a
mistake in my entry_32.S merge:

diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index 62c356c5b7a6..d8fdbe2470cf 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1021,7 +1021,7 @@ error_code:
 	TRACE_IRQS_OFF
 #endif	
 	movl	%esp, %eax			# pt_regs pointer
-	CALL_NOSPEC *%edi
+	CALL_NOSPEC %edi
 #ifdef CONFIG_IPIPE
 	testl %eax,%eax
 	jnz restore_nocheck

The macros does the dereferencing.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [PULL] ipipe 4.4 updates

[Philippe Gerum]

On 03/09/2018 05:51 PM, Jan Kiszka wrote:
> The following changes since commit 5618f90d33b17ab0ee9352705610a5e9f00f9964:
> 
>   Revert "ipipe: printk: defer dev_printk() output to safe context" (2017-12-15 18:10:01 +0100)
> 
> are available in the git repository at:
> 
>   git://git.xenomai.org/ipipe-jki for-upstream/4.4-update
> 
> for you to fetch changes up to c6de6878f74e4b079066eddc373f3e8c125fdaa7:
> 
>   Merge tag 'v4.4.120' into for-upstream/4.4-update (2018-03-08 18:49:28 +0100)
> 
> 
> I-pipe specific commits:
> 
> Jan Kiszka (5):
>       arm/ipipe: Avoid open-coded __ipipe_call_mayday

This one is useless, although it does not hurt. As a matter of fact,
__ipipe_exit_irq is only called from contexts running with hard IRQs
disabled, otherwise the pipeline core would be just fundamentally
broken. Merged it anyway.

>       x86/ipipe: Deny JUMP_LABEL over I-pipe
>       Merge tag 'v4.4.109' into for-upstream/4.4-update
>       x86/ipipe: Disable access_ok context under I-pipe
>       Merge tag 'v4.4.120' into for-upstream/4.4-update
> 

Merged, thanks.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly.

Meaning they do not enter any high CSTATE although they should with
CPU_IDLE* properly set, or something breaks when doing so?

Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here.

Thanks,

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
> 
> If you should look into 4.9-x86 as you indicated, please check this.
> 
> Another catch along that road: maxcpus=1 with >1 CPUs available triggers
> a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well.
> 

In the same vein, CONFIG_MAXSMP w/ CONFIG_IPIPE breaks the kernel at
boot on x86.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-03-20 00:54, Philippe Gerum wrote:
> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>> repo.
>>
>> I started to patch further on this during my flight (wip/4.9 updated),
>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>> properly.
> 
> Meaning they do not enter any high CSTATE although they should with
> CPU_IDLE* properly set, or something breaks when doing so?

This was in KVM, no CSTATEs involved.

> 
> Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here.
> 

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
# CONFIG_INTEL_IDLE is not set

TIA,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 03/20/2018 04:42 AM, Jan Kiszka wrote:
> On 2018-03-20 00:54, Philippe Gerum wrote:
>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>> repo.
>>>
>>> I started to patch further on this during my flight (wip/4.9 updated),
>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>> properly.
>>
>> Meaning they do not enter any high CSTATE although they should with
>> CPU_IDLE* properly set, or something breaks when doing so?
> 
> This was in KVM, no CSTATEs involved.
> 
>>
>> Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here.
>>
> 
> #
> # CPU Idle
> #
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
> # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
> # CONFIG_INTEL_IDLE is not set
> 

Ok, #89146106e8 should restore the previous pipeline behavior from 4.4,
#595606457f in -stable should enable an (hopefully) smarter handling
with Xenomai having its say on whether the idle state should be entered.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
> 
> If you should look into 4.9-x86 as you indicated, please check this.
> 
> Another catch along that road: maxcpus=1 with >1 CPUs available triggers
> a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well.
> 

Can't reproduce this one (over kvm), 4.4 and 4.9 included, so this may
be depending on the Kconfig.

-- 
Philippe.

[Xenomai] 4.9 for x86 sd card , gpio interrupt guide to system no responce

[Шевченко Тарас Григорьевич]

Hi all,
I finished some experiments to figure out what is going on
so  again
I have udoo board x86 ultra

1) Use ubuntu with vanile kernel 4.10.0-28 or 4.13.0-37 or 4.9.51   no problem with sd card , all work perfectly
2) Use 4.9.51 with ipipe patch   -pipe-core -4.9.51-x86-4.patch  guide to system no responce
 when i insert sd-card or insert own rtdm module for gpio test  and gpio interrupt occured
3) I use xenomai from git branch  stable-3.0.y (download yesterday)
4) there is options from menuconfig that I enable/disable

* General setup
  --> Local version - append to kernel release: -xenomai-3.0.5
  --> Timers subsystem
      --> High Resolution Timer Support (Enable)
* Xenomai/cobalt
  --> Sizes and static limits
    --> Number of registry slots (512 --> 4096)
    --> Size of system heap (Kb) (512 --> 4096)
    --> Size of private heap (Kb) (64 --> 256)
    --> Size of shared heap (Kb) (64 --> 256)
    --> Maximum number of POSIX timers per process (128 --> 512)
  --> Drivers
    --> RTnet
        --> RTnet, TCP/IP socket interface (Enable)
            --> Drivers
                --> New intel(R) PRO/1000 PCIe (Enable)
                --> Realtek 8169 (Enable)
                --> Loopback (Enable)
        --> Add-Ons
            --> Real-Time Capturing Support (Enable)
* Power management and ACPI options
  --> CPU Frequency scaling
      --> CPU Frequency scaling (Disable)
  --> ACPI (Advanced Configuration and Power Interface) Support
      --> Processor (Disable)
  --> CPU Idle
      --> CPU idle PM support (Disable)
* Pocessor type and features
  --> Enable maximum number of SMP processors and NUMA nodes (Disable)
  --> Processor family
      --> Core 2/newer Xeon (if "cat /proc/cpuinfo | grep family" returns 6, set as Generic otherwise)
  --> Transparent Hugepage Support (Disable)
  --> Allow for memory compaction (Disable)
  --> Contiguous Memory Allocation (Disable)
  --> Allow for memory compaction
    --> Page Migration (Disable)
* Device Drivers
  --> Staging drivers
      --> Unisys SPAR driver support
         --> Unisys visorbus driver (Disable)

I consider, that something wrong with this options
Help please to start work with xenomai


С уважением и надеждой на сотрудничество,
Шевченко Т.Г.

----- Original Message -----
From: "Philippe Gerum" <rpm@xenomai.org>
To: "Jan Kiszka" <jan.kiszka@web.de>
Cc: "xenomai" <xenomai@xenomai.org>
Sent: Tuesday, March 20, 2018 5:05:07 PM
Subject: Re: [Xenomai] 4.9 for x86 issue

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
> 
> If you should look into 4.9-x86 as you indicated, please check this.
> 
> Another catch along that road: maxcpus=1 with >1 CPUs available triggers
> a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well.
> 

Can't reproduce this one (over kvm), 4.4 and 4.9 included, so this may
be depending on the Kconfig.

-- 
Philippe.

_______________________________________________
Xenomai mailing list
Xenomai@xenomai.org
https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
> 
> If you should look into 4.9-x86 as you indicated, please check this.

Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
far so good.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Шевченко Тарас Григорьевич]

Sorry, but no
I install 4.9.90 and new patch
Ive the same problems   - no responce with  my LKM  and SD card reader
I can add logs

with sd discover new details:
  if boot with sd-card inserted in reader   - sd-card can read , if I remove card  - get no responce 
 
difference with lsmod command output (when inserted card and not) is nls_iso8859_1  
 this module is absent without sd-card

I try boot without sd card and do modprobe nls_iso8859_1  after insert sdcard   -  get no responce 




----- Original Message -----
From: "Philippe Gerum" <rpm@xenomai.org>
To: "Jan Kiszka" <jan.kiszka@web.de>
Cc: "xenomai" <xenomai@xenomai.org>
Sent: Tuesday, March 27, 2018 4:12:32 PM
Subject: Re: [Xenomai] 4.9 for x86 issue

On 03/10/2018 11:06 PM, Jan Kiszka wrote:
> On 2018-03-09 08:51, Jan Kiszka wrote:
>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>> repo.
> 
> I started to patch further on this during my flight (wip/4.9 updated),
> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
> as well (missing 64-bit syscall dispatching), and then had to find 4.9
> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
> 
> If you should look into 4.9-x86 as you indicated, please check this.

Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
far so good.

-- 
Philippe.

_______________________________________________
Xenomai mailing list
Xenomai@xenomai.org
https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-03-27 15:12, Philippe Gerum wrote:
> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>> repo.
>>
>> I started to patch further on this during my flight (wip/4.9 updated),
>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>
>> If you should look into 4.9-x86 as you indicated, please check this.
> 
> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
> far so good.

Great, thanks!

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180327/ae5ee943/attachment.sig>

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-03-27 15:12, Philippe Gerum wrote:
> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>> repo.
>>
>> I started to patch further on this during my flight (wip/4.9 updated),
>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>
>> If you should look into 4.9-x86 as you indicated, please check this.
> 
> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
> far so good.

Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
CPU load. I also triggered this with stable-3.0.x:

[  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
[  237.460728] Modules linked in:
[  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
[  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
[  237.461490] I-pipe domain: Xenomai
[  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
[  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
[  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
[  237.461490] Call Trace:
[  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
[  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
[  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
[  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
[  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
[  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
[  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
[  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
[  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
[  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
[  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
[  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
[  237.461490] ---[ end trace 9d2476a38b0c5379 ]---

I will debug this tomorrow.

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180405/a998b4ce/attachment.sig>

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/05/2018 10:13 PM, Jan Kiszka wrote:
> On 2018-03-27 15:12, Philippe Gerum wrote:
>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>> repo.
>>>
>>> I started to patch further on this during my flight (wip/4.9 updated),
>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>
>>> If you should look into 4.9-x86 as you indicated, please check this.
>>
>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>> far so good.
> 
> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
> CPU load. I also triggered this with stable-3.0.x:
> 
> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
> [  237.460728] Modules linked in:
> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
> [  237.461490] I-pipe domain: Xenomai
> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
> [  237.461490] Call Trace:
> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
> 
> I will debug this tomorrow.
> 

I can't reproduce this, the loadavg on my qemu instance consistently
converges to 0.0x figures while running the latency test (10Khz or 1Khz,
same). I'm now running 4.9.92, but I don't think this should make any
difference, since I could trace the box entering the idle state on .90.

Are you running the ia32emu mode, or x86_64? Also, could you share your
.config for building the guest kernel?

Thanks,

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-06 08:54, Philippe Gerum wrote:
> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>> repo.
>>>>
>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>
>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>
>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>> far so good.
>>
>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>> CPU load. I also triggered this with stable-3.0.x:
>>
>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>> [  237.460728] Modules linked in:
>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>> [  237.461490] I-pipe domain: Xenomai
>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>> [  237.461490] Call Trace:
>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>
>> I will debug this tomorrow.
>>
> 
> I can't reproduce this, the loadavg on my qemu instance consistently
> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
> same). I'm now running 4.9.92, but I don't think this should make any
> difference, since I could trace the box entering the idle state on .90.
> 
> Are you running the ia32emu mode, or x86_64? Also, could you share your
> .config for building the guest kernel?

Config is the same I sent back then. Userland is 64-bit, compat support
enabled.

The reason I see so far: xnclock_core_local_shot never sets XNIDLE. I
suspect we always have a timer registered, that for the host clock. So
we can't become idle this way. I'm not even sure that this test makes
sense because a pending RT timer does not make a non-idle system.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/06/2018 03:38 PM, Jan Kiszka wrote:
> On 2018-04-06 08:54, Philippe Gerum wrote:
>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>> repo.
>>>>>
>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>
>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>
>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>> far so good.
>>>
>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>> CPU load. I also triggered this with stable-3.0.x:
>>>
>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>> [  237.460728] Modules linked in:
>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>> [  237.461490] I-pipe domain: Xenomai
>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>> [  237.461490] Call Trace:
>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>
>>> I will debug this tomorrow.
>>>
>>
>> I can't reproduce this, the loadavg on my qemu instance consistently
>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>> same). I'm now running 4.9.92, but I don't think this should make any
>> difference, since I could trace the box entering the idle state on .90.
>>
>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>> .config for building the guest kernel?
> 
> Config is the same I sent back then. Userland is 64-bit, compat support
> enabled.

You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
whole file now.

> 
> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.

It does here (I traced it). However this should depend on the NO_HZ
settings, mine are :

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ=y

> I
> suspect we always have a timer registered, that for the host clock. So

In that case, the timer is not idle Xenomai-wise.

> we can't become idle this way. I'm not even sure that this test makes
> sense because a pending RT timer does not make a non-idle system.
> 

This is not about testing for Cobalt idleness, but for its core timer
idleness, given that the core timer is shared between both kernels. We
want to know whether we may allow the regular kernel to shutdown the
clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
if you will. I covered this stuff in Documentation/ipipe.rst lately.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-06 16:11, Philippe Gerum wrote:
> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>> repo.
>>>>>>
>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>
>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>
>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>> far so good.
>>>>
>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>
>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>> [  237.460728] Modules linked in:
>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>> [  237.461490] I-pipe domain: Xenomai
>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>> [  237.461490] Call Trace:
>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>
>>>> I will debug this tomorrow.
>>>>
>>>
>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>> same). I'm now running 4.9.92, but I don't think this should make any
>>> difference, since I could trace the box entering the idle state on .90.
>>>
>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>> .config for building the guest kernel?
>>
>> Config is the same I sent back then. Userland is 64-bit, compat support
>> enabled.
> 
> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
> whole file now.

Sorry, though I did. Attached.

> 
>>
>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
> 
> It does here (I traced it). However this should depend on the NO_HZ
> settings, mine are :
> 
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> CONFIG_NO_HZ_IDLE=y
> CONFIG_NO_HZ=y
> 

Same here.

>> I
>> suspect we always have a timer registered, that for the host clock. So
> 
> In that case, the timer is not idle Xenomai-wise.
> 
>> we can't become idle this way. I'm not even sure that this test makes
>> sense because a pending RT timer does not make a non-idle system.
>>
> 
> This is not about testing for Cobalt idleness, but for its core timer
> idleness, given that the core timer is shared between both kernels. We
> want to know whether we may allow the regular kernel to shutdown the
> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
> if you will. I covered this stuff in Documentation/ipipe.rst lately.
> 

I still don't see the problem. We own the timer, Linux does not program
it. And letting Linux call hlt does not disturb the timer programming,
in most cases at least (there might be some weird old broken hardware).

Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.xz
Type: application/x-xz
Size: 21268 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180406/32e24aae/attachment.bin>

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-06 17:11, Jan Kiszka wrote:
>> This is not about testing for Cobalt idleness, but for its core timer
>> idleness, given that the core timer is shared between both kernels. We
>> want to know whether we may allow the regular kernel to shutdown the
>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>
> 
> I still don't see the problem. We own the timer, Linux does not program
> it. And letting Linux call hlt does not disturb the timer programming,
> in most cases at least (there might be some weird old broken hardware).

Looking at ipipe_enter_idle_hook more closely: What might make sense as
logic here is a test for a close-to-expire timer on the CPU. Then we
would do polling until the expiry which may reduce wakeup latencies. But
the question would be what the thresholds should be on the different
targets.

Did you see any concrete issues without that hook? On which architecture?

Jan

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-05 22:13, Jan Kiszka wrote:
> On 2018-03-27 15:12, Philippe Gerum wrote:
>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>> repo.
>>>
>>> I started to patch further on this during my flight (wip/4.9 updated),
>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>
>>> If you should look into 4.9-x86 as you indicated, please check this.
>>
>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>> far so good.
> 
> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
> CPU load. I also triggered this with stable-3.0.x:
> 
> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
> [  237.460728] Modules linked in:
> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
> [  237.461490] I-pipe domain: Xenomai
> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
> [  237.461490] Call Trace:
> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
> 

The WARN_ON is due to missing
http://git.xenomai.org/ipipe-jki.git/commit/?h=for-upstream/4.4-update&id=d95805a2f3448cf65e85897cf43f260b9ff3f9d0
in 4.9. I'm preparing a PR. I've also did a stable update for 4.9.

The idleness bug can be resolved by dropping ipipe_enter_idle_hook from
Xenomai until we have a better logic at hand. If someone wants the
current behavior, there is still idle=poll.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/06/2018 05:11 PM, Jan Kiszka wrote:
> On 2018-04-06 16:11, Philippe Gerum wrote:
>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>> repo.
>>>>>>>
>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>
>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>
>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>> far so good.
>>>>>
>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>
>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>> [  237.460728] Modules linked in:
>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>> [  237.461490] Call Trace:
>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>
>>>>> I will debug this tomorrow.
>>>>>
>>>>
>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>> difference, since I could trace the box entering the idle state on .90.
>>>>
>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>> .config for building the guest kernel?
>>>
>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>> enabled.
>>
>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>> whole file now.
> 
> Sorry, though I did. Attached.
> 

Thanks,

>>
>>>
>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>
>> It does here (I traced it). However this should depend on the NO_HZ
>> settings, mine are :
>>
>> CONFIG_TICK_ONESHOT=y
>> CONFIG_NO_HZ_COMMON=y
>> # CONFIG_HZ_PERIODIC is not set
>> CONFIG_NO_HZ_IDLE=y
>> CONFIG_NO_HZ=y
>>
> 
> Same here.
> 
>>> I
>>> suspect we always have a timer registered, that for the host clock. So
>>
>> In that case, the timer is not idle Xenomai-wise.
>>
>>> we can't become idle this way. I'm not even sure that this test makes
>>> sense because a pending RT timer does not make a non-idle system.
>>>
>>
>> This is not about testing for Cobalt idleness, but for its core timer
>> idleness, given that the core timer is shared between both kernels. We
>> want to know whether we may allow the regular kernel to shutdown the
>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>
> 
> I still don't see the problem. We own the timer, Linux does not program
> it. And letting Linux call hlt does not disturb the timer programming,
> in most cases at least (there might be some weird old broken hardware).


The problem is not with hlt, but with the tick device switch when c3stop
is enabled on the device, and going idle means shutting it down before
switching to a broadcast device. Very unfortunately, this is not even an
x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-07 09:25, Philippe Gerum wrote:
> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>> repo.
>>>>>>>>
>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>
>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>
>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>> far so good.
>>>>>>
>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>
>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>> [  237.460728] Modules linked in:
>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>> [  237.461490] Call Trace:
>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>
>>>>>> I will debug this tomorrow.
>>>>>>
>>>>>
>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>
>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>> .config for building the guest kernel?
>>>>
>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>> enabled.
>>>
>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>> whole file now.
>>
>> Sorry, though I did. Attached.
>>
> 
> Thanks,
> 
>>>
>>>>
>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>
>>> It does here (I traced it). However this should depend on the NO_HZ
>>> settings, mine are :
>>>
>>> CONFIG_TICK_ONESHOT=y
>>> CONFIG_NO_HZ_COMMON=y
>>> # CONFIG_HZ_PERIODIC is not set
>>> CONFIG_NO_HZ_IDLE=y
>>> CONFIG_NO_HZ=y
>>>
>>
>> Same here.
>>
>>>> I
>>>> suspect we always have a timer registered, that for the host clock. So
>>>
>>> In that case, the timer is not idle Xenomai-wise.
>>>
>>>> we can't become idle this way. I'm not even sure that this test makes
>>>> sense because a pending RT timer does not make a non-idle system.
>>>>
>>>
>>> This is not about testing for Cobalt idleness, but for its core timer
>>> idleness, given that the core timer is shared between both kernels. We
>>> want to know whether we may allow the regular kernel to shutdown the
>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>
>>
>> I still don't see the problem. We own the timer, Linux does not program
>> it. And letting Linux call hlt does not disturb the timer programming,
>> in most cases at least (there might be some weird old broken hardware).
> 
> 
> The problem is not with hlt, but with the tick device switch when c3stop
> is enabled on the device, and going idle means shutting it down before
> switching to a broadcast device. Very unfortunately, this is not even an
> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
> 

So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
those clockevent devices that we take over for Xenomai purposes, right?
That excludes the vast majority of systems Xenomai runs on. It
specifically excludes any modern x86 systems which now have ARAT
support. For the remaining ones, we always recommended to have
ACPI_PROCESSOR disabled, thus have no need for this new workaround either.

Regarding the mentioned ARM system: Is there no equivalent to
!ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
completely? We definitely need a much more targeted solution here. Any
suggestions?

Jan

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/07/2018 11:42 AM, Jan Kiszka wrote:
> On 2018-04-07 09:25, Philippe Gerum wrote:
>> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>>> repo.
>>>>>>>>>
>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>>
>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>>
>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>>> far so good.
>>>>>>>
>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>>
>>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>>> [  237.460728] Modules linked in:
>>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>>> [  237.461490] Call Trace:
>>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>>
>>>>>>> I will debug this tomorrow.
>>>>>>>
>>>>>>
>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>>
>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>>> .config for building the guest kernel?
>>>>>
>>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>>> enabled.
>>>>
>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>>> whole file now.
>>>
>>> Sorry, though I did. Attached.
>>>
>>
>> Thanks,
>>
>>>>
>>>>>
>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>>
>>>> It does here (I traced it). However this should depend on the NO_HZ
>>>> settings, mine are :
>>>>
>>>> CONFIG_TICK_ONESHOT=y
>>>> CONFIG_NO_HZ_COMMON=y
>>>> # CONFIG_HZ_PERIODIC is not set
>>>> CONFIG_NO_HZ_IDLE=y
>>>> CONFIG_NO_HZ=y
>>>>
>>>
>>> Same here.
>>>
>>>>> I
>>>>> suspect we always have a timer registered, that for the host clock. So
>>>>
>>>> In that case, the timer is not idle Xenomai-wise.
>>>>
>>>>> we can't become idle this way. I'm not even sure that this test makes
>>>>> sense because a pending RT timer does not make a non-idle system.
>>>>>
>>>>
>>>> This is not about testing for Cobalt idleness, but for its core timer
>>>> idleness, given that the core timer is shared between both kernels. We
>>>> want to know whether we may allow the regular kernel to shutdown the
>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>>
>>>
>>> I still don't see the problem. We own the timer, Linux does not program
>>> it. And letting Linux call hlt does not disturb the timer programming,
>>> in most cases at least (there might be some weird old broken hardware).
>>
>>
>> The problem is not with hlt, but with the tick device switch when c3stop
>> is enabled on the device, and going idle means shutting it down before
>> switching to a broadcast device. Very unfortunately, this is not even an
>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
>>
> 
> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
> those clockevent devices that we take over for Xenomai purposes, right?
> That excludes the vast majority of systems Xenomai runs on. It
> specifically excludes any modern x86 systems which now have ARAT
> support. For the remaining ones, we always recommended to have
> ACPI_PROCESSOR disabled, thus have no need for this new workaround either.
> 

This is not a work around. The fact that linux is not aware that a
second kernel may consider the timer as busy is a general issue, and
asking that second kernel to have its say is not a work around, but a
requirement.

> Regarding the mentioned ARM system: Is there no equivalent to
> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
> completely? We definitely need a much more targeted solution here. Any
> suggestions?
> 

Yes, try to find what does not work in your case. I'll try to reproduce
tomorrow using your Kconfig.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-07 18:55, Philippe Gerum wrote:
> On 04/07/2018 11:42 AM, Jan Kiszka wrote:
>> On 2018-04-07 09:25, Philippe Gerum wrote:
>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>>>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>>>> repo.
>>>>>>>>>>
>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>>>
>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>>>
>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>>>> far so good.
>>>>>>>>
>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>>>
>>>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>>>> [  237.460728] Modules linked in:
>>>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>>>> [  237.461490] Call Trace:
>>>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>>>
>>>>>>>> I will debug this tomorrow.
>>>>>>>>
>>>>>>>
>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>>>
>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>>>> .config for building the guest kernel?
>>>>>>
>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>>>> enabled.
>>>>>
>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>>>> whole file now.
>>>>
>>>> Sorry, though I did. Attached.
>>>>
>>>
>>> Thanks,
>>>
>>>>>
>>>>>>
>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>>>
>>>>> It does here (I traced it). However this should depend on the NO_HZ
>>>>> settings, mine are :
>>>>>
>>>>> CONFIG_TICK_ONESHOT=y
>>>>> CONFIG_NO_HZ_COMMON=y
>>>>> # CONFIG_HZ_PERIODIC is not set
>>>>> CONFIG_NO_HZ_IDLE=y
>>>>> CONFIG_NO_HZ=y
>>>>>
>>>>
>>>> Same here.
>>>>
>>>>>> I
>>>>>> suspect we always have a timer registered, that for the host clock. So
>>>>>
>>>>> In that case, the timer is not idle Xenomai-wise.
>>>>>
>>>>>> we can't become idle this way. I'm not even sure that this test makes
>>>>>> sense because a pending RT timer does not make a non-idle system.
>>>>>>
>>>>>
>>>>> This is not about testing for Cobalt idleness, but for its core timer
>>>>> idleness, given that the core timer is shared between both kernels. We
>>>>> want to know whether we may allow the regular kernel to shutdown the
>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>>>
>>>>
>>>> I still don't see the problem. We own the timer, Linux does not program
>>>> it. And letting Linux call hlt does not disturb the timer programming,
>>>> in most cases at least (there might be some weird old broken hardware).
>>>
>>>
>>> The problem is not with hlt, but with the tick device switch when c3stop
>>> is enabled on the device, and going idle means shutting it down before
>>> switching to a broadcast device. Very unfortunately, this is not even an
>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
>>>
>>
>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
>> those clockevent devices that we take over for Xenomai purposes, right?
>> That excludes the vast majority of systems Xenomai runs on. It
>> specifically excludes any modern x86 systems which now have ARAT
>> support. For the remaining ones, we always recommended to have
>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either.
>>
> 
> This is not a work around. The fact that linux is not aware that a
> second kernel may consider the timer as busy is a general issue, and
> asking that second kernel to have its say is not a work around, but a
> requirement.
> 
>> Regarding the mentioned ARM system: Is there no equivalent to
>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
>> completely? We definitely need a much more targeted solution here. Any
>> suggestions?
>>
> 
> Yes, try to find what does not work in your case. I'll try to reproduce
> tomorrow using your Kconfig.

I think you are still on the wrong track here: x86 is not at all
affected by the issue you see. Please focus on the ARM system where you
saw the issue in the first place.

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/b4daca80/attachment.sig>

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/07/2018 06:58 PM, Jan Kiszka wrote:
> On 2018-04-07 18:55, Philippe Gerum wrote:
>> On 04/07/2018 11:42 AM, Jan Kiszka wrote:
>>> On 2018-04-07 09:25, Philippe Gerum wrote:
>>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>>>>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>>>>> repo.
>>>>>>>>>>>
>>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>>>>
>>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>>>>
>>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>>>>> far so good.
>>>>>>>>>
>>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>>>>
>>>>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>>>>> [  237.460728] Modules linked in:
>>>>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>>>>> [  237.461490] Call Trace:
>>>>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>>>>
>>>>>>>>> I will debug this tomorrow.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>>>>
>>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>>>>> .config for building the guest kernel?
>>>>>>>
>>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>>>>> enabled.
>>>>>>
>>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>>>>> whole file now.
>>>>>
>>>>> Sorry, though I did. Attached.
>>>>>
>>>>
>>>> Thanks,
>>>>
>>>>>>
>>>>>>>
>>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>>>>
>>>>>> It does here (I traced it). However this should depend on the NO_HZ
>>>>>> settings, mine are :
>>>>>>
>>>>>> CONFIG_TICK_ONESHOT=y
>>>>>> CONFIG_NO_HZ_COMMON=y
>>>>>> # CONFIG_HZ_PERIODIC is not set
>>>>>> CONFIG_NO_HZ_IDLE=y
>>>>>> CONFIG_NO_HZ=y
>>>>>>
>>>>>
>>>>> Same here.
>>>>>
>>>>>>> I
>>>>>>> suspect we always have a timer registered, that for the host clock. So
>>>>>>
>>>>>> In that case, the timer is not idle Xenomai-wise.
>>>>>>
>>>>>>> we can't become idle this way. I'm not even sure that this test makes
>>>>>>> sense because a pending RT timer does not make a non-idle system.
>>>>>>>
>>>>>>
>>>>>> This is not about testing for Cobalt idleness, but for its core timer
>>>>>> idleness, given that the core timer is shared between both kernels. We
>>>>>> want to know whether we may allow the regular kernel to shutdown the
>>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>>>>
>>>>>
>>>>> I still don't see the problem. We own the timer, Linux does not program
>>>>> it. And letting Linux call hlt does not disturb the timer programming,
>>>>> in most cases at least (there might be some weird old broken hardware).
>>>>
>>>>
>>>> The problem is not with hlt, but with the tick device switch when c3stop
>>>> is enabled on the device, and going idle means shutting it down before
>>>> switching to a broadcast device. Very unfortunately, this is not even an
>>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
>>>>
>>>
>>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
>>> those clockevent devices that we take over for Xenomai purposes, right?
>>> That excludes the vast majority of systems Xenomai runs on. It
>>> specifically excludes any modern x86 systems which now have ARAT
>>> support. For the remaining ones, we always recommended to have
>>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either.
>>>
>>
>> This is not a work around. The fact that linux is not aware that a
>> second kernel may consider the timer as busy is a general issue, and
>> asking that second kernel to have its say is not a work around, but a
>> requirement.
>>
>>> Regarding the mentioned ARM system: Is there no equivalent to
>>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
>>> completely? We definitely need a much more targeted solution here. Any
>>> suggestions?
>>>
>>
>> Yes, try to find what does not work in your case. I'll try to reproduce
>> tomorrow using your Kconfig.
> 
> I think you are still on the wrong track here: x86 is not at all
> affected by the issue you see.

You misread me: I'm talking about finding why your config never matches
the idle state Xenomai-wise.

 Please focus on the ARM system where you
> saw the issue in the first place.
> 

This is a problem which needs a resolution starting from the generic
code. So let's focus on finding why the x86-specific case does not work
instead.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-07 19:04, Philippe Gerum wrote:
> On 04/07/2018 06:58 PM, Jan Kiszka wrote:
>> On 2018-04-07 18:55, Philippe Gerum wrote:
>>> On 04/07/2018 11:42 AM, Jan Kiszka wrote:
>>>> On 2018-04-07 09:25, Philippe Gerum wrote:
>>>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>>>>>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>>>>>> repo.
>>>>>>>>>>>>
>>>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>>>>>
>>>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>>>>>
>>>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>>>>>> far so good.
>>>>>>>>>>
>>>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>>>>>
>>>>>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>>>>>> [  237.460728] Modules linked in:
>>>>>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>>>>>> [  237.461490] Call Trace:
>>>>>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>>>>>
>>>>>>>>>> I will debug this tomorrow.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>>>>>
>>>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>>>>>> .config for building the guest kernel?
>>>>>>>>
>>>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>>>>>> enabled.
>>>>>>>
>>>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>>>>>> whole file now.
>>>>>>
>>>>>> Sorry, though I did. Attached.
>>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>>>>>
>>>>>>> It does here (I traced it). However this should depend on the NO_HZ
>>>>>>> settings, mine are :
>>>>>>>
>>>>>>> CONFIG_TICK_ONESHOT=y
>>>>>>> CONFIG_NO_HZ_COMMON=y
>>>>>>> # CONFIG_HZ_PERIODIC is not set
>>>>>>> CONFIG_NO_HZ_IDLE=y
>>>>>>> CONFIG_NO_HZ=y
>>>>>>>
>>>>>>
>>>>>> Same here.
>>>>>>
>>>>>>>> I
>>>>>>>> suspect we always have a timer registered, that for the host clock. So
>>>>>>>
>>>>>>> In that case, the timer is not idle Xenomai-wise.
>>>>>>>
>>>>>>>> we can't become idle this way. I'm not even sure that this test makes
>>>>>>>> sense because a pending RT timer does not make a non-idle system.
>>>>>>>>
>>>>>>>
>>>>>>> This is not about testing for Cobalt idleness, but for its core timer
>>>>>>> idleness, given that the core timer is shared between both kernels. We
>>>>>>> want to know whether we may allow the regular kernel to shutdown the
>>>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>>>>>
>>>>>>
>>>>>> I still don't see the problem. We own the timer, Linux does not program
>>>>>> it. And letting Linux call hlt does not disturb the timer programming,
>>>>>> in most cases at least (there might be some weird old broken hardware).
>>>>>
>>>>>
>>>>> The problem is not with hlt, but with the tick device switch when c3stop
>>>>> is enabled on the device, and going idle means shutting it down before
>>>>> switching to a broadcast device. Very unfortunately, this is not even an
>>>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
>>>>>
>>>>
>>>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
>>>> those clockevent devices that we take over for Xenomai purposes, right?
>>>> That excludes the vast majority of systems Xenomai runs on. It
>>>> specifically excludes any modern x86 systems which now have ARAT
>>>> support. For the remaining ones, we always recommended to have
>>>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either.
>>>>
>>>
>>> This is not a work around. The fact that linux is not aware that a
>>> second kernel may consider the timer as busy is a general issue, and
>>> asking that second kernel to have its say is not a work around, but a
>>> requirement.
>>>
>>>> Regarding the mentioned ARM system: Is there no equivalent to
>>>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
>>>> completely? We definitely need a much more targeted solution here. Any
>>>> suggestions?
>>>>
>>>
>>> Yes, try to find what does not work in your case. I'll try to reproduce
>>> tomorrow using your Kconfig.
>>
>> I think you are still on the wrong track here: x86 is not at all
>> affected by the issue you see.
> 
> You misread me: I'm talking about finding why your config never matches
> the idle state Xenomai-wise.
> 
>  Please focus on the ARM system where you
>> saw the issue in the first place.
>>
> 
> This is a problem which needs a resolution starting from the generic
> code. So let's focus on finding why the x86-specific case does not work
> instead.
...because your approach is not correct at all.

The goal must be to prevent to enter that sleep state, which we achieve
on x86 differently - or do not have to worry about anymore. But you hook
into Linux entering idle state in general and prevent that.

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/73a32e24/attachment.sig>

Re: [Xenomai] 4.9 for x86 issue

[Philippe Gerum]

On 04/07/2018 07:10 PM, Jan Kiszka wrote:
> On 2018-04-07 19:04, Philippe Gerum wrote:
>> On 04/07/2018 06:58 PM, Jan Kiszka wrote:
>>> On 2018-04-07 18:55, Philippe Gerum wrote:
>>>> On 04/07/2018 11:42 AM, Jan Kiszka wrote:
>>>>> On 2018-04-07 09:25, Philippe Gerum wrote:
>>>>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote:
>>>>>>> On 2018-04-06 16:11, Philippe Gerum wrote:
>>>>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote:
>>>>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote:
>>>>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote:
>>>>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
>>>>>>>>>>>>>> repo.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated),
>>>>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking
>>>>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9
>>>>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling
>>>>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this.
>>>>>>>>>>>>
>>>>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run
>>>>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So
>>>>>>>>>>>> far so good.
>>>>>>>>>>>
>>>>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual)
>>>>>>>>>>> CPU load. I also triggered this with stable-3.0.x:
>>>>>>>>>>>
>>>>>>>>>>> [  237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350
>>>>>>>>>>> [  237.460728] Modules linked in:
>>>>>>>>>>> [  237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11
>>>>>>>>>>> [  237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
>>>>>>>>>>> [  237.461490] I-pipe domain: Xenomai
>>>>>>>>>>> [  237.461490]  ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000
>>>>>>>>>>> [  237.461490]  ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008
>>>>>>>>>>> [  237.461490]  0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200
>>>>>>>>>>> [  237.461490] Call Trace:
>>>>>>>>>>> [  237.461490]  [<ffffffff8145e395>] dump_stack+0xb2/0xdd
>>>>>>>>>>> [  237.461490]  [<ffffffff810e7261>] __warn+0xd1/0xf0
>>>>>>>>>>> [  237.461490]  [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20
>>>>>>>>>>> [  237.461490]  [<ffffffff812423b6>] timerfd_read+0x2a6/0x350
>>>>>>>>>>> [  237.461490]  [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0
>>>>>>>>>>> [  237.461490]  [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20
>>>>>>>>>>> [  237.461490]  [<ffffffff8122026e>] CoBaLt_read+0xe/0x10
>>>>>>>>>>> [  237.461490]  [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0
>>>>>>>>>>> [  237.461490]  [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20
>>>>>>>>>>> [  237.461490]  [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110
>>>>>>>>>>> [  237.461490]  [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0
>>>>>>>>>>> [  237.461490]  [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
>>>>>>>>>>> [  237.461490] ---[ end trace 9d2476a38b0c5379 ]---
>>>>>>>>>>>
>>>>>>>>>>> I will debug this tomorrow.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently
>>>>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz,
>>>>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any
>>>>>>>>>> difference, since I could trace the box entering the idle state on .90.
>>>>>>>>>>
>>>>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your
>>>>>>>>>> .config for building the guest kernel?
>>>>>>>>>
>>>>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support
>>>>>>>>> enabled.
>>>>>>>>
>>>>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the
>>>>>>>> whole file now.
>>>>>>>
>>>>>>> Sorry, though I did. Attached.
>>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE.
>>>>>>>>
>>>>>>>> It does here (I traced it). However this should depend on the NO_HZ
>>>>>>>> settings, mine are :
>>>>>>>>
>>>>>>>> CONFIG_TICK_ONESHOT=y
>>>>>>>> CONFIG_NO_HZ_COMMON=y
>>>>>>>> # CONFIG_HZ_PERIODIC is not set
>>>>>>>> CONFIG_NO_HZ_IDLE=y
>>>>>>>> CONFIG_NO_HZ=y
>>>>>>>>
>>>>>>>
>>>>>>> Same here.
>>>>>>>
>>>>>>>>> I
>>>>>>>>> suspect we always have a timer registered, that for the host clock. So
>>>>>>>>
>>>>>>>> In that case, the timer is not idle Xenomai-wise.
>>>>>>>>
>>>>>>>>> we can't become idle this way. I'm not even sure that this test makes
>>>>>>>>> sense because a pending RT timer does not make a non-idle system.
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is not about testing for Cobalt idleness, but for its core timer
>>>>>>>> idleness, given that the core timer is shared between both kernels. We
>>>>>>>> want to know whether we may allow the regular kernel to shutdown the
>>>>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE
>>>>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately.
>>>>>>>>
>>>>>>>
>>>>>>> I still don't see the problem. We own the timer, Linux does not program
>>>>>>> it. And letting Linux call hlt does not disturb the timer programming,
>>>>>>> in most cases at least (there might be some weird old broken hardware).
>>>>>>
>>>>>>
>>>>>> The problem is not with hlt, but with the tick device switch when c3stop
>>>>>> is enabled on the device, and going idle means shutting it down before
>>>>>> switching to a broadcast device. Very unfortunately, this is not even an
>>>>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD.
>>>>>>
>>>>>
>>>>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on
>>>>> those clockevent devices that we take over for Xenomai purposes, right?
>>>>> That excludes the vast majority of systems Xenomai runs on. It
>>>>> specifically excludes any modern x86 systems which now have ARAT
>>>>> support. For the remaining ones, we always recommended to have
>>>>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either.
>>>>>
>>>>
>>>> This is not a work around. The fact that linux is not aware that a
>>>> second kernel may consider the timer as busy is a general issue, and
>>>> asking that second kernel to have its say is not a work around, but a
>>>> requirement.
>>>>
>>>>> Regarding the mentioned ARM system: Is there no equivalent to
>>>>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi
>>>>> completely? We definitely need a much more targeted solution here. Any
>>>>> suggestions?
>>>>>
>>>>
>>>> Yes, try to find what does not work in your case. I'll try to reproduce
>>>> tomorrow using your Kconfig.
>>>
>>> I think you are still on the wrong track here: x86 is not at all
>>> affected by the issue you see.
>>
>> You misread me: I'm talking about finding why your config never matches
>> the idle state Xenomai-wise.
>>
>>  Please focus on the ARM system where you
>>> saw the issue in the first place.
>>>
>>
>> This is a problem which needs a resolution starting from the generic
>> code. So let's focus on finding why the x86-specific case does not work
>> instead.
> ...because your approach is not correct at all.
> 
> The goal must be to prevent to enter that sleep state, which we achieve
> on x86 differently - or do not have to worry about anymore. But you hook
> into Linux entering idle state in general and prevent that.
> 

Ok, so here is the situation as of now: you say that this approach is
stupid, I'm saying that I traced it working on x86 and ARM. I'm also
saying that the general issue of linux defining alone what the idle
state might be has to be addressed.

So please provide an explanation to the bug you see on your end, I'll
try to figure out why I can't see that here, and make an informed
opinion about whether the current approach is wrong or too limited in
logic, which is different.

Dropping code as a knee jerk reaction simply because it does not work
for your case may not be the best option.

-- 
Philippe.

Re: [Xenomai] 4.9 for x86 issue

[Jan Kiszka]

On 2018-04-07 19:20, Philippe Gerum wrote:
> Ok, so here is the situation as of now: you say that this approach is
> stupid, I'm saying that I traced it working on x86 and ARM. I'm also
> saying that the general issue of linux defining alone what the idle
> state might be has to be addressed.
> 
> So please provide an explanation to the bug you see on your end, I'll
> try to figure out why I can't see that here, and make an informed
> opinion about whether the current approach is wrong or too limited in
> logic, which is different.
> 
> Dropping code as a knee jerk reaction simply because it does not work
> for your case may not be the best option.

Look, there are a couple of things fishy with this approach:
- there is no documentation of a positive impact on x86 (did you measure
  high latency without this change? On which hardware? With which
  .config)
- there is no reasoning why we suddenly need this since 4.9 (on x86)
- it does not address the issue (some timer stopping in deep sleep
  state) at its root (Linux telling hardware to enter that *specific*
  sleep state)

Again, I cannot talk about ARM as I didn't look on that end yet, but I
can tell you for x86 that - if this is really just about C3 vs. the APIC
timer - you are hunting a ghost, a non-existing problem. At the same
time, the approach caused a regression on x86 that one may only see with
proper instrumentation in place (like running in a VM or having a power
meter attached to the hardware) because it always prevents idleness there.

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/f3df57c7/attachment.sig>