* Re: [Xenomai] 4.9 for x86 issue [not found] ` <1320247074.8207767.1522165874048.JavaMail.zimbra@triolcorp.com.ua> @ 2018-03-28 14:40 ` Шевченко Тарас Григорьевич 0 siblings, 0 replies; 24+ messages in thread From: Шевченко Тарас Григорьевич @ 2018-03-28 14:40 UTC (permalink / raw) To: Greg Gallagher; +Cc: xenomai Hi! I found in boot log failed to start lsb apparmor initialization could it be the problem ? С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> To: "Greg Gallagher" <greg@embeddedgreg.com> Sent: Tuesday, March 27, 2018 6:51:14 PM Subject: Re: [Xenomai] 4.9 for x86 issue Maybe you need some additional output ? С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> To: "Greg Gallagher" <greg@embeddedgreg.com> Sent: Tuesday, March 27, 2018 6:50:32 PM Subject: Re: [Xenomai] 4.9 for x86 issue yeti@yeti-UDOO-x86:~$ uname -a Linux yeti-UDOO-x86 4.9.90 #1 SMP Tue Mar 27 12:22:27 EEST 2018 x86_64 x86_64 x86_64 GNU/Linux yeti@yeti-UDOO-x86:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial yeti@yeti-UDOO-x86:~$ lsmod |grep sd - nothing yeti@yeti-UDOO-x86:~$ I enable mmc debug option at kernel - get many messages like below dmesg output ipipe patch apply, xenomai not installed , without sdcard inserted [sdhci_irq()]: *** mmc0 got interrupt: 0x00000001 [ 58.591263] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000002 [ 58.591280] mmc0: req done <CMD23>: 0: 00000000 00000000 00000000 00000000 [ 58.591284] mmc0: req done (CMD25): 0: 00000900 00000000 00000000 00000000 [ 58.591287] mmc0: 4096 bytes transferred: 0 [ 58.591291] mmc0: (CMD12): 0: 00000000 00000000 00000000 00000000 [ 58.591319] mmc0: starting CMD13 arg 00010000 flags 00000195 [ 58.591352] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000001 [ 58.591367] mmc0: req done (CMD13): 0: 00000900 00000000 00000000 00000000 [ 58.594591] <mmc0: starting CMD23 arg 00000008 flags 00000015> [ 58.594597] mmc0: starting CMD18 arg 000bba38 flags 000000b5 [ 58.594601] mmc0: blksz 512 blocks 8 flags 00000200 tsac 100 ms nsac 0 [ 58.594604] mmc0: CMD12 arg 00000000 flags 00000095 [ 58.594640] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000001 [ 58.594882] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000002 [ 58.594898] mmc0: req done <CMD23>: 0: 00000000 00000000 00000000 00000000 [ 58.594902] mmc0: req done (CMD18): 0: 00000900 00000000 00000000 00000000 [ 58.594905] mmc0: 4096 bytes transferred: 0 [ 58.594908] mmc0: (CMD12): 0: 00000000 00000000 00000000 00000000 [ 58.624730] <mmc0: starting CMD23 arg 00000020 flags 00000015> [ 58.624735] mmc0: starting CMD18 arg 018be070 flags 000000b5 [ 58.624740] mmc0: blksz 512 blocks 32 flags 00000200 tsac 100 ms nsac 0 [ 58.624743] mmc0: CMD12 arg 00000000 flags 00000095 [ 58.624777] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000001 [ 58.625185] sdhci [sdhci_irq()]: *** mmc0 got interrupt: 0x00000002 [ 58.625211] mmc0: req done <CMD23>: 0: 00000000 00000000 00000000 00000000 [ 58.625216] mmc0: req done (CMD18): 0: 00000900 00000000 00000000 00000000 [ 58.625219] mmc0: 16384 bytes transferred: 0 [ 58.625222] mmc0: (CMD12): 0: 00000000 00000000 00000000 00000000 [ 58.625749] <mmc0: starting CMD23 arg 00000058 flags 00000015> [ 58.625753] mmc0: starting CMD18 arg 018be090 flags 000000b5 [ 58.625757] mmc0: blksz 512 blocks 88 flags 00000200 tsac 100 ms nsac 0 [ 58.625760] mmc0: CMD12 arg 00000000 flags 00000095 С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> To: "Greg Gallagher" <greg@embeddedgreg.com> Sent: Tuesday, March 27, 2018 6:41:19 PM Subject: Re: [Xenomai] 4.9 for x86 issue ok yes, UDOO x86 ULTRA https://www.udoo.org/docs-x86/Introduction/Introduction.html gpio https://github.com/torvalds/linux/blob/master/drivers/pinctrl/intel/pinctrl-cherryview.c С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Greg Gallagher" <greg@embeddedgreg.com> To: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> Sent: Tuesday, March 27, 2018 6:36:33 PM Subject: Re: [Xenomai] 4.9 for x86 issue For now I'll just see what is the scope of the work. The gpio I'm pretty familiar with, the sdcard shouldn't be too bad. I'm not familiar with RTNET that much but I'll see what I can do. Sent via the BlackBerry Hub for Android Original Message From: shevchenko.taras@triolcorp.com.ua Sent: March 27, 2018 11:34 AM To: greg@embeddedgreg.com Subject: Re: [Xenomai] 4.9 for x86 issue do you mean remote access ? teamwiever ? С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Greg Gallagher" <greg@embeddedgreg.com> To: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> Sent: Tuesday, March 27, 2018 6:31:04 PM Subject: Re: [Xenomai] 4.9 for x86 issue Probably be closer to the weekend since I work full time on non Xenomai things lol okay let me look into this and see what is there and what's not. Sent via the BlackBerry Hub for Android Original Message From: shevchenko.taras@triolcorp.com.ua Sent: March 27, 2018 11:27 AM To: greg@embeddedgreg.com Subject: Re: [Xenomai] 4.9 for x86 issue Perfect !! I need by priority 1)- RtNet driver to use UDP 2)- Handle gpio interrupt 3)- working card-reader if you have time today, can we use skype or viber or telegramm to communicate faster С уважением и надеждой на сотрудничество, Шевченко Т.Г. ----- Original Message ----- From: "Greg Gallagher" <greg@embeddedgreg.com> To: "Шевченко Тарас Григорьевич" <shevchenko.taras@triolcorp.com.ua> Sent: Tuesday, March 27, 2018 6:21:28 PM Subject: Re: [Xenomai] 4.9 for x86 issue Do you have a list of things that you need working on the board? I may have some time to help. Greg Sent via the BlackBerry Hub for Android Original Message From: shevchenko.taras@triolcorp.com.ua Sent: March 27, 2018 9:30 AM To: rpm@xenomai.org Cc: xenomai@xenomai.org Subject: Re: [Xenomai] 4.9 for x86 issue Sorry, but no I install 4.9.90 and new patch Ive the same problems - no responce with my LKM and SD card reader I can add logs with sd discover new details: if boot with sd-card inserted in reader - sd-card can read , if I remove card - get no responce difference with lsmod command output (when inserted card and not) is nls_iso8859_1 this module is absent without sd-card I try boot without sd card and do modprobe nls_iso8859_1 after insert sdcard - get no responce ----- Original Message ----- From: "Philippe Gerum" <rpm@xenomai.org> To: "Jan Kiszka" <jan.kiszka@web.de> Cc: "xenomai" <xenomai@xenomai.org> Sent: Tuesday, March 27, 2018 4:12:32 PM Subject: Re: [Xenomai] 4.9 for x86 issue On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. > > If you should look into 4.9-x86 as you indicated, please check this. Both issues fixed in 4.9.90/x86 as pushed lately. The result has run overnight in 64bit mode, and for a couple of hours in ia32emu mode. So far so good. -- Philippe. _______________________________________________ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai _______________________________________________ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai ^ permalink raw reply [flat|nested] 24+ messages in thread
* [Xenomai] [PULL] ipipe 4.4 updates
@ 2018-03-09 16:51 Jan Kiszka
2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka
0 siblings, 1 reply; 24+ messages in thread
From: Jan Kiszka @ 2018-03-09 16:51 UTC (permalink / raw)
To: Philippe Gerum; +Cc: Xenomai
The following changes since commit 5618f90d33b17ab0ee9352705610a5e9f00f9964:
Revert "ipipe: printk: defer dev_printk() output to safe context" (2017-12-15 18:10:01 +0100)
are available in the git repository at:
git://git.xenomai.org/ipipe-jki for-upstream/4.4-update
for you to fetch changes up to c6de6878f74e4b079066eddc373f3e8c125fdaa7:
Merge tag 'v4.4.120' into for-upstream/4.4-update (2018-03-08 18:49:28 +0100)
I-pipe specific commits:
Jan Kiszka (5):
arm/ipipe: Avoid open-coded __ipipe_call_mayday
x86/ipipe: Deny JUMP_LABEL over I-pipe
Merge tag 'v4.4.109' into for-upstream/4.4-update
x86/ipipe: Disable access_ok context under I-pipe
Merge tag 'v4.4.120' into for-upstream/4.4-update
Philippe Gerum (1):
ipipe: printk: enable dev_printk() from the head stage
4.9 requires more work, I've pushed the beginning to wip/4.9 in the same
repo.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 24+ messages in thread* [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) 2018-03-09 16:51 [Xenomai] [PULL] ipipe 4.4 updates Jan Kiszka @ 2018-03-10 22:06 ` Jan Kiszka 2018-03-19 16:54 ` [Xenomai] 4.9 for x86 issue Philippe Gerum ` (3 more replies) 0 siblings, 4 replies; 24+ messages in thread From: Jan Kiszka @ 2018-03-10 22:06 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-03-09 08:51, Jan Kiszka wrote: > 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same > repo. I started to patch further on this during my flight (wip/4.9 updated), noticed that the 4.14-wip queue will need a little bit sysentry tweaking as well (missing 64-bit syscall dispatching), and then had to find 4.9 in a rather unfortunate state /wrt x86-64: CPUs are no longer idling properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. If you should look into 4.9-x86 as you indicated, please check this. Another catch along that road: maxcpus=1 with >1 CPUs available triggers a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well. Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180310/1e8dff24/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka @ 2018-03-19 16:54 ` Philippe Gerum 2018-03-20 3:42 ` Jan Kiszka 2018-03-19 16:59 ` Philippe Gerum ` (2 subsequent siblings) 3 siblings, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-03-19 16:54 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. Meaning they do not enter any high CSTATE although they should with CPU_IDLE* properly set, or something breaks when doing so? Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here. Thanks, -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-19 16:54 ` [Xenomai] 4.9 for x86 issue Philippe Gerum @ 2018-03-20 3:42 ` Jan Kiszka 2018-03-20 14:22 ` Philippe Gerum 0 siblings, 1 reply; 24+ messages in thread From: Jan Kiszka @ 2018-03-20 3:42 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-03-20 00:54, Philippe Gerum wrote: > On 03/10/2018 11:06 PM, Jan Kiszka wrote: >> On 2018-03-09 08:51, Jan Kiszka wrote: >>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>> repo. >> >> I started to patch further on this during my flight (wip/4.9 updated), >> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >> properly. > > Meaning they do not enter any high CSTATE although they should with > CPU_IDLE* properly set, or something breaks when doing so? This was in KVM, no CSTATEs involved. > > Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here. > # # CPU Idle # CONFIG_CPU_IDLE=y CONFIG_CPU_IDLE_GOV_LADDER=y CONFIG_CPU_IDLE_GOV_MENU=y # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set # CONFIG_INTEL_IDLE is not set TIA, Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-20 3:42 ` Jan Kiszka @ 2018-03-20 14:22 ` Philippe Gerum 0 siblings, 0 replies; 24+ messages in thread From: Philippe Gerum @ 2018-03-20 14:22 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 03/20/2018 04:42 AM, Jan Kiszka wrote: > On 2018-03-20 00:54, Philippe Gerum wrote: >> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>> repo. >>> >>> I started to patch further on this during my flight (wip/4.9 updated), >>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>> properly. >> >> Meaning they do not enter any high CSTATE although they should with >> CPU_IDLE* properly set, or something breaks when doing so? > > This was in KVM, no CSTATEs involved. > >> >> Please share your CONFIG_CPU_IDLE* settings. I'll try to reproduce here. >> > > # > # CPU Idle > # > CONFIG_CPU_IDLE=y > CONFIG_CPU_IDLE_GOV_LADDER=y > CONFIG_CPU_IDLE_GOV_MENU=y > # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set > # CONFIG_INTEL_IDLE is not set > Ok, #89146106e8 should restore the previous pipeline behavior from 4.4, #595606457f in -stable should enable an (hopefully) smarter handling with Xenomai having its say on whether the idle state should be entered. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka 2018-03-19 16:54 ` [Xenomai] 4.9 for x86 issue Philippe Gerum @ 2018-03-19 16:59 ` Philippe Gerum 2018-03-20 15:05 ` Philippe Gerum 2018-03-27 13:12 ` Philippe Gerum 3 siblings, 0 replies; 24+ messages in thread From: Philippe Gerum @ 2018-03-19 16:59 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. > > If you should look into 4.9-x86 as you indicated, please check this. > > Another catch along that road: maxcpus=1 with >1 CPUs available triggers > a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well. > In the same vein, CONFIG_MAXSMP w/ CONFIG_IPIPE breaks the kernel at boot on x86. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka 2018-03-19 16:54 ` [Xenomai] 4.9 for x86 issue Philippe Gerum 2018-03-19 16:59 ` Philippe Gerum @ 2018-03-20 15:05 ` Philippe Gerum 2018-03-27 13:12 ` Philippe Gerum 3 siblings, 0 replies; 24+ messages in thread From: Philippe Gerum @ 2018-03-20 15:05 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. > > If you should look into 4.9-x86 as you indicated, please check this. > > Another catch along that road: maxcpus=1 with >1 CPUs available triggers > a NULL pointer when Xenomai gets armed. That is the case with 4.4 as well. > Can't reproduce this one (over kvm), 4.4 and 4.9 included, so this may be depending on the Kconfig. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka ` (2 preceding siblings ...) 2018-03-20 15:05 ` Philippe Gerum @ 2018-03-27 13:12 ` Philippe Gerum 2018-03-27 13:30 ` Шевченко Тарас Григорьевич ` (2 more replies) 3 siblings, 3 replies; 24+ messages in thread From: Philippe Gerum @ 2018-03-27 13:12 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. > > If you should look into 4.9-x86 as you indicated, please check this. Both issues fixed in 4.9.90/x86 as pushed lately. The result has run overnight in 64bit mode, and for a couple of hours in ia32emu mode. So far so good. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-27 13:12 ` Philippe Gerum @ 2018-03-27 13:30 ` Шевченко Тарас Григорьевич 2018-03-27 16:42 ` Jan Kiszka 2018-04-05 20:13 ` Jan Kiszka 2 siblings, 0 replies; 24+ messages in thread From: Шевченко Тарас Григорьевич @ 2018-03-27 13:30 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai Sorry, but no I install 4.9.90 and new patch Ive the same problems - no responce with my LKM and SD card reader I can add logs with sd discover new details: if boot with sd-card inserted in reader - sd-card can read , if I remove card - get no responce difference with lsmod command output (when inserted card and not) is nls_iso8859_1 this module is absent without sd-card I try boot without sd card and do modprobe nls_iso8859_1 after insert sdcard - get no responce ----- Original Message ----- From: "Philippe Gerum" <rpm@xenomai.org> To: "Jan Kiszka" <jan.kiszka@web.de> Cc: "xenomai" <xenomai@xenomai.org> Sent: Tuesday, March 27, 2018 4:12:32 PM Subject: Re: [Xenomai] 4.9 for x86 issue On 03/10/2018 11:06 PM, Jan Kiszka wrote: > On 2018-03-09 08:51, Jan Kiszka wrote: >> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >> repo. > > I started to patch further on this during my flight (wip/4.9 updated), > noticed that the 4.14-wip queue will need a little bit sysentry tweaking > as well (missing 64-bit syscall dispatching), and then had to find 4.9 > in a rather unfortunate state /wrt x86-64: CPUs are no longer idling > properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. > > If you should look into 4.9-x86 as you indicated, please check this. Both issues fixed in 4.9.90/x86 as pushed lately. The result has run overnight in 64bit mode, and for a couple of hours in ia32emu mode. So far so good. -- Philippe. _______________________________________________ Xenomai mailing list Xenomai@xenomai.org https://xenomai.org/mailman/listinfo/xenomai ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-27 13:12 ` Philippe Gerum 2018-03-27 13:30 ` Шевченко Тарас Григорьевич @ 2018-03-27 16:42 ` Jan Kiszka 2018-04-05 20:13 ` Jan Kiszka 2 siblings, 0 replies; 24+ messages in thread From: Jan Kiszka @ 2018-03-27 16:42 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-03-27 15:12, Philippe Gerum wrote: > On 03/10/2018 11:06 PM, Jan Kiszka wrote: >> On 2018-03-09 08:51, Jan Kiszka wrote: >>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>> repo. >> >> I started to patch further on this during my flight (wip/4.9 updated), >> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >> >> If you should look into 4.9-x86 as you indicated, please check this. > > Both issues fixed in 4.9.90/x86 as pushed lately. The result has run > overnight in 64bit mode, and for a couple of hours in ia32emu mode. So > far so good. Great, thanks! Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180327/ae5ee943/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-03-27 13:12 ` Philippe Gerum 2018-03-27 13:30 ` Шевченко Тарас Григорьевич 2018-03-27 16:42 ` Jan Kiszka @ 2018-04-05 20:13 ` Jan Kiszka 2018-04-06 6:54 ` Philippe Gerum 2018-04-06 16:09 ` Jan Kiszka 2 siblings, 2 replies; 24+ messages in thread From: Jan Kiszka @ 2018-04-05 20:13 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-03-27 15:12, Philippe Gerum wrote: > On 03/10/2018 11:06 PM, Jan Kiszka wrote: >> On 2018-03-09 08:51, Jan Kiszka wrote: >>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>> repo. >> >> I started to patch further on this during my flight (wip/4.9 updated), >> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >> >> If you should look into 4.9-x86 as you indicated, please check this. > > Both issues fixed in 4.9.90/x86 as pushed lately. The result has run > overnight in 64bit mode, and for a couple of hours in ia32emu mode. So > far so good. Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) CPU load. I also triggered this with stable-3.0.x: [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 [ 237.460728] Modules linked in: [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 [ 237.461490] I-pipe domain: Xenomai [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 [ 237.461490] Call Trace: [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- I will debug this tomorrow. Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180405/a998b4ce/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-05 20:13 ` Jan Kiszka @ 2018-04-06 6:54 ` Philippe Gerum 2018-04-06 13:38 ` Jan Kiszka 2018-04-06 16:09 ` Jan Kiszka 1 sibling, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-06 6:54 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/05/2018 10:13 PM, Jan Kiszka wrote: > On 2018-03-27 15:12, Philippe Gerum wrote: >> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>> repo. >>> >>> I started to patch further on this during my flight (wip/4.9 updated), >>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>> >>> If you should look into 4.9-x86 as you indicated, please check this. >> >> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >> far so good. > > Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) > CPU load. I also triggered this with stable-3.0.x: > > [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 > [ 237.460728] Modules linked in: > [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 > [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 > [ 237.461490] I-pipe domain: Xenomai > [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 > [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 > [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 > [ 237.461490] Call Trace: > [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd > [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 > [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 > [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 > [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 > [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 > [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 > [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 > [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 > [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 > [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 > [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb > [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- > > I will debug this tomorrow. > I can't reproduce this, the loadavg on my qemu instance consistently converges to 0.0x figures while running the latency test (10Khz or 1Khz, same). I'm now running 4.9.92, but I don't think this should make any difference, since I could trace the box entering the idle state on .90. Are you running the ia32emu mode, or x86_64? Also, could you share your .config for building the guest kernel? Thanks, -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-06 6:54 ` Philippe Gerum @ 2018-04-06 13:38 ` Jan Kiszka 2018-04-06 14:11 ` Philippe Gerum 0 siblings, 1 reply; 24+ messages in thread From: Jan Kiszka @ 2018-04-06 13:38 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-06 08:54, Philippe Gerum wrote: > On 04/05/2018 10:13 PM, Jan Kiszka wrote: >> On 2018-03-27 15:12, Philippe Gerum wrote: >>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>> repo. >>>> >>>> I started to patch further on this during my flight (wip/4.9 updated), >>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>> >>>> If you should look into 4.9-x86 as you indicated, please check this. >>> >>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>> far so good. >> >> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >> CPU load. I also triggered this with stable-3.0.x: >> >> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >> [ 237.460728] Modules linked in: >> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >> [ 237.461490] I-pipe domain: Xenomai >> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >> [ 237.461490] Call Trace: >> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >> >> I will debug this tomorrow. >> > > I can't reproduce this, the loadavg on my qemu instance consistently > converges to 0.0x figures while running the latency test (10Khz or 1Khz, > same). I'm now running 4.9.92, but I don't think this should make any > difference, since I could trace the box entering the idle state on .90. > > Are you running the ia32emu mode, or x86_64? Also, could you share your > .config for building the guest kernel? Config is the same I sent back then. Userland is 64-bit, compat support enabled. The reason I see so far: xnclock_core_local_shot never sets XNIDLE. I suspect we always have a timer registered, that for the host clock. So we can't become idle this way. I'm not even sure that this test makes sense because a pending RT timer does not make a non-idle system. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-06 13:38 ` Jan Kiszka @ 2018-04-06 14:11 ` Philippe Gerum 2018-04-06 15:11 ` Jan Kiszka 0 siblings, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-06 14:11 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/06/2018 03:38 PM, Jan Kiszka wrote: > On 2018-04-06 08:54, Philippe Gerum wrote: >> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>> repo. >>>>> >>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>> >>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>> >>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>> far so good. >>> >>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>> CPU load. I also triggered this with stable-3.0.x: >>> >>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>> [ 237.460728] Modules linked in: >>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>> [ 237.461490] I-pipe domain: Xenomai >>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>> [ 237.461490] Call Trace: >>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>> >>> I will debug this tomorrow. >>> >> >> I can't reproduce this, the loadavg on my qemu instance consistently >> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >> same). I'm now running 4.9.92, but I don't think this should make any >> difference, since I could trace the box entering the idle state on .90. >> >> Are you running the ia32emu mode, or x86_64? Also, could you share your >> .config for building the guest kernel? > > Config is the same I sent back then. Userland is 64-bit, compat support > enabled. You only sent me the CONFIG_IDLE* settings I asked for. I'd need the whole file now. > > The reason I see so far: xnclock_core_local_shot never sets XNIDLE. It does here (I traced it). However this should depend on the NO_HZ settings, mine are : CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set CONFIG_NO_HZ_IDLE=y CONFIG_NO_HZ=y > I > suspect we always have a timer registered, that for the host clock. So In that case, the timer is not idle Xenomai-wise. > we can't become idle this way. I'm not even sure that this test makes > sense because a pending RT timer does not make a non-idle system. > This is not about testing for Cobalt idleness, but for its core timer idleness, given that the core timer is shared between both kernels. We want to know whether we may allow the regular kernel to shutdown the clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE if you will. I covered this stuff in Documentation/ipipe.rst lately. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-06 14:11 ` Philippe Gerum @ 2018-04-06 15:11 ` Jan Kiszka 2018-04-06 15:52 ` Jan Kiszka 2018-04-07 7:25 ` Philippe Gerum 0 siblings, 2 replies; 24+ messages in thread From: Jan Kiszka @ 2018-04-06 15:11 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-06 16:11, Philippe Gerum wrote: > On 04/06/2018 03:38 PM, Jan Kiszka wrote: >> On 2018-04-06 08:54, Philippe Gerum wrote: >>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>> repo. >>>>>> >>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>> >>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>> >>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>> far so good. >>>> >>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>> CPU load. I also triggered this with stable-3.0.x: >>>> >>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>> [ 237.460728] Modules linked in: >>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>> [ 237.461490] I-pipe domain: Xenomai >>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>> [ 237.461490] Call Trace: >>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>> >>>> I will debug this tomorrow. >>>> >>> >>> I can't reproduce this, the loadavg on my qemu instance consistently >>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>> same). I'm now running 4.9.92, but I don't think this should make any >>> difference, since I could trace the box entering the idle state on .90. >>> >>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>> .config for building the guest kernel? >> >> Config is the same I sent back then. Userland is 64-bit, compat support >> enabled. > > You only sent me the CONFIG_IDLE* settings I asked for. I'd need the > whole file now. Sorry, though I did. Attached. > >> >> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. > > It does here (I traced it). However this should depend on the NO_HZ > settings, mine are : > > CONFIG_TICK_ONESHOT=y > CONFIG_NO_HZ_COMMON=y > # CONFIG_HZ_PERIODIC is not set > CONFIG_NO_HZ_IDLE=y > CONFIG_NO_HZ=y > Same here. >> I >> suspect we always have a timer registered, that for the host clock. So > > In that case, the timer is not idle Xenomai-wise. > >> we can't become idle this way. I'm not even sure that this test makes >> sense because a pending RT timer does not make a non-idle system. >> > > This is not about testing for Cobalt idleness, but for its core timer > idleness, given that the core timer is shared between both kernels. We > want to know whether we may allow the regular kernel to shutdown the > clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE > if you will. I covered this stuff in Documentation/ipipe.rst lately. > I still don't see the problem. We own the timer, Linux does not program it. And letting Linux call hlt does not disturb the timer programming, in most cases at least (there might be some weird old broken hardware). Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: .config.xz Type: application/x-xz Size: 21268 bytes Desc: not available URL: <http://xenomai.org/pipermail/xenomai/attachments/20180406/32e24aae/attachment.bin> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-06 15:11 ` Jan Kiszka @ 2018-04-06 15:52 ` Jan Kiszka 2018-04-07 7:25 ` Philippe Gerum 1 sibling, 0 replies; 24+ messages in thread From: Jan Kiszka @ 2018-04-06 15:52 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-06 17:11, Jan Kiszka wrote: >> This is not about testing for Cobalt idleness, but for its core timer >> idleness, given that the core timer is shared between both kernels. We >> want to know whether we may allow the regular kernel to shutdown the >> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >> if you will. I covered this stuff in Documentation/ipipe.rst lately. >> > > I still don't see the problem. We own the timer, Linux does not program > it. And letting Linux call hlt does not disturb the timer programming, > in most cases at least (there might be some weird old broken hardware). Looking at ipipe_enter_idle_hook more closely: What might make sense as logic here is a test for a close-to-expire timer on the CPU. Then we would do polling until the expiry which may reduce wakeup latencies. But the question would be what the thresholds should be on the different targets. Did you see any concrete issues without that hook? On which architecture? Jan ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-06 15:11 ` Jan Kiszka 2018-04-06 15:52 ` Jan Kiszka @ 2018-04-07 7:25 ` Philippe Gerum 2018-04-07 9:42 ` Jan Kiszka 1 sibling, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-07 7:25 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/06/2018 05:11 PM, Jan Kiszka wrote: > On 2018-04-06 16:11, Philippe Gerum wrote: >> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>> repo. >>>>>>> >>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>> >>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>> >>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>> far so good. >>>>> >>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>> CPU load. I also triggered this with stable-3.0.x: >>>>> >>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>> [ 237.460728] Modules linked in: >>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>> [ 237.461490] I-pipe domain: Xenomai >>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>> [ 237.461490] Call Trace: >>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>> >>>>> I will debug this tomorrow. >>>>> >>>> >>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>> same). I'm now running 4.9.92, but I don't think this should make any >>>> difference, since I could trace the box entering the idle state on .90. >>>> >>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>> .config for building the guest kernel? >>> >>> Config is the same I sent back then. Userland is 64-bit, compat support >>> enabled. >> >> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >> whole file now. > > Sorry, though I did. Attached. > Thanks, >> >>> >>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >> >> It does here (I traced it). However this should depend on the NO_HZ >> settings, mine are : >> >> CONFIG_TICK_ONESHOT=y >> CONFIG_NO_HZ_COMMON=y >> # CONFIG_HZ_PERIODIC is not set >> CONFIG_NO_HZ_IDLE=y >> CONFIG_NO_HZ=y >> > > Same here. > >>> I >>> suspect we always have a timer registered, that for the host clock. So >> >> In that case, the timer is not idle Xenomai-wise. >> >>> we can't become idle this way. I'm not even sure that this test makes >>> sense because a pending RT timer does not make a non-idle system. >>> >> >> This is not about testing for Cobalt idleness, but for its core timer >> idleness, given that the core timer is shared between both kernels. We >> want to know whether we may allow the regular kernel to shutdown the >> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >> if you will. I covered this stuff in Documentation/ipipe.rst lately. >> > > I still don't see the problem. We own the timer, Linux does not program > it. And letting Linux call hlt does not disturb the timer programming, > in most cases at least (there might be some weird old broken hardware). The problem is not with hlt, but with the tick device switch when c3stop is enabled on the device, and going idle means shutting it down before switching to a broadcast device. Very unfortunately, this is not even an x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 7:25 ` Philippe Gerum @ 2018-04-07 9:42 ` Jan Kiszka 2018-04-07 16:55 ` Philippe Gerum 0 siblings, 1 reply; 24+ messages in thread From: Jan Kiszka @ 2018-04-07 9:42 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-07 09:25, Philippe Gerum wrote: > On 04/06/2018 05:11 PM, Jan Kiszka wrote: >> On 2018-04-06 16:11, Philippe Gerum wrote: >>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>> repo. >>>>>>>> >>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>> >>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>> >>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>> far so good. >>>>>> >>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>> >>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>> [ 237.460728] Modules linked in: >>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>> [ 237.461490] Call Trace: >>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>> >>>>>> I will debug this tomorrow. >>>>>> >>>>> >>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>> difference, since I could trace the box entering the idle state on .90. >>>>> >>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>> .config for building the guest kernel? >>>> >>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>> enabled. >>> >>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>> whole file now. >> >> Sorry, though I did. Attached. >> > > Thanks, > >>> >>>> >>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>> >>> It does here (I traced it). However this should depend on the NO_HZ >>> settings, mine are : >>> >>> CONFIG_TICK_ONESHOT=y >>> CONFIG_NO_HZ_COMMON=y >>> # CONFIG_HZ_PERIODIC is not set >>> CONFIG_NO_HZ_IDLE=y >>> CONFIG_NO_HZ=y >>> >> >> Same here. >> >>>> I >>>> suspect we always have a timer registered, that for the host clock. So >>> >>> In that case, the timer is not idle Xenomai-wise. >>> >>>> we can't become idle this way. I'm not even sure that this test makes >>>> sense because a pending RT timer does not make a non-idle system. >>>> >>> >>> This is not about testing for Cobalt idleness, but for its core timer >>> idleness, given that the core timer is shared between both kernels. We >>> want to know whether we may allow the regular kernel to shutdown the >>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>> >> >> I still don't see the problem. We own the timer, Linux does not program >> it. And letting Linux call hlt does not disturb the timer programming, >> in most cases at least (there might be some weird old broken hardware). > > > The problem is not with hlt, but with the tick device switch when c3stop > is enabled on the device, and going idle means shutting it down before > switching to a broadcast device. Very unfortunately, this is not even an > x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. > So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on those clockevent devices that we take over for Xenomai purposes, right? That excludes the vast majority of systems Xenomai runs on. It specifically excludes any modern x86 systems which now have ARAT support. For the remaining ones, we always recommended to have ACPI_PROCESSOR disabled, thus have no need for this new workaround either. Regarding the mentioned ARM system: Is there no equivalent to !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi completely? We definitely need a much more targeted solution here. Any suggestions? Jan ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 9:42 ` Jan Kiszka @ 2018-04-07 16:55 ` Philippe Gerum 2018-04-07 16:58 ` Jan Kiszka 0 siblings, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-07 16:55 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/07/2018 11:42 AM, Jan Kiszka wrote: > On 2018-04-07 09:25, Philippe Gerum wrote: >> On 04/06/2018 05:11 PM, Jan Kiszka wrote: >>> On 2018-04-06 16:11, Philippe Gerum wrote: >>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>>> repo. >>>>>>>>> >>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>>> >>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>>> >>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>>> far so good. >>>>>>> >>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>>> >>>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>>> [ 237.460728] Modules linked in: >>>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>>> [ 237.461490] Call Trace: >>>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>>> >>>>>>> I will debug this tomorrow. >>>>>>> >>>>>> >>>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>>> difference, since I could trace the box entering the idle state on .90. >>>>>> >>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>>> .config for building the guest kernel? >>>>> >>>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>>> enabled. >>>> >>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>>> whole file now. >>> >>> Sorry, though I did. Attached. >>> >> >> Thanks, >> >>>> >>>>> >>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>>> >>>> It does here (I traced it). However this should depend on the NO_HZ >>>> settings, mine are : >>>> >>>> CONFIG_TICK_ONESHOT=y >>>> CONFIG_NO_HZ_COMMON=y >>>> # CONFIG_HZ_PERIODIC is not set >>>> CONFIG_NO_HZ_IDLE=y >>>> CONFIG_NO_HZ=y >>>> >>> >>> Same here. >>> >>>>> I >>>>> suspect we always have a timer registered, that for the host clock. So >>>> >>>> In that case, the timer is not idle Xenomai-wise. >>>> >>>>> we can't become idle this way. I'm not even sure that this test makes >>>>> sense because a pending RT timer does not make a non-idle system. >>>>> >>>> >>>> This is not about testing for Cobalt idleness, but for its core timer >>>> idleness, given that the core timer is shared between both kernels. We >>>> want to know whether we may allow the regular kernel to shutdown the >>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>>> >>> >>> I still don't see the problem. We own the timer, Linux does not program >>> it. And letting Linux call hlt does not disturb the timer programming, >>> in most cases at least (there might be some weird old broken hardware). >> >> >> The problem is not with hlt, but with the tick device switch when c3stop >> is enabled on the device, and going idle means shutting it down before >> switching to a broadcast device. Very unfortunately, this is not even an >> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. >> > > So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on > those clockevent devices that we take over for Xenomai purposes, right? > That excludes the vast majority of systems Xenomai runs on. It > specifically excludes any modern x86 systems which now have ARAT > support. For the remaining ones, we always recommended to have > ACPI_PROCESSOR disabled, thus have no need for this new workaround either. > This is not a work around. The fact that linux is not aware that a second kernel may consider the timer as busy is a general issue, and asking that second kernel to have its say is not a work around, but a requirement. > Regarding the mentioned ARM system: Is there no equivalent to > !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi > completely? We definitely need a much more targeted solution here. Any > suggestions? > Yes, try to find what does not work in your case. I'll try to reproduce tomorrow using your Kconfig. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 16:55 ` Philippe Gerum @ 2018-04-07 16:58 ` Jan Kiszka 2018-04-07 17:04 ` Philippe Gerum 0 siblings, 1 reply; 24+ messages in thread From: Jan Kiszka @ 2018-04-07 16:58 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-07 18:55, Philippe Gerum wrote: > On 04/07/2018 11:42 AM, Jan Kiszka wrote: >> On 2018-04-07 09:25, Philippe Gerum wrote: >>> On 04/06/2018 05:11 PM, Jan Kiszka wrote: >>>> On 2018-04-06 16:11, Philippe Gerum wrote: >>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>>>> repo. >>>>>>>>>> >>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>>>> >>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>>>> >>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>>>> far so good. >>>>>>>> >>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>>>> >>>>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>>>> [ 237.460728] Modules linked in: >>>>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>>>> [ 237.461490] Call Trace: >>>>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>>>> >>>>>>>> I will debug this tomorrow. >>>>>>>> >>>>>>> >>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>>>> difference, since I could trace the box entering the idle state on .90. >>>>>>> >>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>>>> .config for building the guest kernel? >>>>>> >>>>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>>>> enabled. >>>>> >>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>>>> whole file now. >>>> >>>> Sorry, though I did. Attached. >>>> >>> >>> Thanks, >>> >>>>> >>>>>> >>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>>>> >>>>> It does here (I traced it). However this should depend on the NO_HZ >>>>> settings, mine are : >>>>> >>>>> CONFIG_TICK_ONESHOT=y >>>>> CONFIG_NO_HZ_COMMON=y >>>>> # CONFIG_HZ_PERIODIC is not set >>>>> CONFIG_NO_HZ_IDLE=y >>>>> CONFIG_NO_HZ=y >>>>> >>>> >>>> Same here. >>>> >>>>>> I >>>>>> suspect we always have a timer registered, that for the host clock. So >>>>> >>>>> In that case, the timer is not idle Xenomai-wise. >>>>> >>>>>> we can't become idle this way. I'm not even sure that this test makes >>>>>> sense because a pending RT timer does not make a non-idle system. >>>>>> >>>>> >>>>> This is not about testing for Cobalt idleness, but for its core timer >>>>> idleness, given that the core timer is shared between both kernels. We >>>>> want to know whether we may allow the regular kernel to shutdown the >>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>>>> >>>> >>>> I still don't see the problem. We own the timer, Linux does not program >>>> it. And letting Linux call hlt does not disturb the timer programming, >>>> in most cases at least (there might be some weird old broken hardware). >>> >>> >>> The problem is not with hlt, but with the tick device switch when c3stop >>> is enabled on the device, and going idle means shutting it down before >>> switching to a broadcast device. Very unfortunately, this is not even an >>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. >>> >> >> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on >> those clockevent devices that we take over for Xenomai purposes, right? >> That excludes the vast majority of systems Xenomai runs on. It >> specifically excludes any modern x86 systems which now have ARAT >> support. For the remaining ones, we always recommended to have >> ACPI_PROCESSOR disabled, thus have no need for this new workaround either. >> > > This is not a work around. The fact that linux is not aware that a > second kernel may consider the timer as busy is a general issue, and > asking that second kernel to have its say is not a work around, but a > requirement. > >> Regarding the mentioned ARM system: Is there no equivalent to >> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi >> completely? We definitely need a much more targeted solution here. Any >> suggestions? >> > > Yes, try to find what does not work in your case. I'll try to reproduce > tomorrow using your Kconfig. I think you are still on the wrong track here: x86 is not at all affected by the issue you see. Please focus on the ARM system where you saw the issue in the first place. Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/b4daca80/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 16:58 ` Jan Kiszka @ 2018-04-07 17:04 ` Philippe Gerum 2018-04-07 17:10 ` Jan Kiszka 0 siblings, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-07 17:04 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/07/2018 06:58 PM, Jan Kiszka wrote: > On 2018-04-07 18:55, Philippe Gerum wrote: >> On 04/07/2018 11:42 AM, Jan Kiszka wrote: >>> On 2018-04-07 09:25, Philippe Gerum wrote: >>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote: >>>>> On 2018-04-06 16:11, Philippe Gerum wrote: >>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>>>>> repo. >>>>>>>>>>> >>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>>>>> >>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>>>>> >>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>>>>> far so good. >>>>>>>>> >>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>>>>> >>>>>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>>>>> [ 237.460728] Modules linked in: >>>>>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>>>>> [ 237.461490] Call Trace: >>>>>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>>>>> >>>>>>>>> I will debug this tomorrow. >>>>>>>>> >>>>>>>> >>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>>>>> difference, since I could trace the box entering the idle state on .90. >>>>>>>> >>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>>>>> .config for building the guest kernel? >>>>>>> >>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>>>>> enabled. >>>>>> >>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>>>>> whole file now. >>>>> >>>>> Sorry, though I did. Attached. >>>>> >>>> >>>> Thanks, >>>> >>>>>> >>>>>>> >>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>>>>> >>>>>> It does here (I traced it). However this should depend on the NO_HZ >>>>>> settings, mine are : >>>>>> >>>>>> CONFIG_TICK_ONESHOT=y >>>>>> CONFIG_NO_HZ_COMMON=y >>>>>> # CONFIG_HZ_PERIODIC is not set >>>>>> CONFIG_NO_HZ_IDLE=y >>>>>> CONFIG_NO_HZ=y >>>>>> >>>>> >>>>> Same here. >>>>> >>>>>>> I >>>>>>> suspect we always have a timer registered, that for the host clock. So >>>>>> >>>>>> In that case, the timer is not idle Xenomai-wise. >>>>>> >>>>>>> we can't become idle this way. I'm not even sure that this test makes >>>>>>> sense because a pending RT timer does not make a non-idle system. >>>>>>> >>>>>> >>>>>> This is not about testing for Cobalt idleness, but for its core timer >>>>>> idleness, given that the core timer is shared between both kernels. We >>>>>> want to know whether we may allow the regular kernel to shutdown the >>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>>>>> >>>>> >>>>> I still don't see the problem. We own the timer, Linux does not program >>>>> it. And letting Linux call hlt does not disturb the timer programming, >>>>> in most cases at least (there might be some weird old broken hardware). >>>> >>>> >>>> The problem is not with hlt, but with the tick device switch when c3stop >>>> is enabled on the device, and going idle means shutting it down before >>>> switching to a broadcast device. Very unfortunately, this is not even an >>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. >>>> >>> >>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on >>> those clockevent devices that we take over for Xenomai purposes, right? >>> That excludes the vast majority of systems Xenomai runs on. It >>> specifically excludes any modern x86 systems which now have ARAT >>> support. For the remaining ones, we always recommended to have >>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either. >>> >> >> This is not a work around. The fact that linux is not aware that a >> second kernel may consider the timer as busy is a general issue, and >> asking that second kernel to have its say is not a work around, but a >> requirement. >> >>> Regarding the mentioned ARM system: Is there no equivalent to >>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi >>> completely? We definitely need a much more targeted solution here. Any >>> suggestions? >>> >> >> Yes, try to find what does not work in your case. I'll try to reproduce >> tomorrow using your Kconfig. > > I think you are still on the wrong track here: x86 is not at all > affected by the issue you see. You misread me: I'm talking about finding why your config never matches the idle state Xenomai-wise. Please focus on the ARM system where you > saw the issue in the first place. > This is a problem which needs a resolution starting from the generic code. So let's focus on finding why the x86-specific case does not work instead. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 17:04 ` Philippe Gerum @ 2018-04-07 17:10 ` Jan Kiszka 2018-04-07 17:20 ` Philippe Gerum 0 siblings, 1 reply; 24+ messages in thread From: Jan Kiszka @ 2018-04-07 17:10 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-07 19:04, Philippe Gerum wrote: > On 04/07/2018 06:58 PM, Jan Kiszka wrote: >> On 2018-04-07 18:55, Philippe Gerum wrote: >>> On 04/07/2018 11:42 AM, Jan Kiszka wrote: >>>> On 2018-04-07 09:25, Philippe Gerum wrote: >>>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote: >>>>>> On 2018-04-06 16:11, Philippe Gerum wrote: >>>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>>>>>> repo. >>>>>>>>>>>> >>>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>>>>>> >>>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>>>>>> >>>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>>>>>> far so good. >>>>>>>>>> >>>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>>>>>> >>>>>>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>>>>>> [ 237.460728] Modules linked in: >>>>>>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>>>>>> [ 237.461490] Call Trace: >>>>>>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>>>>>> >>>>>>>>>> I will debug this tomorrow. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>>>>>> difference, since I could trace the box entering the idle state on .90. >>>>>>>>> >>>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>>>>>> .config for building the guest kernel? >>>>>>>> >>>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>>>>>> enabled. >>>>>>> >>>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>>>>>> whole file now. >>>>>> >>>>>> Sorry, though I did. Attached. >>>>>> >>>>> >>>>> Thanks, >>>>> >>>>>>> >>>>>>>> >>>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>>>>>> >>>>>>> It does here (I traced it). However this should depend on the NO_HZ >>>>>>> settings, mine are : >>>>>>> >>>>>>> CONFIG_TICK_ONESHOT=y >>>>>>> CONFIG_NO_HZ_COMMON=y >>>>>>> # CONFIG_HZ_PERIODIC is not set >>>>>>> CONFIG_NO_HZ_IDLE=y >>>>>>> CONFIG_NO_HZ=y >>>>>>> >>>>>> >>>>>> Same here. >>>>>> >>>>>>>> I >>>>>>>> suspect we always have a timer registered, that for the host clock. So >>>>>>> >>>>>>> In that case, the timer is not idle Xenomai-wise. >>>>>>> >>>>>>>> we can't become idle this way. I'm not even sure that this test makes >>>>>>>> sense because a pending RT timer does not make a non-idle system. >>>>>>>> >>>>>>> >>>>>>> This is not about testing for Cobalt idleness, but for its core timer >>>>>>> idleness, given that the core timer is shared between both kernels. We >>>>>>> want to know whether we may allow the regular kernel to shutdown the >>>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>>>>>> >>>>>> >>>>>> I still don't see the problem. We own the timer, Linux does not program >>>>>> it. And letting Linux call hlt does not disturb the timer programming, >>>>>> in most cases at least (there might be some weird old broken hardware). >>>>> >>>>> >>>>> The problem is not with hlt, but with the tick device switch when c3stop >>>>> is enabled on the device, and going idle means shutting it down before >>>>> switching to a broadcast device. Very unfortunately, this is not even an >>>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. >>>>> >>>> >>>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on >>>> those clockevent devices that we take over for Xenomai purposes, right? >>>> That excludes the vast majority of systems Xenomai runs on. It >>>> specifically excludes any modern x86 systems which now have ARAT >>>> support. For the remaining ones, we always recommended to have >>>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either. >>>> >>> >>> This is not a work around. The fact that linux is not aware that a >>> second kernel may consider the timer as busy is a general issue, and >>> asking that second kernel to have its say is not a work around, but a >>> requirement. >>> >>>> Regarding the mentioned ARM system: Is there no equivalent to >>>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi >>>> completely? We definitely need a much more targeted solution here. Any >>>> suggestions? >>>> >>> >>> Yes, try to find what does not work in your case. I'll try to reproduce >>> tomorrow using your Kconfig. >> >> I think you are still on the wrong track here: x86 is not at all >> affected by the issue you see. > > You misread me: I'm talking about finding why your config never matches > the idle state Xenomai-wise. > > Please focus on the ARM system where you >> saw the issue in the first place. >> > > This is a problem which needs a resolution starting from the generic > code. So let's focus on finding why the x86-specific case does not work > instead. ...because your approach is not correct at all. The goal must be to prevent to enter that sleep state, which we achieve on x86 differently - or do not have to worry about anymore. But you hook into Linux entering idle state in general and prevent that. Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/73a32e24/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 17:10 ` Jan Kiszka @ 2018-04-07 17:20 ` Philippe Gerum 2018-04-07 17:33 ` Jan Kiszka 0 siblings, 1 reply; 24+ messages in thread From: Philippe Gerum @ 2018-04-07 17:20 UTC (permalink / raw) To: Jan Kiszka; +Cc: Xenomai On 04/07/2018 07:10 PM, Jan Kiszka wrote: > On 2018-04-07 19:04, Philippe Gerum wrote: >> On 04/07/2018 06:58 PM, Jan Kiszka wrote: >>> On 2018-04-07 18:55, Philippe Gerum wrote: >>>> On 04/07/2018 11:42 AM, Jan Kiszka wrote: >>>>> On 2018-04-07 09:25, Philippe Gerum wrote: >>>>>> On 04/06/2018 05:11 PM, Jan Kiszka wrote: >>>>>>> On 2018-04-06 16:11, Philippe Gerum wrote: >>>>>>>> On 04/06/2018 03:38 PM, Jan Kiszka wrote: >>>>>>>>> On 2018-04-06 08:54, Philippe Gerum wrote: >>>>>>>>>> On 04/05/2018 10:13 PM, Jan Kiszka wrote: >>>>>>>>>>> On 2018-03-27 15:12, Philippe Gerum wrote: >>>>>>>>>>>> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>>>>>>>>>>>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>>>>>>>>>>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>>>>>>>>>>>> repo. >>>>>>>>>>>>> >>>>>>>>>>>>> I started to patch further on this during my flight (wip/4.9 updated), >>>>>>>>>>>>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>>>>>>>>>>>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>>>>>>>>>>>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>>>>>>>>>>>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>>>>>>>>>>>> >>>>>>>>>>>>> If you should look into 4.9-x86 as you indicated, please check this. >>>>>>>>>>>> >>>>>>>>>>>> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >>>>>>>>>>>> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >>>>>>>>>>>> far so good. >>>>>>>>>>> >>>>>>>>>>> Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) >>>>>>>>>>> CPU load. I also triggered this with stable-3.0.x: >>>>>>>>>>> >>>>>>>>>>> [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 >>>>>>>>>>> [ 237.460728] Modules linked in: >>>>>>>>>>> [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 >>>>>>>>>>> [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 >>>>>>>>>>> [ 237.461490] I-pipe domain: Xenomai >>>>>>>>>>> [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 >>>>>>>>>>> [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 >>>>>>>>>>> [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 >>>>>>>>>>> [ 237.461490] Call Trace: >>>>>>>>>>> [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd >>>>>>>>>>> [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 >>>>>>>>>>> [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 >>>>>>>>>>> [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 >>>>>>>>>>> [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 >>>>>>>>>>> [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 >>>>>>>>>>> [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 >>>>>>>>>>> [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 >>>>>>>>>>> [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 >>>>>>>>>>> [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 >>>>>>>>>>> [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 >>>>>>>>>>> [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb >>>>>>>>>>> [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- >>>>>>>>>>> >>>>>>>>>>> I will debug this tomorrow. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I can't reproduce this, the loadavg on my qemu instance consistently >>>>>>>>>> converges to 0.0x figures while running the latency test (10Khz or 1Khz, >>>>>>>>>> same). I'm now running 4.9.92, but I don't think this should make any >>>>>>>>>> difference, since I could trace the box entering the idle state on .90. >>>>>>>>>> >>>>>>>>>> Are you running the ia32emu mode, or x86_64? Also, could you share your >>>>>>>>>> .config for building the guest kernel? >>>>>>>>> >>>>>>>>> Config is the same I sent back then. Userland is 64-bit, compat support >>>>>>>>> enabled. >>>>>>>> >>>>>>>> You only sent me the CONFIG_IDLE* settings I asked for. I'd need the >>>>>>>> whole file now. >>>>>>> >>>>>>> Sorry, though I did. Attached. >>>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> The reason I see so far: xnclock_core_local_shot never sets XNIDLE. >>>>>>>> >>>>>>>> It does here (I traced it). However this should depend on the NO_HZ >>>>>>>> settings, mine are : >>>>>>>> >>>>>>>> CONFIG_TICK_ONESHOT=y >>>>>>>> CONFIG_NO_HZ_COMMON=y >>>>>>>> # CONFIG_HZ_PERIODIC is not set >>>>>>>> CONFIG_NO_HZ_IDLE=y >>>>>>>> CONFIG_NO_HZ=y >>>>>>>> >>>>>>> >>>>>>> Same here. >>>>>>> >>>>>>>>> I >>>>>>>>> suspect we always have a timer registered, that for the host clock. So >>>>>>>> >>>>>>>> In that case, the timer is not idle Xenomai-wise. >>>>>>>> >>>>>>>>> we can't become idle this way. I'm not even sure that this test makes >>>>>>>>> sense because a pending RT timer does not make a non-idle system. >>>>>>>>> >>>>>>>> >>>>>>>> This is not about testing for Cobalt idleness, but for its core timer >>>>>>>> idleness, given that the core timer is shared between both kernels. We >>>>>>>> want to know whether we may allow the regular kernel to shutdown the >>>>>>>> clock event hardware for entering a sleep state. XNIDLE -> XNTIMERIDLE >>>>>>>> if you will. I covered this stuff in Documentation/ipipe.rst lately. >>>>>>>> >>>>>>> >>>>>>> I still don't see the problem. We own the timer, Linux does not program >>>>>>> it. And letting Linux call hlt does not disturb the timer programming, >>>>>>> in most cases at least (there might be some weird old broken hardware). >>>>>> >>>>>> >>>>>> The problem is not with hlt, but with the tick device switch when c3stop >>>>>> is enabled on the device, and going idle means shutting it down before >>>>>> switching to a broadcast device. Very unfortunately, this is not even an >>>>>> x86-specific issue, this may also happen elsewhere, e.g. ARM's TWD. >>>>>> >>>>> >>>>> So, we are talking about systems where CLOCK_EVT_FEAT_C3STOP is set on >>>>> those clockevent devices that we take over for Xenomai purposes, right? >>>>> That excludes the vast majority of systems Xenomai runs on. It >>>>> specifically excludes any modern x86 systems which now have ARAT >>>>> support. For the remaining ones, we always recommended to have >>>>> ACPI_PROCESSOR disabled, thus have no need for this new workaround either. >>>>> >>>> >>>> This is not a work around. The fact that linux is not aware that a >>>> second kernel may consider the timer as busy is a general issue, and >>>> asking that second kernel to have its say is not a work around, but a >>>> requirement. >>>> >>>>> Regarding the mentioned ARM system: Is there no equivalent to >>>>> !ACPI_PROCESSOR, i.e. disabling deep sleep states without denying wfi >>>>> completely? We definitely need a much more targeted solution here. Any >>>>> suggestions? >>>>> >>>> >>>> Yes, try to find what does not work in your case. I'll try to reproduce >>>> tomorrow using your Kconfig. >>> >>> I think you are still on the wrong track here: x86 is not at all >>> affected by the issue you see. >> >> You misread me: I'm talking about finding why your config never matches >> the idle state Xenomai-wise. >> >> Please focus on the ARM system where you >>> saw the issue in the first place. >>> >> >> This is a problem which needs a resolution starting from the generic >> code. So let's focus on finding why the x86-specific case does not work >> instead. > ...because your approach is not correct at all. > > The goal must be to prevent to enter that sleep state, which we achieve > on x86 differently - or do not have to worry about anymore. But you hook > into Linux entering idle state in general and prevent that. > Ok, so here is the situation as of now: you say that this approach is stupid, I'm saying that I traced it working on x86 and ARM. I'm also saying that the general issue of linux defining alone what the idle state might be has to be addressed. So please provide an explanation to the bug you see on your end, I'll try to figure out why I can't see that here, and make an informed opinion about whether the current approach is wrong or too limited in logic, which is different. Dropping code as a knee jerk reaction simply because it does not work for your case may not be the best option. -- Philippe. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-07 17:20 ` Philippe Gerum @ 2018-04-07 17:33 ` Jan Kiszka 0 siblings, 0 replies; 24+ messages in thread From: Jan Kiszka @ 2018-04-07 17:33 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-07 19:20, Philippe Gerum wrote: > Ok, so here is the situation as of now: you say that this approach is > stupid, I'm saying that I traced it working on x86 and ARM. I'm also > saying that the general issue of linux defining alone what the idle > state might be has to be addressed. > > So please provide an explanation to the bug you see on your end, I'll > try to figure out why I can't see that here, and make an informed > opinion about whether the current approach is wrong or too limited in > logic, which is different. > > Dropping code as a knee jerk reaction simply because it does not work > for your case may not be the best option. Look, there are a couple of things fishy with this approach: - there is no documentation of a positive impact on x86 (did you measure high latency without this change? On which hardware? With which .config) - there is no reasoning why we suddenly need this since 4.9 (on x86) - it does not address the issue (some timer stopping in deep sleep state) at its root (Linux telling hardware to enter that *specific* sleep state) Again, I cannot talk about ARM as I didn't look on that end yet, but I can tell you for x86 that - if this is really just about C3 vs. the APIC timer - you are hunting a ghost, a non-existing problem. At the same time, the approach caused a regression on x86 that one may only see with proper instrumentation in place (like running in a VM or having a power meter attached to the hardware) because it always prevents idleness there. Jan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://xenomai.org/pipermail/xenomai/attachments/20180407/f3df57c7/attachment.sig> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [Xenomai] 4.9 for x86 issue 2018-04-05 20:13 ` Jan Kiszka 2018-04-06 6:54 ` Philippe Gerum @ 2018-04-06 16:09 ` Jan Kiszka 1 sibling, 0 replies; 24+ messages in thread From: Jan Kiszka @ 2018-04-06 16:09 UTC (permalink / raw) To: Philippe Gerum; +Cc: Xenomai On 2018-04-05 22:13, Jan Kiszka wrote: > On 2018-03-27 15:12, Philippe Gerum wrote: >> On 03/10/2018 11:06 PM, Jan Kiszka wrote: >>> On 2018-03-09 08:51, Jan Kiszka wrote: >>>> 4.9 requires more work, I've pushed the beginning to wip/4.9 in the same >>>> repo. >>> >>> I started to patch further on this during my flight (wip/4.9 updated), >>> noticed that the 4.14-wip queue will need a little bit sysentry tweaking >>> as well (missing 64-bit syscall dispatching), and then had to find 4.9 >>> in a rather unfortunate state /wrt x86-64: CPUs are no longer idling >>> properly. I went back to ipipe-core-4.9.24-x86-2, without a difference. >>> >>> If you should look into 4.9-x86 as you indicated, please check this. >> >> Both issues fixed in 4.9.90/x86 as pushed lately. The result has run >> overnight in 64bit mode, and for a couple of hours in ia32emu mode. So >> far so good. > > Just trying 4.9.90-x86-6 in KVM, and I'm still finding 100% (virtual) > CPU load. I also triggered this with stable-3.0.x: > > [ 237.455846] WARNING: CPU: 0 PID: 1055 at ../kernel/xenomai/posix/timerfd.c:57 timerfd_read+0x2a6/0x350 > [ 237.460728] Modules linked in: > [ 237.461490] CPU: 0 PID: 1055 Comm: sampling-1052 Not tainted 4.9.90+ #11 > [ 237.461490] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014 > [ 237.461490] I-pipe domain: Xenomai > [ 237.461490] ffffc90001b7fdb0 ffffffff8145e395 0000000000000000 0000000000000000 > [ 237.461490] ffffc90001b7fdf0 ffffffff810e7261 000000393d61d170 ffffc900003e6008 > [ 237.461490] 0000000000000003 0000000000000008 00007f513e8c2de8 0000000000026200 > [ 237.461490] Call Trace: > [ 237.461490] [<ffffffff8145e395>] dump_stack+0xb2/0xdd > [ 237.461490] [<ffffffff810e7261>] __warn+0xd1/0xf0 > [ 237.461490] [<ffffffff810e734d>] warn_slowpath_null+0x1d/0x20 > [ 237.461490] [<ffffffff812423b6>] timerfd_read+0x2a6/0x350 > [ 237.461490] [<ffffffff812174ec>] rtdm_fd_read+0x13c/0x3b0 > [ 237.461490] [<ffffffff81220260>] ? CoBaLt_ioctl+0x20/0x20 > [ 237.461490] [<ffffffff8122026e>] CoBaLt_read+0xe/0x10 > [ 237.461490] [<ffffffff81235894>] handle_head_syscall+0x184/0x4b0 > [ 237.461490] [<ffffffff81236288>] ipipe_fastcall_hook+0x18/0x20 > [ 237.461490] [<ffffffff811a9054>] ipipe_handle_syscall+0x64/0x110 > [ 237.461490] [<ffffffff81002b33>] do_syscall_64+0x43/0x1c0 > [ 237.461490] [<ffffffff81840b43>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb > [ 237.461490] ---[ end trace 9d2476a38b0c5379 ]--- > The WARN_ON is due to missing http://git.xenomai.org/ipipe-jki.git/commit/?h=for-upstream/4.4-update&id=d95805a2f3448cf65e85897cf43f260b9ff3f9d0 in 4.9. I'm preparing a PR. I've also did a stable update for 4.9. The idleness bug can be resolved by dropping ipipe_enter_idle_hook from Xenomai until we have a better logic at hand. If someone wants the current behavior, there is still idle=poll. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2018-04-07 17:33 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <q17d41v98jmvc5qofdfiipjs.1522164993072@embeddedgreg.com>
[not found] ` <289250163.8206221.1522165279222.JavaMail.zimbra@triolcorp.com.ua>
[not found] ` <1221098029.8207596.1522165832579.JavaMail.zimbra@triolcorp.com.ua>
[not found] ` <1320247074.8207767.1522165874048.JavaMail.zimbra@triolcorp.com.ua>
2018-03-28 14:40 ` [Xenomai] 4.9 for x86 issue Шевченко Тарас Григорьевич
2018-03-09 16:51 [Xenomai] [PULL] ipipe 4.4 updates Jan Kiszka
2018-03-10 22:06 ` [Xenomai] 4.9 for x86 issue (was: [PULL] ipipe 4.4 updates) Jan Kiszka
2018-03-19 16:54 ` [Xenomai] 4.9 for x86 issue Philippe Gerum
2018-03-20 3:42 ` Jan Kiszka
2018-03-20 14:22 ` Philippe Gerum
2018-03-19 16:59 ` Philippe Gerum
2018-03-20 15:05 ` Philippe Gerum
2018-03-27 13:12 ` Philippe Gerum
2018-03-27 13:30 ` Шевченко Тарас Григорьевич
2018-03-27 16:42 ` Jan Kiszka
2018-04-05 20:13 ` Jan Kiszka
2018-04-06 6:54 ` Philippe Gerum
2018-04-06 13:38 ` Jan Kiszka
2018-04-06 14:11 ` Philippe Gerum
2018-04-06 15:11 ` Jan Kiszka
2018-04-06 15:52 ` Jan Kiszka
2018-04-07 7:25 ` Philippe Gerum
2018-04-07 9:42 ` Jan Kiszka
2018-04-07 16:55 ` Philippe Gerum
2018-04-07 16:58 ` Jan Kiszka
2018-04-07 17:04 ` Philippe Gerum
2018-04-07 17:10 ` Jan Kiszka
2018-04-07 17:20 ` Philippe Gerum
2018-04-07 17:33 ` Jan Kiszka
2018-04-06 16:09 ` Jan Kiszka
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.