* Stack dump on ipipe 4.9.135-7 x86_64
@ 2018-12-20 8:28 Mauro Salvini
2018-12-20 9:10 ` Jan Kiszka
0 siblings, 1 reply; 5+ messages in thread
From: Mauro Salvini @ 2018-12-20 8:28 UTC (permalink / raw)
To: xenomai
Hi all,
I'm testing Xenomai 3 on an Intel Braswell board (Atom x5-E8000).
I'm using ipipe kernel at last commit from [1], branch ipipe-4.9.y,
64bit build on a Debian Stretch 9.6 64bit.
Xenomai library is from [2], branch stable/v3.0.x on commit bc53d03f (I
haven't two last commits but seems not related). I tried both 32bit
(mixed ABI) and 64bit builds with same following result.
I launch:
xeno-test -l "dohell -s xxx -p yyy -m xxx 90000" -T 90000
After a variable time (from minutes to hours) from latency test start I
get a few overruns that I discovered are generated by a kernel stack
dump (attached to mail the dmesg tail). Latency test doesn't stop, and
after this stackdump never reports other overruns or latency peaks
(seems I need to reboot to reproduce stack).
I read in this mailing list that on last patches much work has done on
FPU part, should it be related?
Glad to give other infos if you need.
Thanks in advance, regards.
Mauro
[1] https://gitlab.denx.de/Xenomai/ipipe
[2] https://gitlab.denx.de/Xenomai/xenomai
-------------- next part --------------
[ 233.205940] /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43: Warning: Linux is compiled to use FPU in kernel-space.
For this reason, switchtest can not test using FPU in Linux kernel-space.
[ 295.660454] ------------[ cut here ]------------
[ 295.660461] WARNING: CPU: 0 PID: 139 at /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/include/asm/fpu/internal.h:502 xnarch_leave_root+0x1a4/0x1b0
[ 295.660465] Modules linked in: binfmt_misc msr iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw snd_pcm gf128mul glue_helper i915 ablk_helper snd_timer cryptd snd soundcore pcspkr drm_kms_helper drm evdev lpc_ich mfd_core fb_sys_fops syscopyarea sysfillrect sysimgblt sg shpchp video button ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel ahci igb i2c_i801 libahci i2c_smbus i2c_algo_bit xhci_pci dca ptp pps_core libata xhci_hcd sdhci_pci sdhci usbcore scsi_mod mmc_core fjes [last unloaded: rtnet]
[ 295.660660] CPU: 0 PID: 139 Comm: jbd2/sda1-8 Tainted: G W 4.9.135+ #1
[ 295.660663] Hardware name: Default string Default string/Q7-BW, BIOS V1.20#KW050220A 03/16/2018
[ 295.660666] I-pipe domain: Xenomai
[ 295.660668] 0000000000000000 ffffffff823402c2 0000000000000000 0000000000000000
[ 295.660680] ffffffff829cd400 ffffffff8206cb68 ffff985636908040 ffff985636504080
[ 295.660706] 0000000000099f50 0000000000000000 ffffbc1580563040 ffff98567ac9b940
[ 295.660718] Call Trace:
[ 295.660721] [<ffffffff823402c2>] ? dump_stack+0xb5/0xd3
[ 295.660724] [<ffffffff8206cb68>] ? __warn+0xc8/0xf0
[ 295.660727] [<ffffffff82068734>] ? xnarch_leave_root+0x1a4/0x1b0
[ 295.660729] [<ffffffff82173b56>] ? ___xnsched_run+0x3f6/0x4b0
[ 295.660732] [<ffffffff8219fbf6>] ? timerfd_handler+0x36/0x50
[ 295.660735] [<ffffffff8217b075>] ? xntimerq_insert+0x5/0xa0
[ 295.660738] [<ffffffff8216a7c9>] ? xnclock_tick+0x1a9/0x2c0
[ 295.660740] [<ffffffff8216cd8a>] ? xnintr_core_clock_handler+0x2fa/0x310
[ 295.660743] [<ffffffff82119b1a>] ? dispatch_irq_head+0x8a/0x120
[ 295.660746] [<ffffffff8203ad75>] ? __ipipe_handle_irq+0x85/0x1b0
[ 295.660749] [<ffffffff82605ba9>] ? apic_timer_interrupt+0x89/0xb0
[ 295.660752] [<ffffffffc01502a6>] ? continue_block+0x22/0x54 [crc32c_intel]
[ 295.660754] [<ffffffffc0150233>] ? crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel]
[ 295.660757] [<ffffffffc031fa90>] ? jbd2_journal_commit_transaction+0x9e0/0x1850 [jbd2]
[ 295.660760] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
[ 295.660763] [<ffffffff820d19af>] ? try_to_del_timer_sync+0x4f/0x80
[ 295.660766] [<ffffffffc0324c92>] ? kjournald2+0xe2/0x290 [jbd2]
[ 295.660768] [<ffffffff820b0320>] ? wake_up_atomic_t+0x30/0x30
[ 295.660771] [<ffffffffc0324bb0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 295.660774] [<ffffffff8208df65>] ? kthread+0xf5/0x110
[ 295.660776] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
[ 295.660779] [<ffffffff8208de70>] ? kthread_park+0x60/0x60
[ 295.660782] [<ffffffff82603de5>] ? ret_from_fork+0x55/0x60
[ 295.660785] ---[ end trace b1bfa97fc17a203c ]---
[ 705.768058] perf: interrupt took too long (2543 > 2500), lowering kernel.perf_event_max_sample_rate to 78500
[ 865.772646] perf: interrupt took too long (3207 > 3178), lowering kernel.perf_event_max_sample_rate to 62250
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack dump on ipipe 4.9.135-7 x86_64
2018-12-20 8:28 Stack dump on ipipe 4.9.135-7 x86_64 Mauro Salvini
@ 2018-12-20 9:10 ` Jan Kiszka
2018-12-21 13:51 ` Henning Schild
0 siblings, 1 reply; 5+ messages in thread
From: Jan Kiszka @ 2018-12-20 9:10 UTC (permalink / raw)
To: Mauro Salvini, xenomai, Henning Schild
On 20.12.18 09:28, Mauro Salvini via Xenomai wrote:
> Hi all,
>
> I'm testing Xenomai 3 on an Intel Braswell board (Atom x5-E8000).
> I'm using ipipe kernel at last commit from [1], branch ipipe-4.9.y,
> 64bit build on a Debian Stretch 9.6 64bit.
> Xenomai library is from [2], branch stable/v3.0.x on commit bc53d03f (I
> haven't two last commits but seems not related). I tried both 32bit
> (mixed ABI) and 64bit builds with same following result.
>
> I launch:
>
> xeno-test -l "dohell -s xxx -p yyy -m xxx 90000" -T 90000
>
> After a variable time (from minutes to hours) from latency test start I
> get a few overruns that I discovered are generated by a kernel stack
> dump (attached to mail the dmesg tail). Latency test doesn't stop, and
> after this stackdump never reports other overruns or latency peaks
> (seems I need to reboot to reproduce stack).
>
> I read in this mailing list that on last patches much work has done on
> FPU part, should it be related?
>
> Glad to give other infos if you need.
Thanks for reporting. Maybe we need your config later on, but I'm first of all
looking at the code, see below.
In general, it's better to run the kernel with frame-pointers enabled to get
more reliable backtraces, at least when an error occurs.
>
> Thanks in advance, regards.
>
> Mauro
>
> [1] https://gitlab.denx.de/Xenomai/ipipe
> [2] https://gitlab.denx.de/Xenomai/xenomai
> -------------- next part --------------
> [ 233.205940] /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43: Warning: Linux is compiled to use FPU in kernel-space.
> For this reason, switchtest can not test using FPU in Linux kernel-space.
> [ 295.660454] ------------[ cut here ]------------
> [ 295.660461] WARNING: CPU: 0 PID: 139 at /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/include/asm/fpu/internal.h:502 xnarch_leave_root+0x1a4/0x1b0
Henning, the kernel checks fpu->fpregs_active here and finds it off while
Xenomai looks at fpu.fpstate_active - intentionally or by accident?
Jan
> [ 295.660465] Modules linked in: binfmt_misc msr iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw snd_pcm gf128mul glue_helper i915 ablk_helper snd_timer cryptd snd soundcore pcspkr drm_kms_helper drm evdev lpc_ich mfd_core fb_sys_fops syscopyarea sysfillrect sysimgblt sg shpchp video button ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel ahci igb i2c_i801 libahci i2c_smbus i2c_algo_bit xhci_pci dca ptp pps_core libata xhci_hcd sdhci_pci sdhci usbcore scsi_mod mmc_core fjes [last unloaded: rtnet]
> [ 295.660660] CPU: 0 PID: 139 Comm: jbd2/sda1-8 Tainted: G W 4.9.135+ #1
> [ 295.660663] Hardware name: Default string Default string/Q7-BW, BIOS V1.20#KW050220A 03/16/2018
> [ 295.660666] I-pipe domain: Xenomai
> [ 295.660668] 0000000000000000 ffffffff823402c2 0000000000000000 0000000000000000
> [ 295.660680] ffffffff829cd400 ffffffff8206cb68 ffff985636908040 ffff985636504080
> [ 295.660706] 0000000000099f50 0000000000000000 ffffbc1580563040 ffff98567ac9b940
> [ 295.660718] Call Trace:
> [ 295.660721] [<ffffffff823402c2>] ? dump_stack+0xb5/0xd3
> [ 295.660724] [<ffffffff8206cb68>] ? __warn+0xc8/0xf0
> [ 295.660727] [<ffffffff82068734>] ? xnarch_leave_root+0x1a4/0x1b0
> [ 295.660729] [<ffffffff82173b56>] ? ___xnsched_run+0x3f6/0x4b0
> [ 295.660732] [<ffffffff8219fbf6>] ? timerfd_handler+0x36/0x50
> [ 295.660735] [<ffffffff8217b075>] ? xntimerq_insert+0x5/0xa0
> [ 295.660738] [<ffffffff8216a7c9>] ? xnclock_tick+0x1a9/0x2c0
> [ 295.660740] [<ffffffff8216cd8a>] ? xnintr_core_clock_handler+0x2fa/0x310
> [ 295.660743] [<ffffffff82119b1a>] ? dispatch_irq_head+0x8a/0x120
> [ 295.660746] [<ffffffff8203ad75>] ? __ipipe_handle_irq+0x85/0x1b0
> [ 295.660749] [<ffffffff82605ba9>] ? apic_timer_interrupt+0x89/0xb0
> [ 295.660752] [<ffffffffc01502a6>] ? continue_block+0x22/0x54 [crc32c_intel]
> [ 295.660754] [<ffffffffc0150233>] ? crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel]
> [ 295.660757] [<ffffffffc031fa90>] ? jbd2_journal_commit_transaction+0x9e0/0x1850 [jbd2]
> [ 295.660760] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
> [ 295.660763] [<ffffffff820d19af>] ? try_to_del_timer_sync+0x4f/0x80
> [ 295.660766] [<ffffffffc0324c92>] ? kjournald2+0xe2/0x290 [jbd2]
> [ 295.660768] [<ffffffff820b0320>] ? wake_up_atomic_t+0x30/0x30
> [ 295.660771] [<ffffffffc0324bb0>] ? commit_timeout+0x10/0x10 [jbd2]
> [ 295.660774] [<ffffffff8208df65>] ? kthread+0xf5/0x110
> [ 295.660776] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
> [ 295.660779] [<ffffffff8208de70>] ? kthread_park+0x60/0x60
> [ 295.660782] [<ffffffff82603de5>] ? ret_from_fork+0x55/0x60
> [ 295.660785] ---[ end trace b1bfa97fc17a203c ]---
> [ 705.768058] perf: interrupt took too long (2543 > 2500), lowering kernel.perf_event_max_sample_rate to 78500
> [ 865.772646] perf: interrupt took too long (3207 > 3178), lowering kernel.perf_event_max_sample_rate to 62250
>
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack dump on ipipe 4.9.135-7 x86_64
2018-12-20 9:10 ` Jan Kiszka
@ 2018-12-21 13:51 ` Henning Schild
2018-12-21 15:11 ` Mauro Salvini
0 siblings, 1 reply; 5+ messages in thread
From: Henning Schild @ 2018-12-21 13:51 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Mauro Salvini, xenomai
Am Thu, 20 Dec 2018 10:10:29 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 20.12.18 09:28, Mauro Salvini via Xenomai wrote:
> > Hi all,
> >
> > I'm testing Xenomai 3 on an Intel Braswell board (Atom x5-E8000).
> > I'm using ipipe kernel at last commit from [1], branch ipipe-4.9.y,
> > 64bit build on a Debian Stretch 9.6 64bit.
> > Xenomai library is from [2], branch stable/v3.0.x on commit
> > bc53d03f (I haven't two last commits but seems not related). I
> > tried both 32bit (mixed ABI) and 64bit builds with same following
> > result.
> >
> > I launch:
> >
> > xeno-test -l "dohell -s xxx -p yyy -m xxx 90000" -T 90000
> >
> > After a variable time (from minutes to hours) from latency test
> > start I get a few overruns that I discovered are generated by a
> > kernel stack dump (attached to mail the dmesg tail). Latency test
> > doesn't stop, and after this stackdump never reports other overruns
> > or latency peaks (seems I need to reboot to reproduce stack).
> >
> > I read in this mailing list that on last patches much work has done
> > on FPU part, should it be related?
> >
> > Glad to give other infos if you need.
>
> Thanks for reporting. Maybe we need your config later on, but I'm
> first of all looking at the code, see below.
>
> In general, it's better to run the kernel with frame-pointers enabled
> to get more reliable backtraces, at least when an error occurs.
>
> >
> > Thanks in advance, regards.
> >
> > Mauro
> >
> > [1] https://gitlab.denx.de/Xenomai/ipipe
> > [2] https://gitlab.denx.de/Xenomai/xenomai
> > -------------- next part --------------
> > [ 233.205940] /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43:
> > Warning: Linux is compiled to use FPU in kernel-space. For this
> > reason, switchtest can not test using FPU in Linux kernel-space.
> > [ 295.660454] ------------[ cut here ]------------ [ 295.660461]
> > WARNING: CPU: 0 PID: 139
> > at /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/include/asm/fpu/internal.h:502
> > xnarch_leave_root+0x1a4/0x1b0
>
> Henning, the kernel checks fpu->fpregs_active here and finds it off
> while Xenomai looks at fpu.fpstate_active - intentionally or by
> accident?
That was by accident. In eager mode they should mostly be in sync,
unless when playing the nasty tricks ipipe has to play. I guess there
is a path where we tried to "unown" a task that we unowned shortly
before. I did not try to find that path, i just sent a patch.
Mauro, since you can reproduce the problem you can probably tell if the
patch fixes it.
Henning
> Jan
>
> > [ 295.660465] Modules linked in: binfmt_misc msr iTCO_wdt
> > iTCO_vendor_support coretemp kvm_intel kvm irqbypass
> > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
> > aes_x86_64 lrw snd_pcm gf128mul glue_helper i915 ablk_helper
> > snd_timer cryptd snd soundcore pcspkr drm_kms_helper drm evdev
> > lpc_ich mfd_core fb_sys_fops syscopyarea sysfillrect sysimgblt sg
> > shpchp video button ip_tables x_tables autofs4 ext4 crc16 jbd2
> > fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel ahci
> > igb i2c_i801 libahci i2c_smbus i2c_algo_bit xhci_pci dca ptp
> > pps_core libata xhci_hcd sdhci_pci sdhci usbcore scsi_mod mmc_core
> > fjes [last unloaded: rtnet] [ 295.660660] CPU: 0 PID: 139 Comm:
> > jbd2/sda1-8 Tainted: G W 4.9.135+ #1 [ 295.660663]
> > Hardware name: Default string Default string/Q7-BW, BIOS
> > V1.20#KW050220A 03/16/2018 [ 295.660666] I-pipe domain: Xenomai
> > [ 295.660668] 0000000000000000 ffffffff823402c2 0000000000000000
> > 0000000000000000 [ 295.660680] ffffffff829cd400 ffffffff8206cb68
> > ffff985636908040 ffff985636504080 [ 295.660706] 0000000000099f50
> > 0000000000000000 ffffbc1580563040 ffff98567ac9b940 [ 295.660718]
> > Call Trace: [ 295.660721] [<ffffffff823402c2>] ?
> > dump_stack+0xb5/0xd3 [ 295.660724] [<ffffffff8206cb68>] ?
> > __warn+0xc8/0xf0 [ 295.660727] [<ffffffff82068734>] ?
> > xnarch_leave_root+0x1a4/0x1b0 [ 295.660729]
> > [<ffffffff82173b56>] ? ___xnsched_run+0x3f6/0x4b0 [ 295.660732]
> > [<ffffffff8219fbf6>] ? timerfd_handler+0x36/0x50 [ 295.660735]
> > [<ffffffff8217b075>] ? xntimerq_insert+0x5/0xa0 [ 295.660738]
> > [<ffffffff8216a7c9>] ? xnclock_tick+0x1a9/0x2c0 [ 295.660740]
> > [<ffffffff8216cd8a>] ? xnintr_core_clock_handler+0x2fa/0x310
> > [ 295.660743] [<ffffffff82119b1a>] ? dispatch_irq_head+0x8a/0x120
> > [ 295.660746] [<ffffffff8203ad75>] ?
> > __ipipe_handle_irq+0x85/0x1b0 [ 295.660749]
> > [<ffffffff82605ba9>] ? apic_timer_interrupt+0x89/0xb0
> > [ 295.660752] [<ffffffffc01502a6>] ? continue_block+0x22/0x54
> > [crc32c_intel] [ 295.660754] [<ffffffffc0150233>] ?
> > crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel] [ 295.660757]
> > [<ffffffffc031fa90>] ? jbd2_journal_commit_transaction+0x9e0/0x1850
> > [jbd2] [ 295.660760] [<ffffffff82603d60>] ?
> > __switch_to_asm+0x40/0x70 [ 295.660763] [<ffffffff820d19af>] ?
> > try_to_del_timer_sync+0x4f/0x80 [ 295.660766]
> > [<ffffffffc0324c92>] ? kjournald2+0xe2/0x290 [jbd2] [ 295.660768]
> > [<ffffffff820b0320>] ? wake_up_atomic_t+0x30/0x30 [ 295.660771]
> > [<ffffffffc0324bb0>] ? commit_timeout+0x10/0x10 [jbd2]
> > [ 295.660774] [<ffffffff8208df65>] ? kthread+0xf5/0x110
> > [ 295.660776] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
> > [ 295.660779] [<ffffffff8208de70>] ? kthread_park+0x60/0x60
> > [ 295.660782] [<ffffffff82603de5>] ? ret_from_fork+0x55/0x60
> > [ 295.660785] ---[ end trace b1bfa97fc17a203c ]--- [ 705.768058]
> > perf: interrupt took too long (2543 > 2500), lowering
> > kernel.perf_event_max_sample_rate to 78500 [ 865.772646] perf:
> > interrupt took too long (3207 > 3178), lowering
> > kernel.perf_event_max_sample_rate to 62250
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack dump on ipipe 4.9.135-7 x86_64
2018-12-21 13:51 ` Henning Schild
@ 2018-12-21 15:11 ` Mauro Salvini
2018-12-21 15:59 ` Mauro Salvini
0 siblings, 1 reply; 5+ messages in thread
From: Mauro Salvini @ 2018-12-21 15:11 UTC (permalink / raw)
To: Henning Schild, Jan Kiszka; +Cc: xenomai
On Fri, 2018-12-21 at 14:51 +0100, Henning Schild wrote:
> Am Thu, 20 Dec 2018 10:10:29 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>
> > On 20.12.18 09:28, Mauro Salvini via Xenomai wrote:
> > > Hi all,
> > >
> > > I'm testing Xenomai 3 on an Intel Braswell board (Atom x5-E8000).
> > > I'm using ipipe kernel at last commit from [1], branch ipipe-
> > > 4.9.y,
> > > 64bit build on a Debian Stretch 9.6 64bit.
> > > Xenomai library is from [2], branch stable/v3.0.x on commit
> > > bc53d03f (I haven't two last commits but seems not related). I
> > > tried both 32bit (mixed ABI) and 64bit builds with same following
> > > result.
> > >
> > > I launch:
> > >
> > > xeno-test -l "dohell -s xxx -p yyy -m xxx 90000" -T 90000
> > >
> > > After a variable time (from minutes to hours) from latency test
> > > start I get a few overruns that I discovered are generated by a
> > > kernel stack dump (attached to mail the dmesg tail). Latency test
> > > doesn't stop, and after this stackdump never reports other
> > > overruns
> > > or latency peaks (seems I need to reboot to reproduce stack).
> > >
> > > I read in this mailing list that on last patches much work has
> > > done
> > > on FPU part, should it be related?
> > >
> > > Glad to give other infos if you need.
> >
> > Thanks for reporting. Maybe we need your config later on, but I'm
> > first of all looking at the code, see below.
> >
> > In general, it's better to run the kernel with frame-pointers
> > enabled
> > to get more reliable backtraces, at least when an error occurs.
> >
> > >
> > > Thanks in advance, regards.
> > >
> > > Mauro
> > >
> > > [1] https://gitlab.denx.de/Xenomai/ipipe
> > > [2] https://gitlab.denx.de/Xenomai/xenomai
> > > -------------- next part --------------
> > > [ 233.205940] /home/build-user/develop/linux-ipipe-
> > > 4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43:
> > > Warning: Linux is compiled to use FPU in kernel-space. For this
> > > reason, switchtest can not test using FPU in Linux kernel-space.
> > > [ 295.660454] ------------[ cut here ]------------
> > > [ 295.660461]
> > > WARNING: CPU: 0 PID: 139
> > > at /home/build-user/develop/linux-ipipe-
> > > 4.9.y/arch/x86/include/asm/fpu/internal.h:502
> > > xnarch_leave_root+0x1a4/0x1b0
> >
> > Henning, the kernel checks fpu->fpregs_active here and finds it off
> > while Xenomai looks at fpu.fpstate_active - intentionally or by
> > accident?
>
> That was by accident. In eager mode they should mostly be in sync,
> unless when playing the nasty tricks ipipe has to play. I guess there
> is a path where we tried to "unown" a task that we unowned shortly
> before. I did not try to find that path, i just sent a patch.
>
> Mauro, since you can reproduce the problem you can probably tell if
> the
> patch fixes it.
Hi Henning,
I tried the patch but result is the same (see attached stackdump).
I attached also my config if could help.
Now I'm compiling a kernel with frame-pointers enabled (as suggested by
Jan).
Thanks in advance
Mauro
>
> Henning
>
> > Jan
> >
> > > [ 295.660465] Modules linked in: binfmt_misc msr iTCO_wdt
> > > iTCO_vendor_support coretemp kvm_intel kvm irqbypass
> > > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
> > > aes_x86_64 lrw snd_pcm gf128mul glue_helper i915 ablk_helper
> > > snd_timer cryptd snd soundcore pcspkr drm_kms_helper drm evdev
> > > lpc_ich mfd_core fb_sys_fops syscopyarea sysfillrect sysimgblt sg
> > > shpchp video button ip_tables x_tables autofs4 ext4 crc16 jbd2
> > > fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel ahci
> > > igb i2c_i801 libahci i2c_smbus i2c_algo_bit xhci_pci dca ptp
> > > pps_core libata xhci_hcd sdhci_pci sdhci usbcore scsi_mod
> > > mmc_core
> > > fjes [last unloaded: rtnet] [ 295.660660] CPU: 0 PID: 139 Comm:
> > > jbd2/sda1-8 Tainted: G W 4.9.135+ #1 [ 295.660663]
> > > Hardware name: Default string Default string/Q7-BW, BIOS
> > > V1.20#KW050220A 03/16/2018 [ 295.660666] I-pipe domain: Xenomai
> > > [ 295.660668] 0000000000000000 ffffffff823402c2
> > > 0000000000000000
> > > 0000000000000000 [ 295.660680] ffffffff829cd400
> > > ffffffff8206cb68
> > > ffff985636908040 ffff985636504080
> > > [ 295.660706] 0000000000099f50
> > > 0000000000000000 ffffbc1580563040 ffff98567ac9b940 [ 295.660718]
> > > Call Trace: [ 295.660721] [<ffffffff823402c2>] ?
> > > dump_stack+0xb5/0xd3 [ 295.660724] [<ffffffff8206cb68>] ?
> > > __warn+0xc8/0xf0 [ 295.660727] [<ffffffff82068734>] ?
> > > xnarch_leave_root+0x1a4/0x1b0 [ 295.660729]
> > > [<ffffffff82173b56>] ? ___xnsched_run+0x3f6/0x4b0 [ 295.660732]
> > > [<ffffffff8219fbf6>] ? timerfd_handler+0x36/0x50 [ 295.660735]
> > > [<ffffffff8217b075>] ? xntimerq_insert+0x5/0xa0 [ 295.660738]
> > > [<ffffffff8216a7c9>] ? xnclock_tick+0x1a9/0x2c0 [ 295.660740]
> > > [<ffffffff8216cd8a>] ? xnintr_core_clock_handler+0x2fa/0x310
> > > [ 295.660743] [<ffffffff82119b1a>] ?
> > > dispatch_irq_head+0x8a/0x120
> > > [ 295.660746] [<ffffffff8203ad75>] ?
> > > __ipipe_handle_irq+0x85/0x1b0 [ 295.660749]
> > > [<ffffffff82605ba9>] ? apic_timer_interrupt+0x89/0xb0
> > > [ 295.660752] [<ffffffffc01502a6>] ? continue_block+0x22/0x54
> > > [crc32c_intel] [ 295.660754] [<ffffffffc0150233>] ?
> > > crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel] [ 295.660757]
> > > [<ffffffffc031fa90>] ?
> > > jbd2_journal_commit_transaction+0x9e0/0x1850
> > > [jbd2] [ 295.660760] [<ffffffff82603d60>] ?
> > > __switch_to_asm+0x40/0x70 [ 295.660763] [<ffffffff820d19af>] ?
> > > try_to_del_timer_sync+0x4f/0x80 [ 295.660766]
> > > [<ffffffffc0324c92>] ? kjournald2+0xe2/0x290 [jbd2]
> > > [ 295.660768]
> > > [<ffffffff820b0320>] ? wake_up_atomic_t+0x30/0x30 [ 295.660771]
> > > [<ffffffffc0324bb0>] ? commit_timeout+0x10/0x10 [jbd2]
> > > [ 295.660774] [<ffffffff8208df65>] ? kthread+0xf5/0x110
> > > [ 295.660776] [<ffffffff82603d60>] ? __switch_to_asm+0x40/0x70
> > > [ 295.660779] [<ffffffff8208de70>] ? kthread_park+0x60/0x60
> > > [ 295.660782] [<ffffffff82603de5>] ? ret_from_fork+0x55/0x60
> > > [ 295.660785] ---[ end trace b1bfa97fc17a203c ]---
> > > [ 705.768058]
> > > perf: interrupt took too long (2543 > 2500), lowering
> > > kernel.perf_event_max_sample_rate to 78500 [ 865.772646] perf:
> > > interrupt took too long (3207 > 3178), lowering
> > > kernel.perf_event_max_sample_rate to 62250
>
>
--
Cordiali Saluti / Best Regards / Saludos Cordiales / Cordialement / 順頌
商祺 / С уважением / مع اطيب التحيات
Mauro Salvini
Automation - Process Control Engineering Dept.
Direct phone: +39 0345 40894
E-Mail: mauro.salvini@smigroup.net
SMITEC S.p.A.
Viale Vittorio Veneto, 4
I-24016, San Pellegrino Terme (BG)
Phone: +39 0345 40800
Fax: +39 0345 40809
Web-site: www.smitec.it
P Please consider the environment before printing
"The content of this message is SMITEC S.p.A.'s exclusive property.
This
message is therefore addressed exclusively to the person indicated as
addressee. If you are not the addressee please delete this message
from your computer. It is severely forbidden to manipulate,
transmit
to third party, divulge, use the information disclosed in
this message
without SMITEC S.p.A. consent."
-------------- next part --------------
[ 156.915389] /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43: Warning: Linux is compiled to use FPU in kernel-space.
For this reason, switchtest can not test using FPU in Linux kernel-space.
[ 191.356974] ------------[ cut here ]------------
[ 191.356980] WARNING: CPU: 0 PID: 140 at /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/include/asm/fpu/internal.h:502 xnarch_leave_root+0x1a4/0x1b0
[ 191.356983] Modules linked in: msr iTCO_wdt iTCO_vendor_support intel_rapl intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 i915 lrw gf128mul glue_helper snd_pcm ablk_helper snd_timer cryptd intel_cstate snd soundcore pcspkr drm_kms_helper evdev drm fb_sys_fops lpc_ich syscopyarea sg mfd_core sysfillrect sysimgblt shpchp video button ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel xhci_pci igb xhci_hcd i2c_algo_bit ahci dca libahci ptp sdhci_pci i2c_i801 usbcore i2c_smbus pps_core libata sdhci scsi_mod mmc_core fjes [last unloaded: rtnet]
[ 191.357181] CPU: 0 PID: 140 Comm: jbd2/sda1-8 Tainted: G W 4.9.135+ #3
[ 191.357185] Hardware name: Default string Default string/Q7-BW, BIOS V1.20#KW050220A 03/16/2018
[ 191.357187] I-pipe domain: Xenomai
[ 191.357190] 0000000000000000 ffffffff8a3402c2 0000000000000000 0000000000000000
[ 191.357202] ffffffff8a9cd400 ffffffff8a06cb68 ffff97a1f6908040 ffff97a1f6a48f40
[ 191.357213] 0000000000099f50 0000000000000000 ffffb73bc0563040 ffff97a23ac9b940
[ 191.357225] Call Trace:
[ 191.357228] [<ffffffff8a3402c2>] ? dump_stack+0xb5/0xd3
[ 191.357231] [<ffffffff8a06cb68>] ? __warn+0xc8/0xf0
[ 191.357233] [<ffffffff8a068734>] ? xnarch_leave_root+0x1a4/0x1b0
[ 191.357236] [<ffffffff8a173b56>] ? ___xnsched_run+0x3f6/0x4b0
[ 191.357239] [<ffffffff8a19fbf6>] ? timerfd_handler+0x36/0x50
[ 191.357241] [<ffffffff8a17b075>] ? xntimerq_insert+0x5/0xa0
[ 191.357244] [<ffffffff8a16a7c9>] ? xnclock_tick+0x1a9/0x2c0
[ 191.357261] [<ffffffff8a16cd8a>] ? xnintr_core_clock_handler+0x2fa/0x310
[ 191.357264] [<ffffffff8a119b1a>] ? dispatch_irq_head+0x8a/0x120
[ 191.357267] [<ffffffff8a03ad75>] ? __ipipe_handle_irq+0x85/0x1b0
[ 191.357270] [<ffffffff8a605ba9>] ? apic_timer_interrupt+0x89/0xb0
[ 191.357273] [<ffffffffc0266316>] ? crc_array+0x1e/0x1e [crc32c_intel]
[ 191.357275] [<ffffffffc0266233>] ? crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel]
[ 191.357278] [<ffffffffc024aa90>] ? jbd2_journal_commit_transaction+0x9e0/0x1850 [jbd2]
[ 191.357281] [<ffffffff8a603d60>] ? __switch_to_asm+0x40/0x70
[ 191.357284] [<ffffffff8a0d19af>] ? try_to_del_timer_sync+0x4f/0x80
[ 191.357287] [<ffffffffc024fc92>] ? kjournald2+0xe2/0x290 [jbd2]
[ 191.357289] [<ffffffff8a0b0320>] ? wake_up_atomic_t+0x30/0x30
[ 191.357292] [<ffffffffc024fbb0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 191.357295] [<ffffffff8a08df65>] ? kthread+0xf5/0x110
[ 191.357298] [<ffffffff8a603d60>] ? __switch_to_asm+0x40/0x70
[ 191.357300] [<ffffffff8a08de70>] ? kthread_park+0x60/0x60
[ 191.357303] [<ffffffff8a603de5>] ? ret_from_fork+0x55/0x60
[ 191.357306] ---[ end trace 4ff3f5c62c8143d3 ]---
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config
Type: text/x-mpsub
Size: 186495 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20181221/8180c64e/attachment.bin>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack dump on ipipe 4.9.135-7 x86_64
2018-12-21 15:11 ` Mauro Salvini
@ 2018-12-21 15:59 ` Mauro Salvini
0 siblings, 0 replies; 5+ messages in thread
From: Mauro Salvini @ 2018-12-21 15:59 UTC (permalink / raw)
To: Henning Schild, Jan Kiszka; +Cc: xenomai
On Fri, 2018-12-21 at 16:11 +0100, Mauro Salvini via Xenomai wrote:
> On Fri, 2018-12-21 at 14:51 +0100, Henning Schild wrote:
> > Am Thu, 20 Dec 2018 10:10:29 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >
> > > On 20.12.18 09:28, Mauro Salvini via Xenomai wrote:
> > > > Hi all,
> > > >
> > > > I'm testing Xenomai 3 on an Intel Braswell board (Atom x5-
> > > > E8000).
> > > > I'm using ipipe kernel at last commit from [1], branch ipipe-
> > > > 4.9.y,
> > > > 64bit build on a Debian Stretch 9.6 64bit.
> > > > Xenomai library is from [2], branch stable/v3.0.x on commit
> > > > bc53d03f (I haven't two last commits but seems not related). I
> > > > tried both 32bit (mixed ABI) and 64bit builds with same
> > > > following
> > > > result.
> > > >
> > > > I launch:
> > > >
> > > > xeno-test -l "dohell -s xxx -p yyy -m xxx 90000" -T 90000
> > > >
> > > > After a variable time (from minutes to hours) from latency test
> > > > start I get a few overruns that I discovered are generated by a
> > > > kernel stack dump (attached to mail the dmesg tail). Latency
> > > > test
> > > > doesn't stop, and after this stackdump never reports other
> > > > overruns
> > > > or latency peaks (seems I need to reboot to reproduce stack).
> > > >
> > > > I read in this mailing list that on last patches much work has
> > > > done
> > > > on FPU part, should it be related?
> > > >
> > > > Glad to give other infos if you need.
> > >
> > > Thanks for reporting. Maybe we need your config later on, but I'm
> > > first of all looking at the code, see below.
> > >
> > > In general, it's better to run the kernel with frame-pointers
> > > enabled
> > > to get more reliable backtraces, at least when an error occurs.
> > >
> > > >
> > > > Thanks in advance, regards.
> > > >
> > > > Mauro
> > > >
> > > > [1] https://gitlab.denx.de/Xenomai/ipipe
> > > > [2] https://gitlab.denx.de/Xenomai/xenomai
> > > > -------------- next part --------------
> > > > [ 233.205940] /home/build-user/develop/linux-ipipe-
> > > > 4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43:
> > > > Warning: Linux is compiled to use FPU in kernel-space. For this
> > > > reason, switchtest can not test using FPU in Linux kernel-
> > > > space.
> > > > [ 295.660454] ------------[ cut here ]------------
> > > > [ 295.660461]
> > > > WARNING: CPU: 0 PID: 139
> > > > at /home/build-user/develop/linux-ipipe-
> > > > 4.9.y/arch/x86/include/asm/fpu/internal.h:502
> > > > xnarch_leave_root+0x1a4/0x1b0
> > >
> > > Henning, the kernel checks fpu->fpregs_active here and finds it
> > > off
> > > while Xenomai looks at fpu.fpstate_active - intentionally or by
> > > accident?
> >
> > That was by accident. In eager mode they should mostly be in sync,
> > unless when playing the nasty tricks ipipe has to play. I guess
> > there
> > is a path where we tried to "unown" a task that we unowned shortly
> > before. I did not try to find that path, i just sent a patch.
> >
> > Mauro, since you can reproduce the problem you can probably tell if
> > the
> > patch fixes it.
>
> Hi Henning,
>
> I tried the patch but result is the same (see attached stackdump).
>
> I attached also my config if could help.
> Now I'm compiling a kernel with frame-pointers enabled (as suggested
> by
> Jan).
>
Hi all,
attached the stackdump with frame-pointers activated (seems very
similar to the previous, but I checked and CONFIG_FRAME_POINTER is
active in config...did I do some mistake?).
Unfortunately, I will not be able to do other tests next weeks (seasons
holidays), but feel free to give me possible tests to do when I'll come
back.
By the way, best wishes to all.
Mauro
-------------- next part --------------
[ 145.774418] /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/xenomai/include/asm/xenomai/fptest.h:43: Warning: Linux is compiled to use FPU in kernel-space.
For this reason, switchtest can not test using FPU in Linux kernel-space.
[ 181.969937] ------------[ cut here ]------------
[ 181.969944] WARNING: CPU: 0 PID: 139 at /home/build-user/develop/linux-ipipe-4.9.y/arch/x86/include/asm/fpu/internal.h:502 xnarch_leave_root+0x15e/0x1c0
[ 181.969971] Modules linked in: msr intel_rapl intel_powerclamp coretemp kvm_intel iTCO_wdt iTCO_vendor_support kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd intel_cstate snd_pcm i915 snd_timer snd soundcore pcspkr lpc_ich mfd_core evdev drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt shpchp sg video button ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache sd_mod hid_generic usbhid hid crc32c_intel i2c_i801 i2c_smbus ahci libahci igb i2c_algo_bit xhci_pci dca ptp sdhci_pci pps_core libata sdhci xhci_hcd usbcore scsi_mod mmc_core fjes [last unloaded: rtnet]
[ 181.970170] CPU: 0 PID: 139 Comm: jbd2/sda1-8 Tainted: G W 4.9.135+ #4
[ 181.970173] Hardware name: Default string Default string/Q7-BW, BIOS V1.20#KW050220A 03/16/2018
[ 181.970176] I-pipe domain: Xenomai
[ 181.970179] ffff992800b4f998 ffffffff9d959436 0000000000000000 0000000000000000
[ 181.970191] ffffffff9dfcd448 ffff992800b4f9d8 ffffffff9d66fb75 000001f63ac9b940
[ 181.970203] ffff8c40f690a040 ffff8c40f6ae4300 0000000000099f50 0000000000000000
[ 181.970215] Call Trace:
[ 181.970218] [<ffffffff9d959436>] dump_stack+0xbc/0xe6
[ 181.970221] [<ffffffff9d66fb75>] __warn+0xd5/0xf0
[ 181.970224] [<ffffffff9d66fcae>] warn_slowpath_null+0x1e/0x20
[ 181.970227] [<ffffffff9d66b5ae>] xnarch_leave_root+0x15e/0x1c0
[ 181.970229] [<ffffffff9d77eb55>] ___xnsched_run+0x405/0x4d0
[ 181.970232] [<ffffffff9d775347>] ? xnclock_tick+0x1d7/0x2b0
[ 181.970235] [<ffffffff9d7779d7>] xnintr_core_clock_handler+0x307/0x310
[ 181.970238] [<ffffffff9d721bbf>] dispatch_irq_head+0x8f/0x120
[ 181.970241] [<ffffffff9d721f72>] __ipipe_dispatch_irq+0xe2/0x1c0
[ 181.970244] [<ffffffff9d63d1da>] __ipipe_handle_irq+0x8a/0x1b0
[ 181.970261] [<ffffffff9dc33f89>] apic_timer_interrupt+0x89/0xb0
[ 181.970264] [<ffffffffc0506f32>] ? crc_24+0xa/0x1e [crc32c_intel]
[ 181.970267] [<ffffffffc0506233>] ? crc32c_pcl_intel_update+0x53/0x60 [crc32c_intel]
[ 181.970270] [<ffffffff9d912e1c>] ? crypto_shash_update+0x3c/0x110
[ 181.970273] [<ffffffffc0408a90>] ? jbd2_journal_commit_transaction+0x9e0/0x1850 [jbd2]
[ 181.970276] [<ffffffff9d6d75fb>] ? lock_timer_base+0x7b/0xa0
[ 181.970279] [<ffffffffc040dc92>] ? kjournald2+0xe2/0x290 [jbd2]
[ 181.970282] [<ffffffff9d6b5110>] ? wake_up_atomic_t+0x30/0x30
[ 181.970285] [<ffffffffc040dbb0>] ? commit_timeout+0x10/0x10 [jbd2]
[ 181.970288] [<ffffffff9d691f4f>] ? kthread+0xff/0x120
[ 181.970290] [<ffffffff9dc32114>] ? __switch_to_asm+0x34/0x70
[ 181.970293] [<ffffffff9d691e50>] ? kthread_park+0x60/0x60
[ 181.970296] [<ffffffff9dc321a5>] ? ret_from_fork+0x55/0x60
[ 181.970299] ---[ end trace d6e0629d074b0a24 ]---
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-12-21 15:59 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-20 8:28 Stack dump on ipipe 4.9.135-7 x86_64 Mauro Salvini
2018-12-20 9:10 ` Jan Kiszka
2018-12-21 13:51 ` Henning Schild
2018-12-21 15:11 ` Mauro Salvini
2018-12-21 15:59 ` Mauro Salvini
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.