[Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Hi,

I got kernel oops when using rtnet loopback with simple udp socket on
Xenomai 3.0

STEPS:
=========
# lspci -knn | grep -i ethernet -A 3
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet
Connection (2) I219-V [8086:15b8] (rev 31)
        Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2)
I219-V [1043:8672]
        Kernel driver in use: e1000e
        Kernel modules: e1000e

-------------
sudo ifconfig lo down
sudo modprobe rtnet
sudo modprobe rtipv4
sudo modprobe rtudp
sudo modprobe rtpacket
sudo modprobe rt_loopback
sudo /usr/xenomai/sbin/rtifconfig rtlo up 127.0.0.1

# /usr/xenomai/sbin/rtifconfig
rtlo      Medium: Local Loopback
          IP address: 127.0.0.1
          UP LOOPBACK RUNNING  MTU: 1500
-------------
Note: I am not installing e1000e driver, as I am checking on loopback
interface now.

# ./rtnet-udp-server 5600 &
# ./rtnet-udp-client 5800 127.0.0.1 5600

server => establish the UDP connection and wait for some message from client.
client => just send the "hello" message to the server

Note: the same program works when build with normal Linux.

INFO:
=============
# xeno-config --info
Xenomai version: Xenomai/cobalt v3.0.6 -- #5956064 (2018-03-20 12:13:33 +0100)
Linux 4.9.51-x86-64-pintu-xeno3-rtdm #2 SMP Wed Apr 25 16:30:53 JST
2018 x86_64 x86_64 x86_64 GNU/Linux
Kernel parameters: initrd=0:\initrd.img-4.9.51-x86-64-pintu-xeno3-rtdm
root=/dev/disk/by-partlabel/system ro ip=off
I-pipe release #4 detected
Cobalt core 3.0.6 detected
Compiler: gcc version 5.4.0
Build args:

=============


---------------------
[  377.304363]
               *** RTnet for Xenomai v3.0.6 ***

[  377.304366] RTnet: initialising real-time networking
[  377.319430] Intel(R) PRO/1000 Network Driver - version 7.1.9
[  377.319433] Copyright (c) 1999-2006 Intel Corporation.
[  379.632211] initializing loopback...
[  379.632218] RTnet: registered rtlo
[  474.312740] BUG: unable to handle kernel paging request at 00007ffcb25562e8
[  474.312744] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
[  474.312748] PGD 80bccc067
[  474.312749] PUD 80f039067
[  474.312750] PMD 80c364067
[  474.312751] PTE 8000000806e09867

[  474.312754] Oops: 0001 [#1] SMP
[  474.312755] Modules linked in: rt_loopback rtpacket rtudp rt_e1000
rtipv4 rtnet arc4 8021q garp mrp stp llc rfcomm bnep
snd_hda_codec_hdmi nls_iso8859_1 ath10k_pci ath10k_core ath mac80211
intel_rapl eeepc_wmi asus_wmi x86_pkg_temp_thermal sparse_keymap
intel_powerclamp mxm_wmi coretemp kvm irqbypass crct10dif_pclmul
cfg80211 crc32_pclmul ghash_clmulni_intel aesni_intel
snd_hda_codec_realtek aes_x86_64 snd_hda_codec_generic lrw gf128mul
glue_helper snd_hda_intel ablk_helper snd_hda_codec cryptd btusb
snd_hda_core btrtl snd_hwdep mei_me mei shpchp serio_raw hci_uart
btbcm btqca btintel bluetooth wmi acpi_als kfifo_buf industrialio
intel_lpss_acpi intel_lpss mac_hid acpi_pad autofs4 e1000e psmouse ptp
pps_core ahci libahci video i2c_hid pinctrl_sunrisepoint pinctrl_intel
hid fjes
[  474.312806] CPU: 0 PID: 3232 Comm: rtnet-server Not tainted
4.9.51-x86-64-pintu-xeno3-rtdm #2
[  474.312808] Hardware name: SkyLake
[  474.312809] I-pipe domain: Linux
[  474.312810] task: ffff8eaa4d090f00 task.stack: ffffa4cc080d4000
[  474.312811] RIP: 0010:[<ffffffffc0621465>]  [<ffffffffc0621465>]
rt_udp_ioctl+0x45/0x5f [rtudp]
[  474.312814] RSP: 0018:ffffa4cc080d7e20  EFLAGS: 00010246
[  474.312815] RAX: 00007ffcb25562e0 RBX: ffff8eaa4cf52800 RCX: ffff8eaa4cf52840
[  474.312816] RDX: 0000000000000000 RSI: 0000000040100022 RDI: ffff8eaa4cf52800
[  474.312817] RBP: ffffa4cc080d7e20 R08: 00007fa113b57d78 R09: 0000000000000105
[  474.312819] R10: 000000010000aa3a R11: ffff8eaa50e39600 R12: 0000000000000003
[  474.312820] R13: ffff8eaa4d090f00 R14: 000000004cf52800 R15: ffffa4cc03856040
[  474.312821] FS:  00007fa1145b3740(0000) GS:ffff8eaa56200000(0000)
knlGS:0000000000000000
[  474.312823] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  474.312824] CR2: 00007ffcb25562e8 CR3: 000000080f0a9000 CR4: 00000000003406f0
[  474.312825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  474.312826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  474.312827] Stack:
[  474.312828]  ffffa4cc080d7eb0 ffffffffa419a0e7 00007ffcb25562e0
4010002200000003
[  474.312831]  ffff8eaa00000010 ffffa4cc080d7ec0 ffffa4cc080d7e58
ffff8eaa4d090f00
[  474.312834]  ffffffffa419e590 00007ffcb25562e0 ffffa4cc080d7e90
ffffffffa409a6f1
[  474.312836] Call Trace:
[  474.312840]  [<ffffffffa419a0e7>] rtdm_fd_ioctl+0xe7/0x280
[  474.312842]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
[  474.312844]  [<ffffffffa409a6f1>] ? __ipipe_migrate_head+0x51/0xb0
[  474.312846]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
[  474.312848]  [<ffffffffa419e59e>] CoBaLt_ioctl+0xe/0x20
[  474.312850]  [<ffffffffa41ab234>] ipipe_syscall_hook+0x114/0x320
[  474.312853]  [<ffffffffa412f9bf>] __ipipe_notify_syscall+0xbf/0x170
[  474.312855]  [<ffffffffa4080c11>] ? sigprocmask+0x51/0x80
[  474.312858]  [<ffffffffa48fd7b7>] pipeline_syscall+0x8/0x1b
[  474.312859] Code: 00 10 40 48 8d 4f 40 74 22 81 fe 23 00 10 40 75
10 8b 50 08 48 8b 30 48 89 cf e8 b7 fd ff ff 5d c3 48 89 c2 e8 5d 26
fd ff 5d c3 <8b> 50 08 48 8b 30 48 89 cf e8 6d fa ff ff 5d c3 48 89 c2
e8 13
[  474.312890] RIP  [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
[  474.312892]  RSP <ffffa4cc080d7e20>
[  474.312893] CR2: 00007ffcb25562e8
[  474.312895] ---[ end trace b2113f37ad3abd56 ]---
[  489.183060] BUG: unable to handle kernel paging request at 00007fff6fe2a908
[  489.183064] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
[  489.183067] PGD 80b14a067
[  489.183068] PUD 812af3067
[  489.183070] PMD 810e50067
[  489.183070] PTE 8000000805867867

---------------------

I am using some 3 months old xenomai-3.0.
But in latest commits also I could not find any fix for it.

If there is any fix available for it please let me know.



Thanks,
Pintu

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-04-25 10:39, Pintu Kumar wrote:
> Hi,
> 
> I got kernel oops when using rtnet loopback with simple udp socket on
> Xenomai 3.0
> 
> STEPS:
> =========
> # lspci -knn | grep -i ethernet -A 3
> 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet
> Connection (2) I219-V [8086:15b8] (rev 31)
>         Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2)
> I219-V [1043:8672]
>         Kernel driver in use: e1000e
>         Kernel modules: e1000e
> 
> -------------
> sudo ifconfig lo down
> sudo modprobe rtnet
> sudo modprobe rtipv4
> sudo modprobe rtudp
> sudo modprobe rtpacket
> sudo modprobe rt_loopback
> sudo /usr/xenomai/sbin/rtifconfig rtlo up 127.0.0.1
> 
> # /usr/xenomai/sbin/rtifconfig
> rtlo      Medium: Local Loopback
>           IP address: 127.0.0.1
>           UP LOOPBACK RUNNING  MTU: 1500
> -------------
> Note: I am not installing e1000e driver, as I am checking on loopback
> interface now.
> 
> # ./rtnet-udp-server 5600 &
> # ./rtnet-udp-client 5800 127.0.0.1 5600
> 
> server => establish the UDP connection and wait for some message from client.
> client => just send the "hello" message to the server
> 
> Note: the same program works when build with normal Linux.
> 
> INFO:
> =============
> # xeno-config --info
> Xenomai version: Xenomai/cobalt v3.0.6 -- #5956064 (2018-03-20 12:13:33 +0100)
> Linux 4.9.51-x86-64-pintu-xeno3-rtdm #2 SMP Wed Apr 25 16:30:53 JST
> 2018 x86_64 x86_64 x86_64 GNU/Linux
> Kernel parameters: initrd=0:\initrd.img-4.9.51-x86-64-pintu-xeno3-rtdm
> root=/dev/disk/by-partlabel/system ro ip=off
> I-pipe release #4 detected
> Cobalt core 3.0.6 detected
> Compiler: gcc version 5.4.0
> Build args:
> 
> =============
> 
> 
> ---------------------
> [  377.304363]
>                *** RTnet for Xenomai v3.0.6 ***
> 
> [  377.304366] RTnet: initialising real-time networking
> [  377.319430] Intel(R) PRO/1000 Network Driver - version 7.1.9
> [  377.319433] Copyright (c) 1999-2006 Intel Corporation.
> [  379.632211] initializing loopback...
> [  379.632218] RTnet: registered rtlo
> [  474.312740] BUG: unable to handle kernel paging request at 00007ffcb25562e8
> [  474.312744] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
> [  474.312748] PGD 80bccc067
> [  474.312749] PUD 80f039067
> [  474.312750] PMD 80c364067
> [  474.312751] PTE 8000000806e09867
> 
> [  474.312754] Oops: 0001 [#1] SMP
> [  474.312755] Modules linked in: rt_loopback rtpacket rtudp rt_e1000
> rtipv4 rtnet arc4 8021q garp mrp stp llc rfcomm bnep
> snd_hda_codec_hdmi nls_iso8859_1 ath10k_pci ath10k_core ath mac80211
> intel_rapl eeepc_wmi asus_wmi x86_pkg_temp_thermal sparse_keymap
> intel_powerclamp mxm_wmi coretemp kvm irqbypass crct10dif_pclmul
> cfg80211 crc32_pclmul ghash_clmulni_intel aesni_intel
> snd_hda_codec_realtek aes_x86_64 snd_hda_codec_generic lrw gf128mul
> glue_helper snd_hda_intel ablk_helper snd_hda_codec cryptd btusb
> snd_hda_core btrtl snd_hwdep mei_me mei shpchp serio_raw hci_uart
> btbcm btqca btintel bluetooth wmi acpi_als kfifo_buf industrialio
> intel_lpss_acpi intel_lpss mac_hid acpi_pad autofs4 e1000e psmouse ptp
> pps_core ahci libahci video i2c_hid pinctrl_sunrisepoint pinctrl_intel
> hid fjes
> [  474.312806] CPU: 0 PID: 3232 Comm: rtnet-server Not tainted
> 4.9.51-x86-64-pintu-xeno3-rtdm #2
> [  474.312808] Hardware name: SkyLake
> [  474.312809] I-pipe domain: Linux
> [  474.312810] task: ffff8eaa4d090f00 task.stack: ffffa4cc080d4000
> [  474.312811] RIP: 0010:[<ffffffffc0621465>]  [<ffffffffc0621465>]
> rt_udp_ioctl+0x45/0x5f [rtudp]
> [  474.312814] RSP: 0018:ffffa4cc080d7e20  EFLAGS: 00010246
> [  474.312815] RAX: 00007ffcb25562e0 RBX: ffff8eaa4cf52800 RCX: ffff8eaa4cf52840
> [  474.312816] RDX: 0000000000000000 RSI: 0000000040100022 RDI: ffff8eaa4cf52800
> [  474.312817] RBP: ffffa4cc080d7e20 R08: 00007fa113b57d78 R09: 0000000000000105
> [  474.312819] R10: 000000010000aa3a R11: ffff8eaa50e39600 R12: 0000000000000003
> [  474.312820] R13: ffff8eaa4d090f00 R14: 000000004cf52800 R15: ffffa4cc03856040
> [  474.312821] FS:  00007fa1145b3740(0000) GS:ffff8eaa56200000(0000)
> knlGS:0000000000000000
> [  474.312823] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  474.312824] CR2: 00007ffcb25562e8 CR3: 000000080f0a9000 CR4: 00000000003406f0
> [  474.312825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  474.312826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [  474.312827] Stack:
> [  474.312828]  ffffa4cc080d7eb0 ffffffffa419a0e7 00007ffcb25562e0
> 4010002200000003
> [  474.312831]  ffff8eaa00000010 ffffa4cc080d7ec0 ffffa4cc080d7e58
> ffff8eaa4d090f00
> [  474.312834]  ffffffffa419e590 00007ffcb25562e0 ffffa4cc080d7e90
> ffffffffa409a6f1
> [  474.312836] Call Trace:
> [  474.312840]  [<ffffffffa419a0e7>] rtdm_fd_ioctl+0xe7/0x280
> [  474.312842]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
> [  474.312844]  [<ffffffffa409a6f1>] ? __ipipe_migrate_head+0x51/0xb0
> [  474.312846]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
> [  474.312848]  [<ffffffffa419e59e>] CoBaLt_ioctl+0xe/0x20
> [  474.312850]  [<ffffffffa41ab234>] ipipe_syscall_hook+0x114/0x320
> [  474.312853]  [<ffffffffa412f9bf>] __ipipe_notify_syscall+0xbf/0x170
> [  474.312855]  [<ffffffffa4080c11>] ? sigprocmask+0x51/0x80
> [  474.312858]  [<ffffffffa48fd7b7>] pipeline_syscall+0x8/0x1b
> [  474.312859] Code: 00 10 40 48 8d 4f 40 74 22 81 fe 23 00 10 40 75
> 10 8b 50 08 48 8b 30 48 89 cf e8 b7 fd ff ff 5d c3 48 89 c2 e8 5d 26
> fd ff 5d c3 <8b> 50 08 48 8b 30 48 89 cf e8 6d fa ff ff 5d c3 48 89 c2
> e8 13
> [  474.312890] RIP  [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
> [  474.312892]  RSP <ffffa4cc080d7e20>
> [  474.312893] CR2: 00007ffcb25562e8
> [  474.312895] ---[ end trace b2113f37ad3abd56 ]---
> [  489.183060] BUG: unable to handle kernel paging request at 00007fff6fe2a908
> [  489.183064] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
> [  489.183067] PGD 80b14a067
> [  489.183068] PUD 812af3067
> [  489.183070] PMD 810e50067
> [  489.183070] PTE 8000000805867867
> 
> ---------------------
> 
> I am using some 3 months old xenomai-3.0.
> But in latest commits also I could not find any fix for it.
> 
> If there is any fix available for it please let me know.
> 

Likely fixed in stable-3.0.x (upcoming 3.0.7). Please retry with that
git branch.

Jan
-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Dear Jan,

Thank you so much for your reply.
I will try the latest stable version to check again.
Is ipipe patches (linux: 4.9.51) also needs to be upgraded for this
issue? Or only xenomai-3/kernel patches are enough?

Actually, now I am stuck with another question.
Hope if you could help me.

As I said, I applied xenomai-3.0.6, kernel patches (using
prepare_kernel script) to my x86_64 kernel, around 4 months back.
I am using it since then. After that I never upgraded any patches.

Now my concern is, how do I apply/upgrade only the latest patches?
I did not remember the last commit until which I applied the patches.

Is prepare_kernel script in intelligent enough to find the patch
difference, and apply on the latest patches ?

Normally how you people upgrade to the latest xenomai patches.
If you have any suggestions, please guide me.


Thanks,
Pintu



On Wed, Apr 25, 2018 at 3:35 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2018-04-25 10:39, Pintu Kumar wrote:
>> Hi,
>>
>> I got kernel oops when using rtnet loopback with simple udp socket on
>> Xenomai 3.0
>>
>> STEPS:
>> =========
>> # lspci -knn | grep -i ethernet -A 3
>> 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet
>> Connection (2) I219-V [8086:15b8] (rev 31)
>>         Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2)
>> I219-V [1043:8672]
>>         Kernel driver in use: e1000e
>>         Kernel modules: e1000e
>>
>> -------------
>> sudo ifconfig lo down
>> sudo modprobe rtnet
>> sudo modprobe rtipv4
>> sudo modprobe rtudp
>> sudo modprobe rtpacket
>> sudo modprobe rt_loopback
>> sudo /usr/xenomai/sbin/rtifconfig rtlo up 127.0.0.1
>>
>> # /usr/xenomai/sbin/rtifconfig
>> rtlo      Medium: Local Loopback
>>           IP address: 127.0.0.1
>>           UP LOOPBACK RUNNING  MTU: 1500
>> -------------
>> Note: I am not installing e1000e driver, as I am checking on loopback
>> interface now.
>>
>> # ./rtnet-udp-server 5600 &
>> # ./rtnet-udp-client 5800 127.0.0.1 5600
>>
>> server => establish the UDP connection and wait for some message from client.
>> client => just send the "hello" message to the server
>>
>> Note: the same program works when build with normal Linux.
>>
>> INFO:
>> =============
>> # xeno-config --info
>> Xenomai version: Xenomai/cobalt v3.0.6 -- #5956064 (2018-03-20 12:13:33 +0100)
>> Linux 4.9.51-x86-64-pintu-xeno3-rtdm #2 SMP Wed Apr 25 16:30:53 JST
>> 2018 x86_64 x86_64 x86_64 GNU/Linux
>> Kernel parameters: initrd=0:\initrd.img-4.9.51-x86-64-pintu-xeno3-rtdm
>> root=/dev/disk/by-partlabel/system ro ip=off
>> I-pipe release #4 detected
>> Cobalt core 3.0.6 detected
>> Compiler: gcc version 5.4.0
>> Build args:
>>
>> =============
>>
>>
>> ---------------------
>> [  377.304363]
>>                *** RTnet for Xenomai v3.0.6 ***
>>
>> [  377.304366] RTnet: initialising real-time networking
>> [  377.319430] Intel(R) PRO/1000 Network Driver - version 7.1.9
>> [  377.319433] Copyright (c) 1999-2006 Intel Corporation.
>> [  379.632211] initializing loopback...
>> [  379.632218] RTnet: registered rtlo
>> [  474.312740] BUG: unable to handle kernel paging request at 00007ffcb25562e8
>> [  474.312744] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
>> [  474.312748] PGD 80bccc067
>> [  474.312749] PUD 80f039067
>> [  474.312750] PMD 80c364067
>> [  474.312751] PTE 8000000806e09867
>>
>> [  474.312754] Oops: 0001 [#1] SMP
>> [  474.312755] Modules linked in: rt_loopback rtpacket rtudp rt_e1000
>> rtipv4 rtnet arc4 8021q garp mrp stp llc rfcomm bnep
>> snd_hda_codec_hdmi nls_iso8859_1 ath10k_pci ath10k_core ath mac80211
>> intel_rapl eeepc_wmi asus_wmi x86_pkg_temp_thermal sparse_keymap
>> intel_powerclamp mxm_wmi coretemp kvm irqbypass crct10dif_pclmul
>> cfg80211 crc32_pclmul ghash_clmulni_intel aesni_intel
>> snd_hda_codec_realtek aes_x86_64 snd_hda_codec_generic lrw gf128mul
>> glue_helper snd_hda_intel ablk_helper snd_hda_codec cryptd btusb
>> snd_hda_core btrtl snd_hwdep mei_me mei shpchp serio_raw hci_uart
>> btbcm btqca btintel bluetooth wmi acpi_als kfifo_buf industrialio
>> intel_lpss_acpi intel_lpss mac_hid acpi_pad autofs4 e1000e psmouse ptp
>> pps_core ahci libahci video i2c_hid pinctrl_sunrisepoint pinctrl_intel
>> hid fjes
>> [  474.312806] CPU: 0 PID: 3232 Comm: rtnet-server Not tainted
>> 4.9.51-x86-64-pintu-xeno3-rtdm #2
>> [  474.312808] Hardware name: SkyLake
>> [  474.312809] I-pipe domain: Linux
>> [  474.312810] task: ffff8eaa4d090f00 task.stack: ffffa4cc080d4000
>> [  474.312811] RIP: 0010:[<ffffffffc0621465>]  [<ffffffffc0621465>]
>> rt_udp_ioctl+0x45/0x5f [rtudp]
>> [  474.312814] RSP: 0018:ffffa4cc080d7e20  EFLAGS: 00010246
>> [  474.312815] RAX: 00007ffcb25562e0 RBX: ffff8eaa4cf52800 RCX: ffff8eaa4cf52840
>> [  474.312816] RDX: 0000000000000000 RSI: 0000000040100022 RDI: ffff8eaa4cf52800
>> [  474.312817] RBP: ffffa4cc080d7e20 R08: 00007fa113b57d78 R09: 0000000000000105
>> [  474.312819] R10: 000000010000aa3a R11: ffff8eaa50e39600 R12: 0000000000000003
>> [  474.312820] R13: ffff8eaa4d090f00 R14: 000000004cf52800 R15: ffffa4cc03856040
>> [  474.312821] FS:  00007fa1145b3740(0000) GS:ffff8eaa56200000(0000)
>> knlGS:0000000000000000
>> [  474.312823] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [  474.312824] CR2: 00007ffcb25562e8 CR3: 000000080f0a9000 CR4: 00000000003406f0
>> [  474.312825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [  474.312826] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [  474.312827] Stack:
>> [  474.312828]  ffffa4cc080d7eb0 ffffffffa419a0e7 00007ffcb25562e0
>> 4010002200000003
>> [  474.312831]  ffff8eaa00000010 ffffa4cc080d7ec0 ffffa4cc080d7e58
>> ffff8eaa4d090f00
>> [  474.312834]  ffffffffa419e590 00007ffcb25562e0 ffffa4cc080d7e90
>> ffffffffa409a6f1
>> [  474.312836] Call Trace:
>> [  474.312840]  [<ffffffffa419a0e7>] rtdm_fd_ioctl+0xe7/0x280
>> [  474.312842]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
>> [  474.312844]  [<ffffffffa409a6f1>] ? __ipipe_migrate_head+0x51/0xb0
>> [  474.312846]  [<ffffffffa419e590>] ? CoBaLt_fcntl+0x20/0x20
>> [  474.312848]  [<ffffffffa419e59e>] CoBaLt_ioctl+0xe/0x20
>> [  474.312850]  [<ffffffffa41ab234>] ipipe_syscall_hook+0x114/0x320
>> [  474.312853]  [<ffffffffa412f9bf>] __ipipe_notify_syscall+0xbf/0x170
>> [  474.312855]  [<ffffffffa4080c11>] ? sigprocmask+0x51/0x80
>> [  474.312858]  [<ffffffffa48fd7b7>] pipeline_syscall+0x8/0x1b
>> [  474.312859] Code: 00 10 40 48 8d 4f 40 74 22 81 fe 23 00 10 40 75
>> 10 8b 50 08 48 8b 30 48 89 cf e8 b7 fd ff ff 5d c3 48 89 c2 e8 5d 26
>> fd ff 5d c3 <8b> 50 08 48 8b 30 48 89 cf e8 6d fa ff ff 5d c3 48 89 c2
>> e8 13
>> [  474.312890] RIP  [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
>> [  474.312892]  RSP <ffffa4cc080d7e20>
>> [  474.312893] CR2: 00007ffcb25562e8
>> [  474.312895] ---[ end trace b2113f37ad3abd56 ]---
>> [  489.183060] BUG: unable to handle kernel paging request at 00007fff6fe2a908
>> [  489.183064] IP: [<ffffffffc0621465>] rt_udp_ioctl+0x45/0x5f [rtudp]
>> [  489.183067] PGD 80b14a067
>> [  489.183068] PUD 812af3067
>> [  489.183070] PMD 810e50067
>> [  489.183070] PTE 8000000805867867
>>
>> ---------------------
>>
>> I am using some 3 months old xenomai-3.0.
>> But in latest commits also I could not find any fix for it.
>>
>> If there is any fix available for it please let me know.
>>
>
> Likely fixed in stable-3.0.x (upcoming 3.0.7). Please retry with that
> git branch.
>
> Jan
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-04-25 13:36, Pintu Kumar wrote:
> Dear Jan,
> 
> Thank you so much for your reply.
> I will try the latest stable version to check again.
> Is ipipe patches (linux: 4.9.51) also needs to be upgraded for this
> issue? Or only xenomai-3/kernel patches are enough?

This particular issue was addressed in the Xenomai core, not the I-pipe
patch.

> 
> Actually, now I am stuck with another question.
> Hope if you could help me.
> 
> As I said, I applied xenomai-3.0.6, kernel patches (using
> prepare_kernel script) to my x86_64 kernel, around 4 months back.
> I am using it since then. After that I never upgraded any patches.
> 
> Now my concern is, how do I apply/upgrade only the latest patches?
> I did not remember the last commit until which I applied the patches.
> 
> Is prepare_kernel script in intelligent enough to find the patch
> difference, and apply on the latest patches ?
> 
> Normally how you people upgrade to the latest xenomai patches.
> If you have any suggestions, please guide me.

Well, best practice is versioning control and build automation. You can
pull the patch into your local kernel or use the i-pipe kernel git tree
as feed (if you have no own patches). Then you just need
prepare-kernel.sh to refresh the xenomai core.

And building everything can also be automated by scripts or more
advanced systems that also track the source revisions for you. Can be
git, can be something like yocto, buildroot etc.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Hi,
I upgraded to xenomai-3-next branch for x86, but still rtnet loopback
is crashing for me.
The xenomai kernel is used from 4.9.51 until
commit: 10605b427b1408cdc6926f7c25d4a4eda527da8d
    Author: Philippe Gerum <rpm@xenomai.org>
    Date:   Mon Mar 26 09:17:02 2018 +0200

        ipipe-core-4.9.51-x86-5

Is the rtnet loopback working with 4.9 ?
Please let me know if this issue is fixed already ?

This is the kernel logs:
------------------------------------------
[612871.612307]
                *** RTnet for Xenomai v3.1-devel ***

[612871.612310] RTnet: initialising real-time networking
[612906.855980] initializing loopback...
[612906.855998] RTnet: registered rtlo
[613075.162006] BUG: unable to handle kernel paging request at 00007ffc7893b148
[613075.162009] IP: [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
[613075.162013] PGD 744f76067
[613075.162014] PUD 80e1ae067
[613075.162015] PMD 6893dd067
[613075.162015] PTE 8000000581853867

[613075.162018] Oops: 0001 [#1] SMP
[613075.162019] Modules linked in: rt_loopback rtpacket rtudp rtipv4
rtnet binfmt_misc 8021q garp mrp stp llc rfcomm bnep
snd_hda_codec_hdmi nls_iso8859_1 eeepc_wmi intel_rapl btusb asus_wmi
x86_pkg_temp_thermal btrtl sparse_keymap intel_powerclamp coretemp
ath10k_pci kvm ath10k_core irqbypass ath mac80211 crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_realtek
aes_x86_64 snd_hda_codec_generic lrw cfg80211 gf128mul glue_helper
ablk_helper snd_hda_intel cryptd snd_hda_codec snd_hda_core input_leds
snd_hwdep mei_me mei shpchp serio_raw hci_uart btbcm btqca btintel
bluetooth acpi_als kfifo_buf intel_lpss_acpi industrialio intel_lpss
mac_hid acpi_pad autofs4 hid_generic usbhid nouveau mxm_wmi ttm
psmouse e1000e ptp pps_core ahci libahci i2c_hid pinctrl_sunrisepoint
wmi hid video
[613075.162049]  pinctrl_intel fjes
[613075.162052] CPU: 0 PID: 12658 Comm: rtnet-client Not tainted
4.9.51-amd-x86-64-pintu-xeno3-rtdm #4
[613075.162053] Hardware name: System manufacturer System Product Name/xxx
[613075.162054] I-pipe domain: Linux
[613075.162055] task: ffff903e8921da00 task.stack: ffff9e2f03348000
[613075.162056] RIP: 0010:[<ffffffffc0a0503e>]  [<ffffffffc0a0503e>]
rt_udp_getfrag+0x3e/0x110 [rtudp]
[613075.162058] RSP: 0018:ffff9e2f0334bb70  EFLAGS: 00010202
[613075.162059] RAX: 0000000000000000 RBX: 000000000000000a RCX:
00007ffc7893b140
[613075.162060] RDX: 0000000000000000 RSI: ffff903e88e431e4 RDI:
ffff9e2f0334bc40
[613075.162061] RBP: ffff9e2f0334bb90 R08: ffff9e2f0334bdb8 R09:
0000000000000000
[613075.162062] R10: ffff903e88e43100 R11: ffff90400c1c8800 R12:
ffff903e88e431e4
[613075.162063] R13: 0000000000000001 R14: ffff9e2f0334bc40 R15:
000000000000001e
[613075.162064] FS:  00007f310b789740(0000) GS:ffff904016200000(0000)
knlGS:0000000000000000
[613075.162065] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[613075.162066] CR2: 00007ffc7893b148 CR3: 0000000744e5b000 CR4:
00000000003406f0
[613075.162067] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[613075.162067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[613075.162068] Stack:
[613075.162069]  ffff9040121bec00 ffff90400e262240 ffff9e2f0334bdb8
000000000000000a
[613075.162071]  ffff9e2f0334bc00 ffffffffc0a0f204 ffff903dd1a84500
ffff9e2f0334bc10
[613075.162074]  0000000000033768 ffff9e2f0334bc40 ffffffffc0a05000
0000000f00000000
[613075.162076] Call Trace:
[613075.162091]  [<ffffffffc0a0f204>] rt_ip_build_xmit+0x1c4/0x2a0 [rtipv4]
[613075.162093]  [<ffffffffc0a05000>] ? 0xffffffffc0a05000
[613075.162094]  [<ffffffffc0a060db>] rt_udp_sendmsg+0x37b/0x3d0 [rtudp]
[613075.162097]  [<ffffffffa70a4676>] ? update_curr+0x66/0x180
[613075.162098]  [<ffffffffa70a67b8>] ? dequeue_entity+0x268/0xc00
[613075.162100]  [<ffffffffa718a117>] ? ___xnsched_run.part.73+0x3d7/0x400
[613075.162102]  [<ffffffffa70a4035>] ? hrtick_update+0x5/0x70
[613075.162103]  [<ffffffffa70a76d7>] ? dequeue_task_fair+0x587/0x900
[613075.162105]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
[613075.162106]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
[613075.162108]  [<ffffffffa719ad7b>] rtdm_fd_sendmsg+0xcb/0x240
[613075.162109]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
[613075.162111]  [<ffffffffa71a03de>] CoBaLt_sendmsg+0x4e/0x70
[613075.162113]  [<ffffffffa71ad017>] ipipe_syscall_hook+0x117/0x340
[613075.162114]  [<ffffffffa712f9ff>] __ipipe_notify_syscall+0xbf/0x170
[613075.162116]  [<ffffffffa79007b7>] pipeline_syscall+0x8/0x1b
[613075.162118] Code: 54 49 89 f4 53 89 cb 8b 57 20 0f 85 c3 00 00 00
85 d2 7e 2f 8b 47 24 45 31 ed 49 63 cd 89 c2 41 83 c5 01 48 c1 e1 04
49 03 4e 18 <8b> 71 08 48 8b 39 e8 e7 f4 a0 e6 41 8b 56 20 41 89 46 24
44 39
[613075.162141] RIP  [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
[613075.162143]  RSP <ffff9e2f0334bb70>
[613075.162144] CR2: 00007ffc7893b148
[613075.162145] ---[ end trace 90458bf1f92e3557 ]---
On Thu, Apr 26, 2018 at 9:53 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2018-04-25 13:36, Pintu Kumar wrote:
> > Dear Jan,
> >
> > Thank you so much for your reply.
> > I will try the latest stable version to check again.
> > Is ipipe patches (linux: 4.9.51) also needs to be upgraded for this
> > issue? Or only xenomai-3/kernel patches are enough?
>
> This particular issue was addressed in the Xenomai core, not the I-pipe
> patch.
>
> >
> > Actually, now I am stuck with another question.
> > Hope if you could help me.
> >
> > As I said, I applied xenomai-3.0.6, kernel patches (using
> > prepare_kernel script) to my x86_64 kernel, around 4 months back.
> > I am using it since then. After that I never upgraded any patches.
> >
> > Now my concern is, how do I apply/upgrade only the latest patches?
> > I did not remember the last commit until which I applied the patches.
> >
> > Is prepare_kernel script in intelligent enough to find the patch
> > difference, and apply on the latest patches ?
> >
> > Normally how you people upgrade to the latest xenomai patches.
> > If you have any suggestions, please guide me.
>
> Well, best practice is versioning control and build automation. You can
> pull the patch into your local kernel or use the i-pipe kernel git tree
> as feed (if you have no own patches). Then you just need
> prepare-kernel.sh to refresh the xenomai core.
>
> And building everything can also be automated by scripts or more
> advanced systems that also track the source revisions for you. Can be
> git, can be something like yocto, buildroot etc.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Hi,

Can someone help me with this issue.
I compared the xenomai-3 next repo
(next/kernel/drivers/net/stack/ipv4/udp/udp.c) and the changes are
almost same.
Now I am stuck with this..
Please help!

Is there any test available for rtnet loopback ?

Regards,
Pintu
On Tue, Jun 19, 2018 at 7:18 PM Pintu Kumar <pintu.ping@gmail.com> wrote:
>
> Hi,
> I upgraded to xenomai-3-next branch for x86, but still rtnet loopback
> is crashing for me.
> The xenomai kernel is used from 4.9.51 until
> commit: 10605b427b1408cdc6926f7c25d4a4eda527da8d
>     Author: Philippe Gerum <rpm@xenomai.org>
>     Date:   Mon Mar 26 09:17:02 2018 +0200
>
>         ipipe-core-4.9.51-x86-5
>
> Is the rtnet loopback working with 4.9 ?
> Please let me know if this issue is fixed already ?
>
> This is the kernel logs:
> ------------------------------------------
> [612871.612307]
>                 *** RTnet for Xenomai v3.1-devel ***
>
> [612871.612310] RTnet: initialising real-time networking
> [612906.855980] initializing loopback...
> [612906.855998] RTnet: registered rtlo
> [613075.162006] BUG: unable to handle kernel paging request at 00007ffc7893b148
> [613075.162009] IP: [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
> [613075.162013] PGD 744f76067
> [613075.162014] PUD 80e1ae067
> [613075.162015] PMD 6893dd067
> [613075.162015] PTE 8000000581853867
>
> [613075.162018] Oops: 0001 [#1] SMP
> [613075.162019] Modules linked in: rt_loopback rtpacket rtudp rtipv4
> rtnet binfmt_misc 8021q garp mrp stp llc rfcomm bnep
> snd_hda_codec_hdmi nls_iso8859_1 eeepc_wmi intel_rapl btusb asus_wmi
> x86_pkg_temp_thermal btrtl sparse_keymap intel_powerclamp coretemp
> ath10k_pci kvm ath10k_core irqbypass ath mac80211 crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_realtek
> aes_x86_64 snd_hda_codec_generic lrw cfg80211 gf128mul glue_helper
> ablk_helper snd_hda_intel cryptd snd_hda_codec snd_hda_core input_leds
> snd_hwdep mei_me mei shpchp serio_raw hci_uart btbcm btqca btintel
> bluetooth acpi_als kfifo_buf intel_lpss_acpi industrialio intel_lpss
> mac_hid acpi_pad autofs4 hid_generic usbhid nouveau mxm_wmi ttm
> psmouse e1000e ptp pps_core ahci libahci i2c_hid pinctrl_sunrisepoint
> wmi hid video
> [613075.162049]  pinctrl_intel fjes
> [613075.162052] CPU: 0 PID: 12658 Comm: rtnet-client Not tainted
> 4.9.51-amd-x86-64-pintu-xeno3-rtdm #4
> [613075.162053] Hardware name: System manufacturer System Product Name/xxx
> [613075.162054] I-pipe domain: Linux
> [613075.162055] task: ffff903e8921da00 task.stack: ffff9e2f03348000
> [613075.162056] RIP: 0010:[<ffffffffc0a0503e>]  [<ffffffffc0a0503e>]
> rt_udp_getfrag+0x3e/0x110 [rtudp]
> [613075.162058] RSP: 0018:ffff9e2f0334bb70  EFLAGS: 00010202
> [613075.162059] RAX: 0000000000000000 RBX: 000000000000000a RCX:
> 00007ffc7893b140
> [613075.162060] RDX: 0000000000000000 RSI: ffff903e88e431e4 RDI:
> ffff9e2f0334bc40
> [613075.162061] RBP: ffff9e2f0334bb90 R08: ffff9e2f0334bdb8 R09:
> 0000000000000000
> [613075.162062] R10: ffff903e88e43100 R11: ffff90400c1c8800 R12:
> ffff903e88e431e4
> [613075.162063] R13: 0000000000000001 R14: ffff9e2f0334bc40 R15:
> 000000000000001e
> [613075.162064] FS:  00007f310b789740(0000) GS:ffff904016200000(0000)
> knlGS:0000000000000000
> [613075.162065] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [613075.162066] CR2: 00007ffc7893b148 CR3: 0000000744e5b000 CR4:
> 00000000003406f0
> [613075.162067] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [613075.162067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [613075.162068] Stack:
> [613075.162069]  ffff9040121bec00 ffff90400e262240 ffff9e2f0334bdb8
> 000000000000000a
> [613075.162071]  ffff9e2f0334bc00 ffffffffc0a0f204 ffff903dd1a84500
> ffff9e2f0334bc10
> [613075.162074]  0000000000033768 ffff9e2f0334bc40 ffffffffc0a05000
> 0000000f00000000
> [613075.162076] Call Trace:
> [613075.162091]  [<ffffffffc0a0f204>] rt_ip_build_xmit+0x1c4/0x2a0 [rtipv4]
> [613075.162093]  [<ffffffffc0a05000>] ? 0xffffffffc0a05000
> [613075.162094]  [<ffffffffc0a060db>] rt_udp_sendmsg+0x37b/0x3d0 [rtudp]
> [613075.162097]  [<ffffffffa70a4676>] ? update_curr+0x66/0x180
> [613075.162098]  [<ffffffffa70a67b8>] ? dequeue_entity+0x268/0xc00
> [613075.162100]  [<ffffffffa718a117>] ? ___xnsched_run.part.73+0x3d7/0x400
> [613075.162102]  [<ffffffffa70a4035>] ? hrtick_update+0x5/0x70
> [613075.162103]  [<ffffffffa70a76d7>] ? dequeue_task_fair+0x587/0x900
> [613075.162105]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> [613075.162106]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> [613075.162108]  [<ffffffffa719ad7b>] rtdm_fd_sendmsg+0xcb/0x240
> [613075.162109]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> [613075.162111]  [<ffffffffa71a03de>] CoBaLt_sendmsg+0x4e/0x70
> [613075.162113]  [<ffffffffa71ad017>] ipipe_syscall_hook+0x117/0x340
> [613075.162114]  [<ffffffffa712f9ff>] __ipipe_notify_syscall+0xbf/0x170
> [613075.162116]  [<ffffffffa79007b7>] pipeline_syscall+0x8/0x1b
> [613075.162118] Code: 54 49 89 f4 53 89 cb 8b 57 20 0f 85 c3 00 00 00
> 85 d2 7e 2f 8b 47 24 45 31 ed 49 63 cd 89 c2 41 83 c5 01 48 c1 e1 04
> 49 03 4e 18 <8b> 71 08 48 8b 39 e8 e7 f4 a0 e6 41 8b 56 20 41 89 46 24
> 44 39
> [613075.162141] RIP  [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
> [613075.162143]  RSP <ffff9e2f0334bb70>
> [613075.162144] CR2: 00007ffc7893b148
> [613075.162145] ---[ end trace 90458bf1f92e3557 ]---
> On Thu, Apr 26, 2018 at 9:53 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >
> > On 2018-04-25 13:36, Pintu Kumar wrote:
> > > Dear Jan,
> > >
> > > Thank you so much for your reply.
> > > I will try the latest stable version to check again.
> > > Is ipipe patches (linux: 4.9.51) also needs to be upgraded for this
> > > issue? Or only xenomai-3/kernel patches are enough?
> >
> > This particular issue was addressed in the Xenomai core, not the I-pipe
> > patch.
> >
> > >
> > > Actually, now I am stuck with another question.
> > > Hope if you could help me.
> > >
> > > As I said, I applied xenomai-3.0.6, kernel patches (using
> > > prepare_kernel script) to my x86_64 kernel, around 4 months back.
> > > I am using it since then. After that I never upgraded any patches.
> > >
> > > Now my concern is, how do I apply/upgrade only the latest patches?
> > > I did not remember the last commit until which I applied the patches.
> > >
> > > Is prepare_kernel script in intelligent enough to find the patch
> > > difference, and apply on the latest patches ?
> > >
> > > Normally how you people upgrade to the latest xenomai patches.
> > > If you have any suggestions, please guide me.
> >
> > Well, best practice is versioning control and build automation. You can
> > pull the patch into your local kernel or use the i-pipe kernel git tree
> > as feed (if you have no own patches). Then you just need
> > prepare-kernel.sh to refresh the xenomai core.
> >
> > And building everything can also be automated by scripts or more
> > advanced systems that also track the source revisions for you. Can be
> > git, can be something like yocto, buildroot etc.
> >
> > Jan
> >
> > --
> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Dear Jan, Greg,

Is there any pointer about this issue?
This is blocking my next work..

If there is already a fix available for it, please let me know.
Or, should I need analyze this freshly....
I wonder why no body else is facing this issue....

Thanks,
Pintu



On Wed, Jun 20, 2018 at 1:24 PM Pintu Kumar <pintu.ping@gmail.com> wrote:
>
> Hi,
>
> Can someone help me with this issue.
> I compared the xenomai-3 next repo
> (next/kernel/drivers/net/stack/ipv4/udp/udp.c) and the changes are
> almost same.
> Now I am stuck with this..
> Please help!
>
> Is there any test available for rtnet loopback ?
>
> Regards,
> Pintu
> On Tue, Jun 19, 2018 at 7:18 PM Pintu Kumar <pintu.ping@gmail.com> wrote:
> >
> > Hi,
> > I upgraded to xenomai-3-next branch for x86, but still rtnet loopback
> > is crashing for me.
> > The xenomai kernel is used from 4.9.51 until
> > commit: 10605b427b1408cdc6926f7c25d4a4eda527da8d
> >     Author: Philippe Gerum <rpm@xenomai.org>
> >     Date:   Mon Mar 26 09:17:02 2018 +0200
> >
> >         ipipe-core-4.9.51-x86-5
> >
> > Is the rtnet loopback working with 4.9 ?
> > Please let me know if this issue is fixed already ?
> >
> > This is the kernel logs:
> > ------------------------------------------
> > [612871.612307]
> >                 *** RTnet for Xenomai v3.1-devel ***
> >
> > [612871.612310] RTnet: initialising real-time networking
> > [612906.855980] initializing loopback...
> > [612906.855998] RTnet: registered rtlo
> > [613075.162006] BUG: unable to handle kernel paging request at 00007ffc7893b148
> > [613075.162009] IP: [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
> > [613075.162013] PGD 744f76067
> > [613075.162014] PUD 80e1ae067
> > [613075.162015] PMD 6893dd067
> > [613075.162015] PTE 8000000581853867
> >
> > [613075.162018] Oops: 0001 [#1] SMP
> > [613075.162019] Modules linked in: rt_loopback rtpacket rtudp rtipv4
> > rtnet binfmt_misc 8021q garp mrp stp llc rfcomm bnep
> > snd_hda_codec_hdmi nls_iso8859_1 eeepc_wmi intel_rapl btusb asus_wmi
> > x86_pkg_temp_thermal btrtl sparse_keymap intel_powerclamp coretemp
> > ath10k_pci kvm ath10k_core irqbypass ath mac80211 crct10dif_pclmul
> > crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_realtek
> > aes_x86_64 snd_hda_codec_generic lrw cfg80211 gf128mul glue_helper
> > ablk_helper snd_hda_intel cryptd snd_hda_codec snd_hda_core input_leds
> > snd_hwdep mei_me mei shpchp serio_raw hci_uart btbcm btqca btintel
> > bluetooth acpi_als kfifo_buf intel_lpss_acpi industrialio intel_lpss
> > mac_hid acpi_pad autofs4 hid_generic usbhid nouveau mxm_wmi ttm
> > psmouse e1000e ptp pps_core ahci libahci i2c_hid pinctrl_sunrisepoint
> > wmi hid video
> > [613075.162049]  pinctrl_intel fjes
> > [613075.162052] CPU: 0 PID: 12658 Comm: rtnet-client Not tainted
> > 4.9.51-amd-x86-64-pintu-xeno3-rtdm #4
> > [613075.162053] Hardware name: System manufacturer System Product Name/xxx
> > [613075.162054] I-pipe domain: Linux
> > [613075.162055] task: ffff903e8921da00 task.stack: ffff9e2f03348000
> > [613075.162056] RIP: 0010:[<ffffffffc0a0503e>]  [<ffffffffc0a0503e>]
> > rt_udp_getfrag+0x3e/0x110 [rtudp]
> > [613075.162058] RSP: 0018:ffff9e2f0334bb70  EFLAGS: 00010202
> > [613075.162059] RAX: 0000000000000000 RBX: 000000000000000a RCX:
> > 00007ffc7893b140
> > [613075.162060] RDX: 0000000000000000 RSI: ffff903e88e431e4 RDI:
> > ffff9e2f0334bc40
> > [613075.162061] RBP: ffff9e2f0334bb90 R08: ffff9e2f0334bdb8 R09:
> > 0000000000000000
> > [613075.162062] R10: ffff903e88e43100 R11: ffff90400c1c8800 R12:
> > ffff903e88e431e4
> > [613075.162063] R13: 0000000000000001 R14: ffff9e2f0334bc40 R15:
> > 000000000000001e
> > [613075.162064] FS:  00007f310b789740(0000) GS:ffff904016200000(0000)
> > knlGS:0000000000000000
> > [613075.162065] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [613075.162066] CR2: 00007ffc7893b148 CR3: 0000000744e5b000 CR4:
> > 00000000003406f0
> > [613075.162067] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [613075.162067] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [613075.162068] Stack:
> > [613075.162069]  ffff9040121bec00 ffff90400e262240 ffff9e2f0334bdb8
> > 000000000000000a
> > [613075.162071]  ffff9e2f0334bc00 ffffffffc0a0f204 ffff903dd1a84500
> > ffff9e2f0334bc10
> > [613075.162074]  0000000000033768 ffff9e2f0334bc40 ffffffffc0a05000
> > 0000000f00000000
> > [613075.162076] Call Trace:
> > [613075.162091]  [<ffffffffc0a0f204>] rt_ip_build_xmit+0x1c4/0x2a0 [rtipv4]
> > [613075.162093]  [<ffffffffc0a05000>] ? 0xffffffffc0a05000
> > [613075.162094]  [<ffffffffc0a060db>] rt_udp_sendmsg+0x37b/0x3d0 [rtudp]
> > [613075.162097]  [<ffffffffa70a4676>] ? update_curr+0x66/0x180
> > [613075.162098]  [<ffffffffa70a67b8>] ? dequeue_entity+0x268/0xc00
> > [613075.162100]  [<ffffffffa718a117>] ? ___xnsched_run.part.73+0x3d7/0x400
> > [613075.162102]  [<ffffffffa70a4035>] ? hrtick_update+0x5/0x70
> > [613075.162103]  [<ffffffffa70a76d7>] ? dequeue_task_fair+0x587/0x900
> > [613075.162105]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> > [613075.162106]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> > [613075.162108]  [<ffffffffa719ad7b>] rtdm_fd_sendmsg+0xcb/0x240
> > [613075.162109]  [<ffffffffa71a0390>] ? CoBaLt_recvmmsg+0x30/0x30
> > [613075.162111]  [<ffffffffa71a03de>] CoBaLt_sendmsg+0x4e/0x70
> > [613075.162113]  [<ffffffffa71ad017>] ipipe_syscall_hook+0x117/0x340
> > [613075.162114]  [<ffffffffa712f9ff>] __ipipe_notify_syscall+0xbf/0x170
> > [613075.162116]  [<ffffffffa79007b7>] pipeline_syscall+0x8/0x1b
> > [613075.162118] Code: 54 49 89 f4 53 89 cb 8b 57 20 0f 85 c3 00 00 00
> > 85 d2 7e 2f 8b 47 24 45 31 ed 49 63 cd 89 c2 41 83 c5 01 48 c1 e1 04
> > 49 03 4e 18 <8b> 71 08 48 8b 39 e8 e7 f4 a0 e6 41 8b 56 20 41 89 46 24
> > 44 39
> > [613075.162141] RIP  [<ffffffffc0a0503e>] rt_udp_getfrag+0x3e/0x110 [rtudp]
> > [613075.162143]  RSP <ffff9e2f0334bb70>
> > [613075.162144] CR2: 00007ffc7893b148
> > [613075.162145] ---[ end trace 90458bf1f92e3557 ]---
> > On Thu, Apr 26, 2018 at 9:53 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> > >
> > > On 2018-04-25 13:36, Pintu Kumar wrote:
> > > > Dear Jan,
> > > >
> > > > Thank you so much for your reply.
> > > > I will try the latest stable version to check again.
> > > > Is ipipe patches (linux: 4.9.51) also needs to be upgraded for this
> > > > issue? Or only xenomai-3/kernel patches are enough?
> > >
> > > This particular issue was addressed in the Xenomai core, not the I-pipe
> > > patch.
> > >
> > > >
> > > > Actually, now I am stuck with another question.
> > > > Hope if you could help me.
> > > >
> > > > As I said, I applied xenomai-3.0.6, kernel patches (using
> > > > prepare_kernel script) to my x86_64 kernel, around 4 months back.
> > > > I am using it since then. After that I never upgraded any patches.
> > > >
> > > > Now my concern is, how do I apply/upgrade only the latest patches?
> > > > I did not remember the last commit until which I applied the patches.
> > > >
> > > > Is prepare_kernel script in intelligent enough to find the patch
> > > > difference, and apply on the latest patches ?
> > > >
> > > > Normally how you people upgrade to the latest xenomai patches.
> > > > If you have any suggestions, please guide me.
> > >
> > > Well, best practice is versioning control and build automation. You can
> > > pull the patch into your local kernel or use the i-pipe kernel git tree
> > > as feed (if you have no own patches). Then you just need
> > > prepare-kernel.sh to refresh the xenomai core.
> > >
> > > And building everything can also be automated by scripts or more
> > > advanced systems that also track the source revisions for you. Can be
> > > git, can be something like yocto, buildroot etc.
> > >
> > > Jan
> > >
> > > --
> > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > > Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-21 13:20, Pintu Kumar wrote:
> Dear Jan, Greg,
> 
> Is there any pointer about this issue?
> This is blocking my next work..

Does this solve the issue AND still generate valid UDP checksums (please
check with wireshark or against a normal networking stack?

diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
index 8e80d3e0b..bb0b0fc12 100644
--- a/kernel/drivers/net/stack/ipv4/udp/udp.c
+++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
@@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
     if (offset)
 	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
 
-    /* Checksum of the complete data part of the UDP message: */
-    for (i = 0; i < ufh->iovlen; i++) {
-            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
-                                       ufh->wcheck);
-    }
-
     ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
 			      to + sizeof(struct udphdr),
 			      fraglen - sizeof(struct udphdr));
     if (ret)
 	    return ret;
 
+    /* Checksum of the complete data part of the UDP message: */
+    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
+			       fraglen - sizeof(struct udphdr),
+			       ufh->wcheck);
+
     /* Checksum of the udp header: */
     ufh->wcheck = csum_partial((unsigned char *)ufh,
 			       sizeof(struct udphdr), ufh->wcheck);

> 
> If there is already a fix available for it, please let me know.
> Or, should I need analyze this freshly....
> I wonder why no body else is facing this issue....

Users may still have the related memory protection features off which
surface this issue.

Jan
-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Xenomai] Kernel panic during kernel bootup on arm64 qemu (e1000e)

[gengdongjiu]

Hi
   I use the stable 3.x Xenomai and ipipe-4.9.y, and run it using below qemu commands:
 qemu-kvm -enable-kvm -m 1024 -cpu host -machine virt,gic-version=3  -smp 10 -nographic -kernel Image -append "rdinit=/init console=ttyAMA0 mem=512M root=/dev/ram0 earlycon=pl011,0x9000000 rw" -initrd guestfs_new.cpio.gz
 it will happen below panic, do you know the reason? thanks

[    1.286223] Hardware name: linux,dummy-virt (DT)
[    1.286773] I-pipe domain: Linux
[    1.287153] task: ffff80001d8d0000 task.stack: ffff80001d8a4000
[    1.287840] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.288395] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.288982] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.289818] sp : ffff80001ef12fd0
[    1.290201] x29: ffff80001ef12fd0 x28: ffff80001d8a4000
[    1.290866] x27: 0000000000000003 x26: ffff80001ef13090
[    1.291506] x25: ffff80001ef0f0a0 x24: ffff00000817ad00
[    1.292147] x23: ffff000008db2000 x22: 0000000000000003
[    1.292787] x21: 0000000000000000 x20: ffff80001ef1df80
[    1.293427] x19: ffff80001d804a00 x18: 000000000000000e
[    1.294066] x17: 00000000000045bf x16: 0140000000000000
[    1.294716] x15: ffff000049f61000 x14: ffff80001effd290
[    1.295357] x13: ffff000008e66078 x12: 0000000000000000
[    1.295998] x11: 0000000000000000 x10: 000000003b9aca00
[    1.296639] x9 : 0000000000000000 x8 : 0000000000000000
[    1.297281] x7 : 0000000000000000 x6 : 0000000000000000
[    1.297922] x5 : 00008000161bd000 x4 : ffff000009050000
[    1.298582] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.299223] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.299865]
[    1.300049] Process swapper/0 (pid: 1, stack limit = 0xffff80001d8a4038)
[    1.300816] Stack: (0xffff80001ef12fd0 to 0xffff80001d8a8000)
[    1.301474] Call trace:
[    1.301763] Exception stack(0xffff80001ef12e70 to 0xffff80001ef12fa0)
[    1.302500] 2e60:                                   ffff80001d804a00 0001000000000000
[    1.303403] 2e80: ffff80001ef12fd0 ffff0000087899d0 00000000800001c5 ffff0000083d5b04
[    1.304297] 2ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.305191] 2ec0: ffff000009050000 00008000161bd000 0000000000000000 0000000000000000
[    1.306084] 2ee0: 0000000000000000 0000000000000000 000000003b9aca00 0000000000000000
[    1.307000] 2f00: 0000000000000000 ffff000008e66078 ffff80001effd290 ffff000049f61000
[    1.307894] 2f20: 0140000000000000 00000000000045bf 000000000000000e ffff80001d804a00
[    1.308789] 2f40: ffff80001ef1df80 0000000000000000 0000000000000003 ffff000008db2000
[    1.309687] 2f60: ffff00000817ad00 ffff80001ef0f0a0 ffff80001ef13090 0000000000000003
[    1.310592] 2f80: ffff80001d8a4000 ffff80001ef12fd0 ffff000008156e18 ffff80001ef12fd0
[    1.311497] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.312176] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.312873] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.313519] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
[    1.314262] 7b00:                                   0000000000000000 ffff000008eab1a8
[    1.315181] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
[    1.316081] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.316981] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
[    1.317880] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
[    1.318786] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
[    1.319688] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
[    1.320587] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
[    1.321489] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
[    1.322390] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
[    1.323306] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.323942] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
[    1.324687] 7b00:                                   0000000000000000 ffff000008eab1a8
[    1.325588] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
[    1.326490] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.327402] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
[    1.328308] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
[    1.329212] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
[    1.330117] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
[    1.331035] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
[    1.331933] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
[    1.332830] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
[    1.333730] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.334295] [<ffff000008155594>] ipipe_critical_exit+0x9c/0xa8
[    1.334976] [<ffff0000081576a0>] ipipe_timer_start+0x90/0x1d0
[    1.335654] [<ffff00000817b184>] xntimer_grab_hardware+0xfc/0x380
[    1.336368] [<ffff000008ca2660>] xenomai_init+0x3c8/0x4e0
[    1.336997] [<ffff000008083b08>] do_one_initcall+0x38/0x128
[    1.337654] [<ffff000008c90cf8>] kernel_init_freeable+0x1b8/0x254
[    1.338367] [<ffff0000088de8a8>] kernel_init+0x10/0x100
[    1.338991] [<ffff0000080837e4>] ret_from_fork+0x14/0x30
[    1.339609] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.340325] Internal error: undefined instruction: 0 [#2] PREEMPT SMP
[    1.340337] ---[ end trace ee076ceca10e9294 ]---
[    1.340351] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.340351]
[    1.340352] SMP: stopping secondary CPUs
[    1.343408] Modules linked in:
[    1.343819] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    1.345016] Hardware name: linux,dummy-virt (DT)
[    1.345596] I-pipe domain: Linux
[    1.346007] task: ffff80001d923e80 task.stack: ffff80001d93c000
[    1.346768] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.347372] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.348018] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.348936] sp : ffff80001ef48fd0
[    1.349357] x29: ffff80001ef48fd0 x28: ffff80001d93c000
[    1.350057] x27: ffff80001d93c000 x26: ffff80001ef49090
[    1.350779] x25: ffff80001ef450a0 x24: ffff000008d60d20
[    1.351482] x23: ffff000008db2000 x22: 0000000000000003
[    1.352186] x21: 0000000000000000 x20: ffff80001ef53f80
[    1.352885] x19: ffff80001d804a00 x18: 0000000000000007
[    1.353585] x17: 000000000000000e x16: 0000000000000001
[    1.354284] x15: 0000000000000019 x14: 0000000000000033
[    1.355005] x13: 000000000000004c x12: 0000000000000068
[    1.355710] x11: 0000000000000000 x10: ffff80001d923e88
[    1.356411] x9 : ffff80001d93c000 x8 : ffff80001d924760
[    1.357114] x7 : 0000000000000001 x6 : 003aa6a600000000
[    1.357818] x5 : 00008000161f3000 x4 : ffff000009090000
[    1.358521] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.359239] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.359940]
[    1.360141] Process swapper/4 (pid: 0, stack limit = 0xffff80001d93c038)
[    1.360979] Stack: (0xffff80001ef48fd0 to 0xffff80001d940000)
[    1.361699] Call trace:
[    1.362018] Exception stack(0xffff80001ef48e70 to 0xffff80001ef48fa0)
[    1.362842] 8e60:                                   ffff80001d804a00 0001000000000000
[    1.363823] 8e80: ffff80001ef48fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
[    1.364801] 8ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.365780] 8ec0: ffff000009090000 00008000161f3000 003aa6a600000000 0000000000000001
[    1.366774] 8ee0: ffff80001d924760 ffff80001d93c000 ffff80001d923e88 0000000000000000
[    1.367752] 8f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
[    1.368730] 8f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
[    1.369707] 8f40: ffff80001ef53f80 0000000000000000 0000000000000003 ffff000008db2000
[    1.370740] 8f60: ffff000008d60d20 ffff80001ef450a0 ffff80001ef49090 ffff80001d93c000
[    1.371723] 8f80: ffff80001d93c000 ffff80001ef48fd0 ffff000008156e18 ffff80001ef48fd0
[    1.372706] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.373442] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.374197] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.374916] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
[    1.375726] fe00:                                   0000000000000000 00008000161f3000
[    1.376708] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
[    1.377688] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
[    1.378685] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
[    1.379668] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.380648] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
[    1.381630] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
[    1.382627] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
[    1.383609] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
[    1.384590] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.385572] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.386263] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
[    1.387089] fe00:                                   0000000000000000 00008000161f3000
[    1.388069] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
[    1.389049] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
[    1.390029] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
[    1.391025] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.392005] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
[    1.392986] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
[    1.393967] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
[    1.394962] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
[    1.395941] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.396921] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.397538] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    1.398263] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    1.398958] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    1.399679] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    1.400421] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
[    1.401211] [<00000000408ea1a4>] 0x408ea1a4
[    1.401745] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.402516] Internal error: undefined instruction: 0 [#3] PREEMPT SMP
[    1.403371] Modules linked in:
[    1.403780] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    1.404974] Hardware name: linux,dummy-virt (DT)
[    1.405555] I-pipe domain: Linux
[    1.405965] task: ffff80001d925780 task.stack: ffff80001d94c000
[    1.406727] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.407329] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.407972] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.408889] sp : ffff80001ef7efd0
[    1.409311] x29: ffff80001ef7efd0 x28: ffff80001d94c000
[    1.410010] x27: ffff80001d94c000 x26: ffff80001ef7f090
[    1.410726] x25: ffff80001ef7b0a0 x24: ffff000008d60d20
[    1.411425] x23: ffff000008db2000 x22: 0000000000000003
[    1.412122] x21: 0000000000000000 x20: ffff80001ef89f80
[    1.412819] x19: ffff80001d804a00 x18: 0000000000000007
[    1.413516] x17: 000000000000000e x16: 0000000000000001
[    1.414214] x15: 0000000000000019 x14: 0000000000000033
[    1.414929] x13: 000000000000004c x12: 0000000000000068
[    1.415623] x11: 0000000000000000 x10: ffff80001d925788
[    1.416317] x9 : ffff80001d94c000 x8 : ffff80001d926060
[    1.417011] x7 : 0000000000000001 x6 : 003aa6a600000000
[    1.417703] x5 : 0000800016229000 x4 : ffff0000090d0000
[    1.418396] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.419107] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.419799]
[    1.419996] Process swapper/6 (pid: 0, stack limit = 0xffff80001d94c038)
[    1.420823] Stack: (0xffff80001ef7efd0 to 0xffff80001d950000)
[    1.421533] Call trace:
[    1.421846] Exception stack(0xffff80001ef7ee70 to 0xffff80001ef7efa0)
[    1.422680] ee60:                                   ffff80001d804a00 0001000000000000
[    1.423641] ee80: ffff80001ef7efd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
[    1.424601] eea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.425559] eec0: ffff0000090d0000 0000800016229000 003aa6a600000000 0000000000000001
[    1.426518] eee0: ffff80001d926060 ffff80001d94c000 ffff80001d925788 0000000000000000
[    1.427497] ef00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
[    1.428456] ef20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
[    1.429415] ef40: ffff80001ef89f80 0000000000000000 0000000000000003 ffff000008db2000
[    1.430375] ef60: ffff000008d60d20 ffff80001ef7b0a0 ffff80001ef7f090 ffff80001d94c000
[    1.431356] ef80: ffff80001d94c000 ffff80001ef7efd0 ffff000008156e18 ffff80001ef7efd0
[    1.432324] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.433050] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.433796] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.434488] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
[    1.435312] fe00:                                   0000000000000000 0000800016229000
[    1.436272] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
[    1.437234] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
[    1.438194] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
[    1.439174] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.440137] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
[    1.441098] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
[    1.442058] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
[    1.443038] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
[    1.443999] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.444959] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.445638] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
[    1.446431] fe00:                                   0000000000000000 0000800016229000
[    1.447407] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
[    1.448368] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
[    1.449328] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
[    1.450288] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.451266] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
[    1.452226] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
[    1.453186] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
[    1.454146] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
[    1.455122] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.456083] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.456689] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    1.457399] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    1.458059] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    1.458808] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    1.459535] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
[    1.460310] [<00000000408ea1a4>] 0x408ea1a4
[    1.460833] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.461589] Internal error: undefined instruction: 0 [#4] PREEMPT SMP
[    1.462402] Modules linked in:
[    1.462822] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    1.464019] Hardware name: linux,dummy-virt (DT)
[    1.464599] I-pipe domain: Linux
[    1.465010] task: ffff80001d921900 task.stack: ffff80001d930000
[    1.465748] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.466347] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.467020] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.467929] sp : ffff80001eef7fd0
[    1.468345] x29: ffff80001eef7fd0 x28: ffff80001d930000
[    1.469038] x27: ffff80001d930000 x26: ffff80001eef8090
[    1.469729] x25: ffff80001eef40a0 x24: ffff000008d60d20
[    1.470420] x23: ffff000008db2000 x22: 0000000000000003
[    1.471134] x21: 0000000000000000 x20: ffff80001ef02f80
[    1.471828] x19: ffff80001d804a00 x18: 0000000000000007
[    1.472522] x17: 000000000000000e x16: 0000000000000001
[    1.473215] x15: 0000000000000019 x14: 0000000000000033
[    1.473908] x13: 000000000000004c x12: 0000000000000068
[    1.474634] x11: 0000000000000000 x10: ffff80001d921908
[    1.475328] x9 : ffff80001d930000 x8 : ffff80001d9221e0
[    1.476022] x7 : 0000000000000001 x6 : 003aa6a600000000
[    1.476714] x5 : 00008000161a2000 x4 : ffff000009030000
[    1.477407] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.478100] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.478822]
[    1.479021] Process swapper/1 (pid: 0, stack limit = 0xffff80001d930038)
[    1.479851] Stack: (0xffff80001eef7fd0 to 0xffff80001d934000)
[    1.480563] Call trace:
[    1.480877] Exception stack(0xffff80001eef7e70 to 0xffff80001eef7fa0)
[    1.481675] 7e60:                                   ffff80001d804a00 0001000000000000
[    1.482676] 7e80: ffff80001eef7fd0 ffff0000087899d0 00000000800001c5 ffff000008128070
[    1.483640] 7ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.484603] 7ec0: ffff000009030000 00008000161a2000 003aa6a600000000 0000000000000001
[    1.485565] 7ee0: ffff80001d9221e0 ffff80001d930000 ffff80001d921908 0000000000000000
[    1.486526] 7f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
[    1.487516] 7f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
[    1.488480] 7f40: ffff80001ef02f80 0000000000000000 0000000000000003 ffff000008db2000
[    1.489445] 7f60: ffff000008d60d20 ffff80001eef40a0 ffff80001eef8090 ffff80001d930000
[    1.490411] 7f80: ffff80001d930000 ffff80001eef7fd0 ffff000008156e18 ffff80001eef7fd0
[    1.491399] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.492124] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.492869] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.493562] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
[    1.494360] 3e00:                                   0000000000000000 00008000161a2000
[    1.495344] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
[    1.496310] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
[    1.497275] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
[    1.498240] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.499221] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
[    1.500189] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
[    1.501156] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
[    1.502122] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
[    1.503107] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.504074] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.504756] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
[    1.505555] 3e00:                                   0000000000000000 00008000161a2000
[    1.506522] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
[    1.507518] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
[    1.508488] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
[    1.509457] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.510426] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
[    1.511427] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
[    1.512398] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
[    1.513366] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
[    1.514334] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.515321] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.515930] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    1.516645] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    1.517309] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    1.518014] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    1.518769] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
[    1.519549] [<00000000408ea1a4>] 0x408ea1a4
[    1.520076] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.520837] Internal error: undefined instruction: 0 [#5] PREEMPT SMP
[    1.521669] Modules linked in:
[    1.522075] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    1.523272] Hardware name: linux,dummy-virt (DT)
[    1.523850] I-pipe domain: Linux
[    1.524260] task: ffff80001d926400 task.stack: ffff80001d950000
[    1.524999] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.525600] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.526242] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.527176] sp : ffff80001ef99fd0
[    1.527595] x29: ffff80001ef99fd0 x28: ffff80001d950000
[    1.528291] x27: ffff80001d950000 x26: ffff80001ef9a090
[    1.528988] x25: ffff80001ef960a0 x24: ffff000008d60d20
[    1.529685] x23: ffff000008db2000 x22: 0000000000000003
[    1.530381] x21: 0000000000000000 x20: ffff80001efa4f80
[    1.531143] x19: ffff80001d804a00 x18: 0000000000000007
[    1.531842] x17: 000000000000000e x16: 0000000000000001
[    1.532540] x15: 0000000000000019 x14: 0000000000000033
[    1.533236] x13: 000000000000004c x12: 0000000000000068
[    1.533933] x11: 0000000000000000 x10: ffff80001d926408
[    1.534655] x9 : ffff80001d950000 x8 : ffff80001d926ce0
[    1.535354] x7 : 0000000000000001 x6 : 003aa6a600000000
[    1.536052] x5 : 0000800016244000 x4 : ffff0000090f0000
[    1.536750] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.537449] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.538146]
[    1.538347] Process swapper/7 (pid: 0, stack limit = 0xffff80001d950038)
[    1.539207] Stack: (0xffff80001ef99fd0 to 0xffff80001d954000)
[    1.539926] Call trace:
[    1.540242] Exception stack(0xffff80001ef99e70 to 0xffff80001ef99fa0)
[    1.541046] 9e60:                                   ffff80001d804a00 0001000000000000
[    1.542019] 9e80: ffff80001ef99fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
[    1.543010] 9ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.543987] 9ec0: ffff0000090f0000 0000800016244000 003aa6a600000000 0000000000000001
[    1.544963] 9ee0: ffff80001d926ce0 ffff80001d950000 ffff80001d926408 0000000000000000
[    1.545940] 9f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
[    1.546947] 9f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
[    1.547905] 9f40: ffff80001efa4f80 0000000000000000 0000000000000003 ffff000008db2000
[    1.548862] 9f60: ffff000008d60d20 ffff80001ef960a0 ffff80001ef9a090 ffff80001d950000
[    1.549820] 9f80: ffff80001d950000 ffff80001ef99fd0 ffff000008156e18 ffff80001ef99fd0
[    1.550799] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.551520] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.552260] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.552947] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
[    1.553739] 3e00:                                   0000000000000000 0000800016244000
[    1.554715] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
[    1.555676] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
[    1.556636] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
[    1.557597] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.558573] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
[    1.559537] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
[    1.560496] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
[    1.561454] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
[    1.562412] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.563391] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.564068] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
[    1.564859] 3e00:                                   0000000000000000 0000800016244000
[    1.565816] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
[    1.566797] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
[    1.567758] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
[    1.568718] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
[    1.569679] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
[    1.570657] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
[    1.571617] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
[    1.572579] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
[    1.573540] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.574502] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.575127] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    1.575837] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    1.576497] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    1.577198] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    1.577919] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
[    1.578716] [<00000000408ea1a4>] 0x408ea1a4
[    1.579238] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.579992] Internal error: undefined instruction: 0 [#6] PREEMPT SMP
[    1.580808] Modules linked in:
[    1.581208] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    1.582377] Hardware name: linux,dummy-virt (DT)
[    1.582954] I-pipe domain: Linux
[    1.583356] task: ffff80001d923200 task.stack: ffff80001d938000
[    1.584081] PC is at arch_itimer_ack_phys+0x0/0x18
[    1.584671] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    1.585303] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    1.586203] sp : ffff80001ef2dfd0
[    1.586633] x29: ffff80001ef2dfd0 x28: ffff80001d938000
[    1.587320] x27: ffff80001d938000 x26: ffff80001ef2e090
[    1.588006] x25: ffff80001ef2a0a0 x24: ffff000008d60d20
[    1.588691] x23: ffff000008db2000 x22: 0000000000000003
[    1.589377] x21: 0000000000000000 x20: ffff80001ef38f80
[    1.590061] x19: ffff80001d804a00 x18: 0000000000000007
[    1.590763] x17: 000000000000000e x16: 0000000000000001
[    1.591448] x15: 0000000000000019 x14: ffffffffffffffff
[    1.592133] x13: ffffffffffffffff x12: 0000000000000000
[    1.592817] x11: 0000000000000001 x10: ffff80001d923208
[    1.593502] x9 : ffff80001d938000 x8 : ffff80001d923ae0
[    1.594186] x7 : 0000000000000001 x6 : 003aa6a600000000
[    1.594888] x5 : 00008000161d8000 x4 : ffff000009070000
[    1.595574] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    1.596259] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    1.596945]
[    1.597140] Process swapper/3 (pid: 0, stack limit = 0xffff80001d938038)
[    1.597960] Stack: (0xffff80001ef2dfd0 to 0xffff80001d93c000)
[    1.598680] Call trace:
[    1.598991] Exception stack(0xffff80001ef2de70 to 0xffff80001ef2dfa0)
[    1.599781] de60:                                   ffff80001d804a00 0001000000000000
[    1.600737] de80: ffff80001ef2dfd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
[    1.601693] dea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
[    1.602663] dec0: ffff000009070000 00008000161d8000 003aa6a600000000 0000000000000001
[    1.603618] dee0: ffff80001d923ae0 ffff80001d938000 ffff80001d923208 0000000000000001
[    1.604573] df00: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
[    1.605527] df20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
[    1.606483] df40: ffff80001ef38f80 0000000000000000 0000000000000003 ffff000008db2000
[    1.607453] df60: ffff000008d60d20 ffff80001ef2a0a0 ffff80001ef2e090 ffff80001d938000
[    1.608408] df80: ffff80001d938000 ffff80001ef2dfd0 ffff000008156e18 ffff80001ef2dfd0
[    1.609363] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    1.610078] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    1.610831] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    1.611515] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
[    1.612302] be00:                                   0000000000000000 00008000161d8000
[    1.613256] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
[    1.614209] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
[    1.615179] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
[    1.616134] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
[    1.617088] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
[    1.618085] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
[    1.619047] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
[    1.620003] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
[    1.620957] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.621912] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    1.622608] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
[    1.623394] be00:                                   0000000000000000 00008000161d8000
[    1.624342] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
[    1.625291] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
[    1.626240] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
[    1.627208] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
[    1.628156] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
[    1.629103] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
[    1.630049] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
[    1.631013] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
[    1.631960] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
[    1.632908] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    1.633505] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    1.634206] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    1.634894] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    1.635595] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    1.636314] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
[    1.637082] [<00000000408ea1a4>] 0x408ea1a4
[    1.637602] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    1.638352] Internal error: undefined instruction: 0 [#7] PREEMPT SMP
[    2.174474] Modules linked in:
[    2.174895] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
[    2.175975] Hardware name: linux,dummy-virt (DT)
[    2.176500] I-pipe domain: Linux
[    2.176873] task: ffff000008da0080 task.stack: ffff000008d90000
[    2.177549] PC is at arch_itimer_ack_phys+0x0/0x18
[    2.178097] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
[    2.178698] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
[    2.179532] sp : ffff80001eedcda0
[    2.179914] x29: ffff80001eedcda0 x28: ffff000008d90000
[    2.180548] x27: ffff000008d90000 x26: 0000000000000002
[    2.181181] x25: ffff80001eeddbb4 x24: ffff000008d60d20
[    2.181814] x23: ffff000008db2000 x22: 0000000000000003
[    2.182447] x21: 0000000000000000 x20: ffff80001eee7f80
[    2.183096] x19: ffff80001d804a00 x18: 0000000000000007
[    2.183728] x17: 000000000000000e x16: 0000000000000001
[    2.184361] x15: 0000000000000019 x14: ffffffffffffffff
[    2.184994] x13: ffffffffffffffff x12: 0000000000000000
[    2.185628] x11: 0000000000000004 x10: ffff80001eee0968
[    2.186260] x9 : ffff80001eee08f0 x8 : ffff80001eee0978
[    2.186909] x7 : ffff80001eee08e8 x6 : 0000000000000000
[    2.187543] x5 : 0000800016187000 x4 : ffff000009010000
[    2.188176] x3 : 0000000008000000 x2 : ffff0000083d5ae8
[    2.188810] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
[    2.189443]
[    2.189625] Process swapper/0 (pid: 0, stack limit = 0xffff000008d90038)
[    2.190382] Stack: (0xffff80001eedcda0 to 0xffff000008d94000)
[    2.191047] Call trace:
[    2.191334] Exception stack(0xffff80001eedcc40 to 0xffff80001eedcd70)
[    2.192063] cc40: ffff80001d804a00 0001000000000000 ffff80001eedcda0 ffff0000087899d0
[    2.192945] cc60: 00000000800001c5 0000000000000400 ffff0000087899d0 ffff0000083d4ed8
[    2.193827] cc80: ffff0000083d5ae8 0000000008000000 ffff000009010000 0000800016187000
[    2.194726] cca0: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
[    2.195607] ccc0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
[    2.196491] cce0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
[    2.197373] cd00: 0000000000000007 ffff80001d804a00 ffff80001eee7f80 0000000000000000
[    2.198255] cd20: 0000000000000003 ffff000008db2000 ffff000008d60d20 ffff80001eeddbb4
[    2.199152] cd40: 0000000000000002 ffff000008d90000 ffff000008d90000 ffff80001eedcda0
[    2.200033] cd60: ffff000008156e18 ffff80001eedcda0
[    2.200589] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
[    2.201251] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
[    2.201932] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
[    2.202579] Exception stack(0xffff80001eedce10 to 0xffff80001eedcf40)
[    2.203307] ce00:                                   0000000000001fff ffff000008118c28
[    2.204188] ce20: ffff80001eedcf80 ffff000008082fc8 ffff80001eedce50 0001000000000000
[    2.205069] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
[    2.205949] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
[    2.206867] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
[    2.207743] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
[    2.208618] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
[    2.209494] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
[    2.210369] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
[    2.211269] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
[    2.212147] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
[    2.212766] Exception stack(0xffff80001eedce30 to 0xffff80001eedcf60)
[    2.213490] ce20:                                   ffff80001eedce50 0001000000000000
[    2.214368] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
[    2.215262] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
[    2.216138] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
[    2.217015] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
[    2.217892] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
[    2.218785] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
[    2.219664] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
[    2.220543] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
[    2.221422] cf40: ffff000008155fd4 ffff80001eedcf80 ffff000008155e6c 0000000020000145
[    2.222300] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    2.222881] [<ffff000008155fd4>] __ipipe_do_sync_pipeline+0xa4/0xb8
[    2.223587] [<ffff000008156730>] __ipipe_dispatch_irq+0x170/0x1d8
[    2.224274] [<ffff000008081274>] __ipipe_grab_ipi+0x1c/0x30
[    2.224902] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
[    2.225626] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
[    2.226501] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
[    2.227442] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
[    2.228329] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
[    2.229215] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
[    2.230100] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
[    2.231007] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
[    2.231895] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
[    2.232782] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
[    2.233669] 3ee0: ffffffffffffffff ffff000008d60d20
[    2.234228] [<ffff00000808178c>] gic_handle_irq+0x64/0x100
[    2.234885] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
[    2.235610] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
[    2.236488] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
[    2.237365] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
[    2.238241] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
[    2.239151] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
[    2.240028] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
[    2.240906] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
[    2.241783] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
[    2.242686] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
[    2.243565] 3ee0: ffffffffffffffff ffff000008d60d20
[    2.244120] [<ffff000008082fc8>] el1_irq+0xc8/0x108
[    2.244675] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
[    2.245326] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
[    2.245932] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
[    2.246594] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
[    2.247256] [<ffff0000088de888>] rest_init+0x80/0x90
[    2.247824] [<ffff000008c90b2c>] start_kernel+0x380/0x394
[    2.248437] [<ffff000008c901d8>] __primary_switched+0x5c/0x64
[    2.249090] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
[    2.249784] ---[ end trace ee076ceca10e9295 ]---
[    2.837878] SMP: failed to stop secondary CPUs 0,2
[    2.838465] Kernel Offset: disabled
[    2.838887] Memory Limit: 512 MB

Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu (e1000e)

[Greg Gallagher]

For ARM64 you need to use the Xenomai next branch not the stable release.

-Greg

On Thu, Jun 21, 2018 at 8:27 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
> Hi
>    I use the stable 3.x Xenomai and ipipe-4.9.y, and run it using below qemu commands:
>  qemu-kvm -enable-kvm -m 1024 -cpu host -machine virt,gic-version=3  -smp 10 -nographic -kernel Image -append "rdinit=/init console=ttyAMA0 mem=512M root=/dev/ram0 earlycon=pl011,0x9000000 rw" -initrd guestfs_new.cpio.gz
>  it will happen below panic, do you know the reason? thanks
>
> [    1.286223] Hardware name: linux,dummy-virt (DT)
> [    1.286773] I-pipe domain: Linux
> [    1.287153] task: ffff80001d8d0000 task.stack: ffff80001d8a4000
> [    1.287840] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.288395] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.288982] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.289818] sp : ffff80001ef12fd0
> [    1.290201] x29: ffff80001ef12fd0 x28: ffff80001d8a4000
> [    1.290866] x27: 0000000000000003 x26: ffff80001ef13090
> [    1.291506] x25: ffff80001ef0f0a0 x24: ffff00000817ad00
> [    1.292147] x23: ffff000008db2000 x22: 0000000000000003
> [    1.292787] x21: 0000000000000000 x20: ffff80001ef1df80
> [    1.293427] x19: ffff80001d804a00 x18: 000000000000000e
> [    1.294066] x17: 00000000000045bf x16: 0140000000000000
> [    1.294716] x15: ffff000049f61000 x14: ffff80001effd290
> [    1.295357] x13: ffff000008e66078 x12: 0000000000000000
> [    1.295998] x11: 0000000000000000 x10: 000000003b9aca00
> [    1.296639] x9 : 0000000000000000 x8 : 0000000000000000
> [    1.297281] x7 : 0000000000000000 x6 : 0000000000000000
> [    1.297922] x5 : 00008000161bd000 x4 : ffff000009050000
> [    1.298582] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.299223] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.299865]
> [    1.300049] Process swapper/0 (pid: 1, stack limit = 0xffff80001d8a4038)
> [    1.300816] Stack: (0xffff80001ef12fd0 to 0xffff80001d8a8000)
> [    1.301474] Call trace:
> [    1.301763] Exception stack(0xffff80001ef12e70 to 0xffff80001ef12fa0)
> [    1.302500] 2e60:                                   ffff80001d804a00 0001000000000000
> [    1.303403] 2e80: ffff80001ef12fd0 ffff0000087899d0 00000000800001c5 ffff0000083d5b04
> [    1.304297] 2ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.305191] 2ec0: ffff000009050000 00008000161bd000 0000000000000000 0000000000000000
> [    1.306084] 2ee0: 0000000000000000 0000000000000000 000000003b9aca00 0000000000000000
> [    1.307000] 2f00: 0000000000000000 ffff000008e66078 ffff80001effd290 ffff000049f61000
> [    1.307894] 2f20: 0140000000000000 00000000000045bf 000000000000000e ffff80001d804a00
> [    1.308789] 2f40: ffff80001ef1df80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.309687] 2f60: ffff00000817ad00 ffff80001ef0f0a0 ffff80001ef13090 0000000000000003
> [    1.310592] 2f80: ffff80001d8a4000 ffff80001ef12fd0 ffff000008156e18 ffff80001ef12fd0
> [    1.311497] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.312176] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.312873] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.313519] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
> [    1.314262] 7b00:                                   0000000000000000 ffff000008eab1a8
> [    1.315181] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
> [    1.316081] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    1.316981] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
> [    1.317880] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
> [    1.318786] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
> [    1.319688] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
> [    1.320587] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
> [    1.321489] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
> [    1.322390] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
> [    1.323306] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.323942] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
> [    1.324687] 7b00:                                   0000000000000000 ffff000008eab1a8
> [    1.325588] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
> [    1.326490] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    1.327402] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
> [    1.328308] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
> [    1.329212] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
> [    1.330117] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
> [    1.331035] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
> [    1.331933] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
> [    1.332830] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
> [    1.333730] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.334295] [<ffff000008155594>] ipipe_critical_exit+0x9c/0xa8
> [    1.334976] [<ffff0000081576a0>] ipipe_timer_start+0x90/0x1d0
> [    1.335654] [<ffff00000817b184>] xntimer_grab_hardware+0xfc/0x380
> [    1.336368] [<ffff000008ca2660>] xenomai_init+0x3c8/0x4e0
> [    1.336997] [<ffff000008083b08>] do_one_initcall+0x38/0x128
> [    1.337654] [<ffff000008c90cf8>] kernel_init_freeable+0x1b8/0x254
> [    1.338367] [<ffff0000088de8a8>] kernel_init+0x10/0x100
> [    1.338991] [<ffff0000080837e4>] ret_from_fork+0x14/0x30
> [    1.339609] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.340325] Internal error: undefined instruction: 0 [#2] PREEMPT SMP
> [    1.340337] ---[ end trace ee076ceca10e9294 ]---
> [    1.340351] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> [    1.340351]
> [    1.340352] SMP: stopping secondary CPUs
> [    1.343408] Modules linked in:
> [    1.343819] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.345016] Hardware name: linux,dummy-virt (DT)
> [    1.345596] I-pipe domain: Linux
> [    1.346007] task: ffff80001d923e80 task.stack: ffff80001d93c000
> [    1.346768] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.347372] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.348018] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.348936] sp : ffff80001ef48fd0
> [    1.349357] x29: ffff80001ef48fd0 x28: ffff80001d93c000
> [    1.350057] x27: ffff80001d93c000 x26: ffff80001ef49090
> [    1.350779] x25: ffff80001ef450a0 x24: ffff000008d60d20
> [    1.351482] x23: ffff000008db2000 x22: 0000000000000003
> [    1.352186] x21: 0000000000000000 x20: ffff80001ef53f80
> [    1.352885] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.353585] x17: 000000000000000e x16: 0000000000000001
> [    1.354284] x15: 0000000000000019 x14: 0000000000000033
> [    1.355005] x13: 000000000000004c x12: 0000000000000068
> [    1.355710] x11: 0000000000000000 x10: ffff80001d923e88
> [    1.356411] x9 : ffff80001d93c000 x8 : ffff80001d924760
> [    1.357114] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.357818] x5 : 00008000161f3000 x4 : ffff000009090000
> [    1.358521] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.359239] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.359940]
> [    1.360141] Process swapper/4 (pid: 0, stack limit = 0xffff80001d93c038)
> [    1.360979] Stack: (0xffff80001ef48fd0 to 0xffff80001d940000)
> [    1.361699] Call trace:
> [    1.362018] Exception stack(0xffff80001ef48e70 to 0xffff80001ef48fa0)
> [    1.362842] 8e60:                                   ffff80001d804a00 0001000000000000
> [    1.363823] 8e80: ffff80001ef48fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.364801] 8ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.365780] 8ec0: ffff000009090000 00008000161f3000 003aa6a600000000 0000000000000001
> [    1.366774] 8ee0: ffff80001d924760 ffff80001d93c000 ffff80001d923e88 0000000000000000
> [    1.367752] 8f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.368730] 8f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.369707] 8f40: ffff80001ef53f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.370740] 8f60: ffff000008d60d20 ffff80001ef450a0 ffff80001ef49090 ffff80001d93c000
> [    1.371723] 8f80: ffff80001d93c000 ffff80001ef48fd0 ffff000008156e18 ffff80001ef48fd0
> [    1.372706] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.373442] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.374197] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.374916] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
> [    1.375726] fe00:                                   0000000000000000 00008000161f3000
> [    1.376708] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
> [    1.377688] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
> [    1.378685] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
> [    1.379668] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.380648] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
> [    1.381630] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
> [    1.382627] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
> [    1.383609] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
> [    1.384590] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.385572] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.386263] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
> [    1.387089] fe00:                                   0000000000000000 00008000161f3000
> [    1.388069] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
> [    1.389049] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
> [    1.390029] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
> [    1.391025] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.392005] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
> [    1.392986] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
> [    1.393967] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
> [    1.394962] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
> [    1.395941] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.396921] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.397538] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.398263] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.398958] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.399679] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.400421] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.401211] [<00000000408ea1a4>] 0x408ea1a4
> [    1.401745] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.402516] Internal error: undefined instruction: 0 [#3] PREEMPT SMP
> [    1.403371] Modules linked in:
> [    1.403780] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.404974] Hardware name: linux,dummy-virt (DT)
> [    1.405555] I-pipe domain: Linux
> [    1.405965] task: ffff80001d925780 task.stack: ffff80001d94c000
> [    1.406727] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.407329] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.407972] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.408889] sp : ffff80001ef7efd0
> [    1.409311] x29: ffff80001ef7efd0 x28: ffff80001d94c000
> [    1.410010] x27: ffff80001d94c000 x26: ffff80001ef7f090
> [    1.410726] x25: ffff80001ef7b0a0 x24: ffff000008d60d20
> [    1.411425] x23: ffff000008db2000 x22: 0000000000000003
> [    1.412122] x21: 0000000000000000 x20: ffff80001ef89f80
> [    1.412819] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.413516] x17: 000000000000000e x16: 0000000000000001
> [    1.414214] x15: 0000000000000019 x14: 0000000000000033
> [    1.414929] x13: 000000000000004c x12: 0000000000000068
> [    1.415623] x11: 0000000000000000 x10: ffff80001d925788
> [    1.416317] x9 : ffff80001d94c000 x8 : ffff80001d926060
> [    1.417011] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.417703] x5 : 0000800016229000 x4 : ffff0000090d0000
> [    1.418396] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.419107] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.419799]
> [    1.419996] Process swapper/6 (pid: 0, stack limit = 0xffff80001d94c038)
> [    1.420823] Stack: (0xffff80001ef7efd0 to 0xffff80001d950000)
> [    1.421533] Call trace:
> [    1.421846] Exception stack(0xffff80001ef7ee70 to 0xffff80001ef7efa0)
> [    1.422680] ee60:                                   ffff80001d804a00 0001000000000000
> [    1.423641] ee80: ffff80001ef7efd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.424601] eea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.425559] eec0: ffff0000090d0000 0000800016229000 003aa6a600000000 0000000000000001
> [    1.426518] eee0: ffff80001d926060 ffff80001d94c000 ffff80001d925788 0000000000000000
> [    1.427497] ef00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.428456] ef20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.429415] ef40: ffff80001ef89f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.430375] ef60: ffff000008d60d20 ffff80001ef7b0a0 ffff80001ef7f090 ffff80001d94c000
> [    1.431356] ef80: ffff80001d94c000 ffff80001ef7efd0 ffff000008156e18 ffff80001ef7efd0
> [    1.432324] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.433050] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.433796] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.434488] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
> [    1.435312] fe00:                                   0000000000000000 0000800016229000
> [    1.436272] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
> [    1.437234] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
> [    1.438194] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
> [    1.439174] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.440137] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
> [    1.441098] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
> [    1.442058] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
> [    1.443038] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
> [    1.443999] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.444959] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.445638] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
> [    1.446431] fe00:                                   0000000000000000 0000800016229000
> [    1.447407] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
> [    1.448368] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
> [    1.449328] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
> [    1.450288] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.451266] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
> [    1.452226] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
> [    1.453186] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
> [    1.454146] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
> [    1.455122] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.456083] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.456689] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.457399] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.458059] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.458808] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.459535] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.460310] [<00000000408ea1a4>] 0x408ea1a4
> [    1.460833] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.461589] Internal error: undefined instruction: 0 [#4] PREEMPT SMP
> [    1.462402] Modules linked in:
> [    1.462822] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.464019] Hardware name: linux,dummy-virt (DT)
> [    1.464599] I-pipe domain: Linux
> [    1.465010] task: ffff80001d921900 task.stack: ffff80001d930000
> [    1.465748] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.466347] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.467020] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.467929] sp : ffff80001eef7fd0
> [    1.468345] x29: ffff80001eef7fd0 x28: ffff80001d930000
> [    1.469038] x27: ffff80001d930000 x26: ffff80001eef8090
> [    1.469729] x25: ffff80001eef40a0 x24: ffff000008d60d20
> [    1.470420] x23: ffff000008db2000 x22: 0000000000000003
> [    1.471134] x21: 0000000000000000 x20: ffff80001ef02f80
> [    1.471828] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.472522] x17: 000000000000000e x16: 0000000000000001
> [    1.473215] x15: 0000000000000019 x14: 0000000000000033
> [    1.473908] x13: 000000000000004c x12: 0000000000000068
> [    1.474634] x11: 0000000000000000 x10: ffff80001d921908
> [    1.475328] x9 : ffff80001d930000 x8 : ffff80001d9221e0
> [    1.476022] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.476714] x5 : 00008000161a2000 x4 : ffff000009030000
> [    1.477407] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.478100] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.478822]
> [    1.479021] Process swapper/1 (pid: 0, stack limit = 0xffff80001d930038)
> [    1.479851] Stack: (0xffff80001eef7fd0 to 0xffff80001d934000)
> [    1.480563] Call trace:
> [    1.480877] Exception stack(0xffff80001eef7e70 to 0xffff80001eef7fa0)
> [    1.481675] 7e60:                                   ffff80001d804a00 0001000000000000
> [    1.482676] 7e80: ffff80001eef7fd0 ffff0000087899d0 00000000800001c5 ffff000008128070
> [    1.483640] 7ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.484603] 7ec0: ffff000009030000 00008000161a2000 003aa6a600000000 0000000000000001
> [    1.485565] 7ee0: ffff80001d9221e0 ffff80001d930000 ffff80001d921908 0000000000000000
> [    1.486526] 7f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.487516] 7f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.488480] 7f40: ffff80001ef02f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.489445] 7f60: ffff000008d60d20 ffff80001eef40a0 ffff80001eef8090 ffff80001d930000
> [    1.490411] 7f80: ffff80001d930000 ffff80001eef7fd0 ffff000008156e18 ffff80001eef7fd0
> [    1.491399] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.492124] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.492869] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.493562] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
> [    1.494360] 3e00:                                   0000000000000000 00008000161a2000
> [    1.495344] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
> [    1.496310] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
> [    1.497275] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
> [    1.498240] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.499221] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
> [    1.500189] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
> [    1.501156] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
> [    1.502122] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
> [    1.503107] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.504074] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.504756] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
> [    1.505555] 3e00:                                   0000000000000000 00008000161a2000
> [    1.506522] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
> [    1.507518] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
> [    1.508488] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
> [    1.509457] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.510426] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
> [    1.511427] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
> [    1.512398] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
> [    1.513366] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
> [    1.514334] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.515321] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.515930] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.516645] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.517309] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.518014] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.518769] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.519549] [<00000000408ea1a4>] 0x408ea1a4
> [    1.520076] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.520837] Internal error: undefined instruction: 0 [#5] PREEMPT SMP
> [    1.521669] Modules linked in:
> [    1.522075] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.523272] Hardware name: linux,dummy-virt (DT)
> [    1.523850] I-pipe domain: Linux
> [    1.524260] task: ffff80001d926400 task.stack: ffff80001d950000
> [    1.524999] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.525600] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.526242] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.527176] sp : ffff80001ef99fd0
> [    1.527595] x29: ffff80001ef99fd0 x28: ffff80001d950000
> [    1.528291] x27: ffff80001d950000 x26: ffff80001ef9a090
> [    1.528988] x25: ffff80001ef960a0 x24: ffff000008d60d20
> [    1.529685] x23: ffff000008db2000 x22: 0000000000000003
> [    1.530381] x21: 0000000000000000 x20: ffff80001efa4f80
> [    1.531143] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.531842] x17: 000000000000000e x16: 0000000000000001
> [    1.532540] x15: 0000000000000019 x14: 0000000000000033
> [    1.533236] x13: 000000000000004c x12: 0000000000000068
> [    1.533933] x11: 0000000000000000 x10: ffff80001d926408
> [    1.534655] x9 : ffff80001d950000 x8 : ffff80001d926ce0
> [    1.535354] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.536052] x5 : 0000800016244000 x4 : ffff0000090f0000
> [    1.536750] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.537449] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.538146]
> [    1.538347] Process swapper/7 (pid: 0, stack limit = 0xffff80001d950038)
> [    1.539207] Stack: (0xffff80001ef99fd0 to 0xffff80001d954000)
> [    1.539926] Call trace:
> [    1.540242] Exception stack(0xffff80001ef99e70 to 0xffff80001ef99fa0)
> [    1.541046] 9e60:                                   ffff80001d804a00 0001000000000000
> [    1.542019] 9e80: ffff80001ef99fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.543010] 9ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.543987] 9ec0: ffff0000090f0000 0000800016244000 003aa6a600000000 0000000000000001
> [    1.544963] 9ee0: ffff80001d926ce0 ffff80001d950000 ffff80001d926408 0000000000000000
> [    1.545940] 9f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.546947] 9f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.547905] 9f40: ffff80001efa4f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.548862] 9f60: ffff000008d60d20 ffff80001ef960a0 ffff80001ef9a090 ffff80001d950000
> [    1.549820] 9f80: ffff80001d950000 ffff80001ef99fd0 ffff000008156e18 ffff80001ef99fd0
> [    1.550799] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.551520] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.552260] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.552947] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
> [    1.553739] 3e00:                                   0000000000000000 0000800016244000
> [    1.554715] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
> [    1.555676] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
> [    1.556636] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
> [    1.557597] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.558573] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
> [    1.559537] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
> [    1.560496] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
> [    1.561454] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
> [    1.562412] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.563391] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.564068] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
> [    1.564859] 3e00:                                   0000000000000000 0000800016244000
> [    1.565816] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
> [    1.566797] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
> [    1.567758] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
> [    1.568718] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.569679] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
> [    1.570657] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
> [    1.571617] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
> [    1.572579] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
> [    1.573540] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.574502] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.575127] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.575837] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.576497] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.577198] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.577919] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.578716] [<00000000408ea1a4>] 0x408ea1a4
> [    1.579238] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.579992] Internal error: undefined instruction: 0 [#6] PREEMPT SMP
> [    1.580808] Modules linked in:
> [    1.581208] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.582377] Hardware name: linux,dummy-virt (DT)
> [    1.582954] I-pipe domain: Linux
> [    1.583356] task: ffff80001d923200 task.stack: ffff80001d938000
> [    1.584081] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.584671] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.585303] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.586203] sp : ffff80001ef2dfd0
> [    1.586633] x29: ffff80001ef2dfd0 x28: ffff80001d938000
> [    1.587320] x27: ffff80001d938000 x26: ffff80001ef2e090
> [    1.588006] x25: ffff80001ef2a0a0 x24: ffff000008d60d20
> [    1.588691] x23: ffff000008db2000 x22: 0000000000000003
> [    1.589377] x21: 0000000000000000 x20: ffff80001ef38f80
> [    1.590061] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.590763] x17: 000000000000000e x16: 0000000000000001
> [    1.591448] x15: 0000000000000019 x14: ffffffffffffffff
> [    1.592133] x13: ffffffffffffffff x12: 0000000000000000
> [    1.592817] x11: 0000000000000001 x10: ffff80001d923208
> [    1.593502] x9 : ffff80001d938000 x8 : ffff80001d923ae0
> [    1.594186] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.594888] x5 : 00008000161d8000 x4 : ffff000009070000
> [    1.595574] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.596259] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.596945]
> [    1.597140] Process swapper/3 (pid: 0, stack limit = 0xffff80001d938038)
> [    1.597960] Stack: (0xffff80001ef2dfd0 to 0xffff80001d93c000)
> [    1.598680] Call trace:
> [    1.598991] Exception stack(0xffff80001ef2de70 to 0xffff80001ef2dfa0)
> [    1.599781] de60:                                   ffff80001d804a00 0001000000000000
> [    1.600737] de80: ffff80001ef2dfd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.601693] dea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.602663] dec0: ffff000009070000 00008000161d8000 003aa6a600000000 0000000000000001
> [    1.603618] dee0: ffff80001d923ae0 ffff80001d938000 ffff80001d923208 0000000000000001
> [    1.604573] df00: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    1.605527] df20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.606483] df40: ffff80001ef38f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.607453] df60: ffff000008d60d20 ffff80001ef2a0a0 ffff80001ef2e090 ffff80001d938000
> [    1.608408] df80: ffff80001d938000 ffff80001ef2dfd0 ffff000008156e18 ffff80001ef2dfd0
> [    1.609363] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.610078] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.610831] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.611515] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
> [    1.612302] be00:                                   0000000000000000 00008000161d8000
> [    1.613256] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
> [    1.614209] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
> [    1.615179] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
> [    1.616134] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    1.617088] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
> [    1.618085] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
> [    1.619047] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
> [    1.620003] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
> [    1.620957] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.621912] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.622608] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
> [    1.623394] be00:                                   0000000000000000 00008000161d8000
> [    1.624342] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
> [    1.625291] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
> [    1.626240] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
> [    1.627208] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    1.628156] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
> [    1.629103] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
> [    1.630049] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
> [    1.631013] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
> [    1.631960] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.632908] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.633505] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.634206] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.634894] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.635595] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.636314] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.637082] [<00000000408ea1a4>] 0x408ea1a4
> [    1.637602] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.638352] Internal error: undefined instruction: 0 [#7] PREEMPT SMP
> [    2.174474] Modules linked in:
> [    2.174895] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    2.175975] Hardware name: linux,dummy-virt (DT)
> [    2.176500] I-pipe domain: Linux
> [    2.176873] task: ffff000008da0080 task.stack: ffff000008d90000
> [    2.177549] PC is at arch_itimer_ack_phys+0x0/0x18
> [    2.178097] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    2.178698] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    2.179532] sp : ffff80001eedcda0
> [    2.179914] x29: ffff80001eedcda0 x28: ffff000008d90000
> [    2.180548] x27: ffff000008d90000 x26: 0000000000000002
> [    2.181181] x25: ffff80001eeddbb4 x24: ffff000008d60d20
> [    2.181814] x23: ffff000008db2000 x22: 0000000000000003
> [    2.182447] x21: 0000000000000000 x20: ffff80001eee7f80
> [    2.183096] x19: ffff80001d804a00 x18: 0000000000000007
> [    2.183728] x17: 000000000000000e x16: 0000000000000001
> [    2.184361] x15: 0000000000000019 x14: ffffffffffffffff
> [    2.184994] x13: ffffffffffffffff x12: 0000000000000000
> [    2.185628] x11: 0000000000000004 x10: ffff80001eee0968
> [    2.186260] x9 : ffff80001eee08f0 x8 : ffff80001eee0978
> [    2.186909] x7 : ffff80001eee08e8 x6 : 0000000000000000
> [    2.187543] x5 : 0000800016187000 x4 : ffff000009010000
> [    2.188176] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    2.188810] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    2.189443]
> [    2.189625] Process swapper/0 (pid: 0, stack limit = 0xffff000008d90038)
> [    2.190382] Stack: (0xffff80001eedcda0 to 0xffff000008d94000)
> [    2.191047] Call trace:
> [    2.191334] Exception stack(0xffff80001eedcc40 to 0xffff80001eedcd70)
> [    2.192063] cc40: ffff80001d804a00 0001000000000000 ffff80001eedcda0 ffff0000087899d0
> [    2.192945] cc60: 00000000800001c5 0000000000000400 ffff0000087899d0 ffff0000083d4ed8
> [    2.193827] cc80: ffff0000083d5ae8 0000000008000000 ffff000009010000 0000800016187000
> [    2.194726] cca0: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.195607] ccc0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.196491] cce0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.197373] cd00: 0000000000000007 ffff80001d804a00 ffff80001eee7f80 0000000000000000
> [    2.198255] cd20: 0000000000000003 ffff000008db2000 ffff000008d60d20 ffff80001eeddbb4
> [    2.199152] cd40: 0000000000000002 ffff000008d90000 ffff000008d90000 ffff80001eedcda0
> [    2.200033] cd60: ffff000008156e18 ffff80001eedcda0
> [    2.200589] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    2.201251] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    2.201932] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    2.202579] Exception stack(0xffff80001eedce10 to 0xffff80001eedcf40)
> [    2.203307] ce00:                                   0000000000001fff ffff000008118c28
> [    2.204188] ce20: ffff80001eedcf80 ffff000008082fc8 ffff80001eedce50 0001000000000000
> [    2.205069] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
> [    2.205949] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
> [    2.206867] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.207743] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.208618] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.209494] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
> [    2.210369] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
> [    2.211269] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
> [    2.212147] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    2.212766] Exception stack(0xffff80001eedce30 to 0xffff80001eedcf60)
> [    2.213490] ce20:                                   ffff80001eedce50 0001000000000000
> [    2.214368] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
> [    2.215262] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
> [    2.216138] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.217015] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.217892] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.218785] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
> [    2.219664] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
> [    2.220543] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
> [    2.221422] cf40: ffff000008155fd4 ffff80001eedcf80 ffff000008155e6c 0000000020000145
> [    2.222300] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    2.222881] [<ffff000008155fd4>] __ipipe_do_sync_pipeline+0xa4/0xb8
> [    2.223587] [<ffff000008156730>] __ipipe_dispatch_irq+0x170/0x1d8
> [    2.224274] [<ffff000008081274>] __ipipe_grab_ipi+0x1c/0x30
> [    2.224902] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
> [    2.225626] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
> [    2.226501] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
> [    2.227442] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
> [    2.228329] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    2.229215] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
> [    2.230100] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
> [    2.231007] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
> [    2.231895] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
> [    2.232782] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
> [    2.233669] 3ee0: ffffffffffffffff ffff000008d60d20
> [    2.234228] [<ffff00000808178c>] gic_handle_irq+0x64/0x100
> [    2.234885] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
> [    2.235610] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
> [    2.236488] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
> [    2.237365] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
> [    2.238241] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    2.239151] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
> [    2.240028] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
> [    2.240906] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
> [    2.241783] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
> [    2.242686] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
> [    2.243565] 3ee0: ffffffffffffffff ffff000008d60d20
> [    2.244120] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    2.244675] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    2.245326] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    2.245932] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    2.246594] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    2.247256] [<ffff0000088de888>] rest_init+0x80/0x90
> [    2.247824] [<ffff000008c90b2c>] start_kernel+0x380/0x394
> [    2.248437] [<ffff000008c901d8>] __primary_switched+0x5c/0x64
> [    2.249090] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    2.249784] ---[ end trace ee076ceca10e9295 ]---
> [    2.837878] SMP: failed to stop secondary CPUs 0,2
> [    2.838465] Kernel Offset: disabled
> [    2.838887] Memory Limit: 512 MB
>

Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu

[gengdongjiu]

Sorry, I discription is not right, the Xenomai branch is next not stable 3.x.  Do you ever run it using kvm qemu? thanks a lot



发件人：Greg Gallagher
收件人：gengdongjiu,
抄 送：Jan Kiszka,Pintu Kumar,Philippe Gerum,Xenomai@xenomai.org,
时间：2018-06-21 21:48:00
主 题：Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu (e1000e)

For ARM64 you need to use the Xenomai next branch not the stable release.

-Greg

On Thu, Jun 21, 2018 at 8:27 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
> Hi
>    I use the stable 3.x Xenomai and ipipe-4.9.y, and run it using below qemu commands:
>  qemu-kvm -enable-kvm -m 1024 -cpu host -machine virt,gic-version=3  -smp 10 -nographic -kernel Image -append "rdinit=/init console=ttyAMA0 mem=512M root=/dev/ram0 earlycon=pl011,0x9000000 rw" -initrd guestfs_new.cpio.gz
>  it will happen below panic, do you know the reason? thanks
>
> [    1.286223] Hardware name: linux,dummy-virt (DT)
> [    1.286773] I-pipe domain: Linux
> [    1.287153] task: ffff80001d8d0000 task.stack: ffff80001d8a4000
> [    1.287840] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.288395] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.288982] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.289818] sp : ffff80001ef12fd0
> [    1.290201] x29: ffff80001ef12fd0 x28: ffff80001d8a4000
> [    1.290866] x27: 0000000000000003 x26: ffff80001ef13090
> [    1.291506] x25: ffff80001ef0f0a0 x24: ffff00000817ad00
> [    1.292147] x23: ffff000008db2000 x22: 0000000000000003
> [    1.292787] x21: 0000000000000000 x20: ffff80001ef1df80
> [    1.293427] x19: ffff80001d804a00 x18: 000000000000000e
> [    1.294066] x17: 00000000000045bf x16: 0140000000000000
> [    1.294716] x15: ffff000049f61000 x14: ffff80001effd290
> [    1.295357] x13: ffff000008e66078 x12: 0000000000000000
> [    1.295998] x11: 0000000000000000 x10: 000000003b9aca00
> [    1.296639] x9 : 0000000000000000 x8 : 0000000000000000
> [    1.297281] x7 : 0000000000000000 x6 : 0000000000000000
> [    1.297922] x5 : 00008000161bd000 x4 : ffff000009050000
> [    1.298582] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.299223] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.299865]
> [    1.300049] Process swapper/0 (pid: 1, stack limit = 0xffff80001d8a4038)
> [    1.300816] Stack: (0xffff80001ef12fd0 to 0xffff80001d8a8000)
> [    1.301474] Call trace:
> [    1.301763] Exception stack(0xffff80001ef12e70 to 0xffff80001ef12fa0)
> [    1.302500] 2e60:                                   ffff80001d804a00 0001000000000000
> [    1.303403] 2e80: ffff80001ef12fd0 ffff0000087899d0 00000000800001c5 ffff0000083d5b04
> [    1.304297] 2ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.305191] 2ec0: ffff000009050000 00008000161bd000 0000000000000000 0000000000000000
> [    1.306084] 2ee0: 0000000000000000 0000000000000000 000000003b9aca00 0000000000000000
> [    1.307000] 2f00: 0000000000000000 ffff000008e66078 ffff80001effd290 ffff000049f61000
> [    1.307894] 2f20: 0140000000000000 00000000000045bf 000000000000000e ffff80001d804a00
> [    1.308789] 2f40: ffff80001ef1df80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.309687] 2f60: ffff00000817ad00 ffff80001ef0f0a0 ffff80001ef13090 0000000000000003
> [    1.310592] 2f80: ffff80001d8a4000 ffff80001ef12fd0 ffff000008156e18 ffff80001ef12fd0
> [    1.311497] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.312176] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.312873] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.313519] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
> [    1.314262] 7b00:                                   0000000000000000 ffff000008eab1a8
> [    1.315181] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
> [    1.316081] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    1.316981] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
> [    1.317880] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
> [    1.318786] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
> [    1.319688] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
> [    1.320587] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
> [    1.321489] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
> [    1.322390] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
> [    1.323306] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.323942] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
> [    1.324687] 7b00:                                   0000000000000000 ffff000008eab1a8
> [    1.325588] 7b20: 0000000000000000 0000000000000001 ffff000008eab188 00000000ffffffff
> [    1.326490] 7b40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    1.327402] 7b60: 000000003b9aca00 0000000000000000 0000000000000000 ffff000008e66078
> [    1.328308] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000 00000000000045bf
> [    1.329212] 7ba0: 000000000000000e ffff000008eab000 0000000000000040 ffff000008170738
> [    1.330117] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00 0000000000000001
> [    1.331035] 7be0: ffff000008ecd000 0000000000000003 0000000000000000 ffff80001d8a7c40
> [    1.331933] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594 0000000020000045
> [    1.332830] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff 0000000000000040
> [    1.333730] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.334295] [<ffff000008155594>] ipipe_critical_exit+0x9c/0xa8
> [    1.334976] [<ffff0000081576a0>] ipipe_timer_start+0x90/0x1d0
> [    1.335654] [<ffff00000817b184>] xntimer_grab_hardware+0xfc/0x380
> [    1.336368] [<ffff000008ca2660>] xenomai_init+0x3c8/0x4e0
> [    1.336997] [<ffff000008083b08>] do_one_initcall+0x38/0x128
> [    1.337654] [<ffff000008c90cf8>] kernel_init_freeable+0x1b8/0x254
> [    1.338367] [<ffff0000088de8a8>] kernel_init+0x10/0x100
> [    1.338991] [<ffff0000080837e4>] ret_from_fork+0x14/0x30
> [    1.339609] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.340325] Internal error: undefined instruction: 0 [#2] PREEMPT SMP
> [    1.340337] ---[ end trace ee076ceca10e9294 ]---
> [    1.340351] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> [    1.340351]
> [    1.340352] SMP: stopping secondary CPUs
> [    1.343408] Modules linked in:
> [    1.343819] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.345016] Hardware name: linux,dummy-virt (DT)
> [    1.345596] I-pipe domain: Linux
> [    1.346007] task: ffff80001d923e80 task.stack: ffff80001d93c000
> [    1.346768] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.347372] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.348018] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.348936] sp : ffff80001ef48fd0
> [    1.349357] x29: ffff80001ef48fd0 x28: ffff80001d93c000
> [    1.350057] x27: ffff80001d93c000 x26: ffff80001ef49090
> [    1.350779] x25: ffff80001ef450a0 x24: ffff000008d60d20
> [    1.351482] x23: ffff000008db2000 x22: 0000000000000003
> [    1.352186] x21: 0000000000000000 x20: ffff80001ef53f80
> [    1.352885] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.353585] x17: 000000000000000e x16: 0000000000000001
> [    1.354284] x15: 0000000000000019 x14: 0000000000000033
> [    1.355005] x13: 000000000000004c x12: 0000000000000068
> [    1.355710] x11: 0000000000000000 x10: ffff80001d923e88
> [    1.356411] x9 : ffff80001d93c000 x8 : ffff80001d924760
> [    1.357114] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.357818] x5 : 00008000161f3000 x4 : ffff000009090000
> [    1.358521] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.359239] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.359940]
> [    1.360141] Process swapper/4 (pid: 0, stack limit = 0xffff80001d93c038)
> [    1.360979] Stack: (0xffff80001ef48fd0 to 0xffff80001d940000)
> [    1.361699] Call trace:
> [    1.362018] Exception stack(0xffff80001ef48e70 to 0xffff80001ef48fa0)
> [    1.362842] 8e60:                                   ffff80001d804a00 0001000000000000
> [    1.363823] 8e80: ffff80001ef48fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.364801] 8ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.365780] 8ec0: ffff000009090000 00008000161f3000 003aa6a600000000 0000000000000001
> [    1.366774] 8ee0: ffff80001d924760 ffff80001d93c000 ffff80001d923e88 0000000000000000
> [    1.367752] 8f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.368730] 8f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.369707] 8f40: ffff80001ef53f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.370740] 8f60: ffff000008d60d20 ffff80001ef450a0 ffff80001ef49090 ffff80001d93c000
> [    1.371723] 8f80: ffff80001d93c000 ffff80001ef48fd0 ffff000008156e18 ffff80001ef48fd0
> [    1.372706] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.373442] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.374197] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.374916] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
> [    1.375726] fe00:                                   0000000000000000 00008000161f3000
> [    1.376708] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
> [    1.377688] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
> [    1.378685] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
> [    1.379668] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.380648] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
> [    1.381630] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
> [    1.382627] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
> [    1.383609] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
> [    1.384590] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.385572] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.386263] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
> [    1.387089] fe00:                                   0000000000000000 00008000161f3000
> [    1.388069] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000 00008000161f3000
> [    1.389049] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760 ffff80001d93c000
> [    1.390029] fe60: ffff80001d923e88 0000000000000000 0000000000000068 000000000000004c
> [    1.391025] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.392005] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000010
> [    1.392986] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d93c000
> [    1.393967] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000 ffff80001d93ff40
> [    1.394962] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c 0000000060000045
> [    1.395941] ff20: 0000000000000010 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.396921] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.397538] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.398263] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.398958] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.399679] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.400421] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.401211] [<00000000408ea1a4>] 0x408ea1a4
> [    1.401745] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.402516] Internal error: undefined instruction: 0 [#3] PREEMPT SMP
> [    1.403371] Modules linked in:
> [    1.403780] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.404974] Hardware name: linux,dummy-virt (DT)
> [    1.405555] I-pipe domain: Linux
> [    1.405965] task: ffff80001d925780 task.stack: ffff80001d94c000
> [    1.406727] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.407329] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.407972] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.408889] sp : ffff80001ef7efd0
> [    1.409311] x29: ffff80001ef7efd0 x28: ffff80001d94c000
> [    1.410010] x27: ffff80001d94c000 x26: ffff80001ef7f090
> [    1.410726] x25: ffff80001ef7b0a0 x24: ffff000008d60d20
> [    1.411425] x23: ffff000008db2000 x22: 0000000000000003
> [    1.412122] x21: 0000000000000000 x20: ffff80001ef89f80
> [    1.412819] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.413516] x17: 000000000000000e x16: 0000000000000001
> [    1.414214] x15: 0000000000000019 x14: 0000000000000033
> [    1.414929] x13: 000000000000004c x12: 0000000000000068
> [    1.415623] x11: 0000000000000000 x10: ffff80001d925788
> [    1.416317] x9 : ffff80001d94c000 x8 : ffff80001d926060
> [    1.417011] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.417703] x5 : 0000800016229000 x4 : ffff0000090d0000
> [    1.418396] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.419107] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.419799]
> [    1.419996] Process swapper/6 (pid: 0, stack limit = 0xffff80001d94c038)
> [    1.420823] Stack: (0xffff80001ef7efd0 to 0xffff80001d950000)
> [    1.421533] Call trace:
> [    1.421846] Exception stack(0xffff80001ef7ee70 to 0xffff80001ef7efa0)
> [    1.422680] ee60:                                   ffff80001d804a00 0001000000000000
> [    1.423641] ee80: ffff80001ef7efd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.424601] eea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.425559] eec0: ffff0000090d0000 0000800016229000 003aa6a600000000 0000000000000001
> [    1.426518] eee0: ffff80001d926060 ffff80001d94c000 ffff80001d925788 0000000000000000
> [    1.427497] ef00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.428456] ef20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.429415] ef40: ffff80001ef89f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.430375] ef60: ffff000008d60d20 ffff80001ef7b0a0 ffff80001ef7f090 ffff80001d94c000
> [    1.431356] ef80: ffff80001d94c000 ffff80001ef7efd0 ffff000008156e18 ffff80001ef7efd0
> [    1.432324] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.433050] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.433796] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.434488] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
> [    1.435312] fe00:                                   0000000000000000 0000800016229000
> [    1.436272] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
> [    1.437234] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
> [    1.438194] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
> [    1.439174] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.440137] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
> [    1.441098] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
> [    1.442058] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
> [    1.443038] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
> [    1.443999] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.444959] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.445638] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
> [    1.446431] fe00:                                   0000000000000000 0000800016229000
> [    1.447407] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000 0000800016229000
> [    1.448368] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060 ffff80001d94c000
> [    1.449328] fe60: ffff80001d925788 0000000000000000 0000000000000068 000000000000004c
> [    1.450288] fe80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.451266] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000040
> [    1.452226] fec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d94c000
> [    1.453186] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000 ffff80001d94ff40
> [    1.454146] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c 0000000060000045
> [    1.455122] ff20: 0000000000000040 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.456083] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.456689] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.457399] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.458059] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.458808] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.459535] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.460310] [<00000000408ea1a4>] 0x408ea1a4
> [    1.460833] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.461589] Internal error: undefined instruction: 0 [#4] PREEMPT SMP
> [    1.462402] Modules linked in:
> [    1.462822] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.464019] Hardware name: linux,dummy-virt (DT)
> [    1.464599] I-pipe domain: Linux
> [    1.465010] task: ffff80001d921900 task.stack: ffff80001d930000
> [    1.465748] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.466347] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.467020] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.467929] sp : ffff80001eef7fd0
> [    1.468345] x29: ffff80001eef7fd0 x28: ffff80001d930000
> [    1.469038] x27: ffff80001d930000 x26: ffff80001eef8090
> [    1.469729] x25: ffff80001eef40a0 x24: ffff000008d60d20
> [    1.470420] x23: ffff000008db2000 x22: 0000000000000003
> [    1.471134] x21: 0000000000000000 x20: ffff80001ef02f80
> [    1.471828] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.472522] x17: 000000000000000e x16: 0000000000000001
> [    1.473215] x15: 0000000000000019 x14: 0000000000000033
> [    1.473908] x13: 000000000000004c x12: 0000000000000068
> [    1.474634] x11: 0000000000000000 x10: ffff80001d921908
> [    1.475328] x9 : ffff80001d930000 x8 : ffff80001d9221e0
> [    1.476022] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.476714] x5 : 00008000161a2000 x4 : ffff000009030000
> [    1.477407] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.478100] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.478822]
> [    1.479021] Process swapper/1 (pid: 0, stack limit = 0xffff80001d930038)
> [    1.479851] Stack: (0xffff80001eef7fd0 to 0xffff80001d934000)
> [    1.480563] Call trace:
> [    1.480877] Exception stack(0xffff80001eef7e70 to 0xffff80001eef7fa0)
> [    1.481675] 7e60:                                   ffff80001d804a00 0001000000000000
> [    1.482676] 7e80: ffff80001eef7fd0 ffff0000087899d0 00000000800001c5 ffff000008128070
> [    1.483640] 7ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.484603] 7ec0: ffff000009030000 00008000161a2000 003aa6a600000000 0000000000000001
> [    1.485565] 7ee0: ffff80001d9221e0 ffff80001d930000 ffff80001d921908 0000000000000000
> [    1.486526] 7f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.487516] 7f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.488480] 7f40: ffff80001ef02f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.489445] 7f60: ffff000008d60d20 ffff80001eef40a0 ffff80001eef8090 ffff80001d930000
> [    1.490411] 7f80: ffff80001d930000 ffff80001eef7fd0 ffff000008156e18 ffff80001eef7fd0
> [    1.491399] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.492124] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.492869] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.493562] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
> [    1.494360] 3e00:                                   0000000000000000 00008000161a2000
> [    1.495344] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
> [    1.496310] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
> [    1.497275] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
> [    1.498240] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.499221] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
> [    1.500189] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
> [    1.501156] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
> [    1.502122] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
> [    1.503107] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.504074] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.504756] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
> [    1.505555] 3e00:                                   0000000000000000 00008000161a2000
> [    1.506522] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000 00008000161a2000
> [    1.507518] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0 ffff80001d930000
> [    1.508488] 3e60: ffff80001d921908 0000000000000000 0000000000000068 000000000000004c
> [    1.509457] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.510426] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000002
> [    1.511427] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d930000
> [    1.512398] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000 ffff80001d933f40
> [    1.513366] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c 0000000060000045
> [    1.514334] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.515321] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.515930] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.516645] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.517309] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.518014] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.518769] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.519549] [<00000000408ea1a4>] 0x408ea1a4
> [    1.520076] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.520837] Internal error: undefined instruction: 0 [#5] PREEMPT SMP
> [    1.521669] Modules linked in:
> [    1.522075] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.523272] Hardware name: linux,dummy-virt (DT)
> [    1.523850] I-pipe domain: Linux
> [    1.524260] task: ffff80001d926400 task.stack: ffff80001d950000
> [    1.524999] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.525600] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.526242] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.527176] sp : ffff80001ef99fd0
> [    1.527595] x29: ffff80001ef99fd0 x28: ffff80001d950000
> [    1.528291] x27: ffff80001d950000 x26: ffff80001ef9a090
> [    1.528988] x25: ffff80001ef960a0 x24: ffff000008d60d20
> [    1.529685] x23: ffff000008db2000 x22: 0000000000000003
> [    1.530381] x21: 0000000000000000 x20: ffff80001efa4f80
> [    1.531143] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.531842] x17: 000000000000000e x16: 0000000000000001
> [    1.532540] x15: 0000000000000019 x14: 0000000000000033
> [    1.533236] x13: 000000000000004c x12: 0000000000000068
> [    1.533933] x11: 0000000000000000 x10: ffff80001d926408
> [    1.534655] x9 : ffff80001d950000 x8 : ffff80001d926ce0
> [    1.535354] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.536052] x5 : 0000800016244000 x4 : ffff0000090f0000
> [    1.536750] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.537449] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.538146]
> [    1.538347] Process swapper/7 (pid: 0, stack limit = 0xffff80001d950038)
> [    1.539207] Stack: (0xffff80001ef99fd0 to 0xffff80001d954000)
> [    1.539926] Call trace:
> [    1.540242] Exception stack(0xffff80001ef99e70 to 0xffff80001ef99fa0)
> [    1.541046] 9e60:                                   ffff80001d804a00 0001000000000000
> [    1.542019] 9e80: ffff80001ef99fd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.543010] 9ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.543987] 9ec0: ffff0000090f0000 0000800016244000 003aa6a600000000 0000000000000001
> [    1.544963] 9ee0: ffff80001d926ce0 ffff80001d950000 ffff80001d926408 0000000000000000
> [    1.545940] 9f00: 0000000000000068 000000000000004c 0000000000000033 0000000000000019
> [    1.546947] 9f20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.547905] 9f40: ffff80001efa4f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.548862] 9f60: ffff000008d60d20 ffff80001ef960a0 ffff80001ef9a090 ffff80001d950000
> [    1.549820] 9f80: ffff80001d950000 ffff80001ef99fd0 ffff000008156e18 ffff80001ef99fd0
> [    1.550799] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.551520] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.552260] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.552947] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
> [    1.553739] 3e00:                                   0000000000000000 0000800016244000
> [    1.554715] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
> [    1.555676] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
> [    1.556636] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
> [    1.557597] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.558573] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
> [    1.559537] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
> [    1.560496] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
> [    1.561454] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
> [    1.562412] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.563391] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.564068] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
> [    1.564859] 3e00:                                   0000000000000000 0000800016244000
> [    1.565816] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000 0000800016244000
> [    1.566797] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0 ffff80001d950000
> [    1.567758] 3e60: ffff80001d926408 0000000000000000 0000000000000068 000000000000004c
> [    1.568718] 3e80: 0000000000000033 0000000000000019 0000000000000001 000000000000000e
> [    1.569679] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000080
> [    1.570657] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d950000
> [    1.571617] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000 ffff80001d953f40
> [    1.572579] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c 0000000060000045
> [    1.573540] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.574502] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.575127] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.575837] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.576497] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.577198] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.577919] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.578716] [<00000000408ea1a4>] 0x408ea1a4
> [    1.579238] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.579992] Internal error: undefined instruction: 0 [#6] PREEMPT SMP
> [    1.580808] Modules linked in:
> [    1.581208] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    1.582377] Hardware name: linux,dummy-virt (DT)
> [    1.582954] I-pipe domain: Linux
> [    1.583356] task: ffff80001d923200 task.stack: ffff80001d938000
> [    1.584081] PC is at arch_itimer_ack_phys+0x0/0x18
> [    1.584671] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    1.585303] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    1.586203] sp : ffff80001ef2dfd0
> [    1.586633] x29: ffff80001ef2dfd0 x28: ffff80001d938000
> [    1.587320] x27: ffff80001d938000 x26: ffff80001ef2e090
> [    1.588006] x25: ffff80001ef2a0a0 x24: ffff000008d60d20
> [    1.588691] x23: ffff000008db2000 x22: 0000000000000003
> [    1.589377] x21: 0000000000000000 x20: ffff80001ef38f80
> [    1.590061] x19: ffff80001d804a00 x18: 0000000000000007
> [    1.590763] x17: 000000000000000e x16: 0000000000000001
> [    1.591448] x15: 0000000000000019 x14: ffffffffffffffff
> [    1.592133] x13: ffffffffffffffff x12: 0000000000000000
> [    1.592817] x11: 0000000000000001 x10: ffff80001d923208
> [    1.593502] x9 : ffff80001d938000 x8 : ffff80001d923ae0
> [    1.594186] x7 : 0000000000000001 x6 : 003aa6a600000000
> [    1.594888] x5 : 00008000161d8000 x4 : ffff000009070000
> [    1.595574] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    1.596259] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    1.596945]
> [    1.597140] Process swapper/3 (pid: 0, stack limit = 0xffff80001d938038)
> [    1.597960] Stack: (0xffff80001ef2dfd0 to 0xffff80001d93c000)
> [    1.598680] Call trace:
> [    1.598991] Exception stack(0xffff80001ef2de70 to 0xffff80001ef2dfa0)
> [    1.599781] de60:                                   ffff80001d804a00 0001000000000000
> [    1.600737] de80: ffff80001ef2dfd0 ffff0000087899d0 00000000800001c5 ffff00000811e52c
> [    1.601693] dea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8 0000000008000000
> [    1.602663] dec0: ffff000009070000 00008000161d8000 003aa6a600000000 0000000000000001
> [    1.603618] dee0: ffff80001d923ae0 ffff80001d938000 ffff80001d923208 0000000000000001
> [    1.604573] df00: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    1.605527] df20: 0000000000000001 000000000000000e 0000000000000007 ffff80001d804a00
> [    1.606483] df40: ffff80001ef38f80 0000000000000000 0000000000000003 ffff000008db2000
> [    1.607453] df60: ffff000008d60d20 ffff80001ef2a0a0 ffff80001ef2e090 ffff80001d938000
> [    1.608408] df80: ffff80001d938000 ffff80001ef2dfd0 ffff000008156e18 ffff80001ef2dfd0
> [    1.609363] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    1.610078] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    1.610831] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    1.611515] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
> [    1.612302] be00:                                   0000000000000000 00008000161d8000
> [    1.613256] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
> [    1.614209] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
> [    1.615179] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
> [    1.616134] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    1.617088] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
> [    1.618085] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
> [    1.619047] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
> [    1.620003] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
> [    1.620957] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.621912] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    1.622608] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
> [    1.623394] be00:                                   0000000000000000 00008000161d8000
> [    1.624342] be20: 0000000000000000 ffff80001ef318e8 0000000000000000 00008000161d8000
> [    1.625291] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0 ffff80001d938000
> [    1.626240] be60: ffff80001d923208 0000000000000001 0000000000000000 ffffffffffffffff
> [    1.627208] be80: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    1.628156] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8 0000000000000008
> [    1.629103] bec0: 0000000000000000 0000000000000000 ffff000008d60d20 ffff80001d938000
> [    1.630049] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000 ffff80001d93bf40
> [    1.631013] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c 0000000060000045
> [    1.631960] bf20: 0000000000000008 0000000000000000 ffffffffffffffff ffff000008d60d20
> [    1.632908] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    1.633505] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    1.634206] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    1.634894] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    1.635595] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    1.636314] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
> [    1.637082] [<00000000408ea1a4>] 0x408ea1a4
> [    1.637602] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    1.638352] Internal error: undefined instruction: 0 [#7] PREEMPT SMP
> [    2.174474] Modules linked in:
> [    2.174895] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D         4.9.92-ipipe-00107-ge02d0928-dirty #10
> [    2.175975] Hardware name: linux,dummy-virt (DT)
> [    2.176500] I-pipe domain: Linux
> [    2.176873] task: ffff000008da0080 task.stack: ffff000008d90000
> [    2.177549] PC is at arch_itimer_ack_phys+0x0/0x18
> [    2.178097] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
> [    2.178698] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate: 800001c5
> [    2.179532] sp : ffff80001eedcda0
> [    2.179914] x29: ffff80001eedcda0 x28: ffff000008d90000
> [    2.180548] x27: ffff000008d90000 x26: 0000000000000002
> [    2.181181] x25: ffff80001eeddbb4 x24: ffff000008d60d20
> [    2.181814] x23: ffff000008db2000 x22: 0000000000000003
> [    2.182447] x21: 0000000000000000 x20: ffff80001eee7f80
> [    2.183096] x19: ffff80001d804a00 x18: 0000000000000007
> [    2.183728] x17: 000000000000000e x16: 0000000000000001
> [    2.184361] x15: 0000000000000019 x14: ffffffffffffffff
> [    2.184994] x13: ffffffffffffffff x12: 0000000000000000
> [    2.185628] x11: 0000000000000004 x10: ffff80001eee0968
> [    2.186260] x9 : ffff80001eee08f0 x8 : ffff80001eee0978
> [    2.186909] x7 : ffff80001eee08e8 x6 : 0000000000000000
> [    2.187543] x5 : 0000800016187000 x4 : ffff000009010000
> [    2.188176] x3 : 0000000008000000 x2 : ffff0000083d5ae8
> [    2.188810] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
> [    2.189443]
> [    2.189625] Process swapper/0 (pid: 0, stack limit = 0xffff000008d90038)
> [    2.190382] Stack: (0xffff80001eedcda0 to 0xffff000008d94000)
> [    2.191047] Call trace:
> [    2.191334] Exception stack(0xffff80001eedcc40 to 0xffff80001eedcd70)
> [    2.192063] cc40: ffff80001d804a00 0001000000000000 ffff80001eedcda0 ffff0000087899d0
> [    2.192945] cc60: 00000000800001c5 0000000000000400 ffff0000087899d0 ffff0000083d4ed8
> [    2.193827] cc80: ffff0000083d5ae8 0000000008000000 ffff000009010000 0000800016187000
> [    2.194726] cca0: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.195607] ccc0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.196491] cce0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.197373] cd00: 0000000000000007 ffff80001d804a00 ffff80001eee7f80 0000000000000000
> [    2.198255] cd20: 0000000000000003 ffff000008db2000 ffff000008d60d20 ffff80001eeddbb4
> [    2.199152] cd40: 0000000000000002 ffff000008d90000 ffff000008d90000 ffff80001eedcda0
> [    2.200033] cd60: ffff000008156e18 ffff80001eedcda0
> [    2.200589] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
> [    2.201251] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
> [    2.201932] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
> [    2.202579] Exception stack(0xffff80001eedce10 to 0xffff80001eedcf40)
> [    2.203307] ce00:                                   0000000000001fff ffff000008118c28
> [    2.204188] ce20: ffff80001eedcf80 ffff000008082fc8 ffff80001eedce50 0001000000000000
> [    2.205069] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
> [    2.205949] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
> [    2.206867] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.207743] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.208618] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.209494] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
> [    2.210369] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
> [    2.211269] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
> [    2.212147] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
> [    2.212766] Exception stack(0xffff80001eedce30 to 0xffff80001eedcf60)
> [    2.213490] ce20:                                   ffff80001eedce50 0001000000000000
> [    2.214368] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402 ffff000008eab880
> [    2.215262] ce60: 0000000000000000 0000000000000002 0000000000010000 0000000000000000
> [    2.216138] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978 ffff80001eee08f0
> [    2.217015] cea0: ffff80001eee0968 0000000000000004 0000000000000000 ffffffffffffffff
> [    2.217892] cec0: ffffffffffffffff 0000000000000019 0000000000000001 000000000000000e
> [    2.218785] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080 0000000000000001
> [    2.219664] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20 ffff80001eed90a0
> [    2.220543] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000 ffff80001eedcf80
> [    2.221422] cf40: ffff000008155fd4 ffff80001eedcf80 ffff000008155e6c 0000000020000145
> [    2.222300] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    2.222881] [<ffff000008155fd4>] __ipipe_do_sync_pipeline+0xa4/0xb8
> [    2.223587] [<ffff000008156730>] __ipipe_dispatch_irq+0x170/0x1d8
> [    2.224274] [<ffff000008081274>] __ipipe_grab_ipi+0x1c/0x30
> [    2.224902] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
> [    2.225626] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
> [    2.226501] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
> [    2.227442] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
> [    2.228329] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    2.229215] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
> [    2.230100] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
> [    2.231007] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
> [    2.231895] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
> [    2.232782] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
> [    2.233669] 3ee0: ffffffffffffffff ffff000008d60d20
> [    2.234228] [<ffff00000808178c>] gic_handle_irq+0x64/0x100
> [    2.234885] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
> [    2.235610] 3dc0: 0000000000000000 0000800016187000 0000000000000000 ffff80001eee08e8
> [    2.236488] 3de0: 0000000000000000 0000000000000020 003aa6a600000000 0000000000000001
> [    2.237365] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088 0000000000000001
> [    2.238241] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff 0000000000000019
> [    2.239151] 3e40: 0000000000000001 000000000000000e 0000000000000007 ffff000008d97258
> [    2.240028] 3e60: ffff000008d972e8 0000000000000001 0000000000000000 0000000000000000
> [    2.240906] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555 ffff000008d90000
> [    2.241783] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28 ffff000008d93ef0
> [    2.242686] 3ec0: ffff00000815601c 0000000060000045 0000000000000001 0000000000000000
> [    2.243565] 3ee0: ffffffffffffffff ffff000008d60d20
> [    2.244120] [<ffff000008082fc8>] el1_irq+0xc8/0x108
> [    2.244675] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
> [    2.245326] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
> [    2.245932] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
> [    2.246594] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
> [    2.247256] [<ffff0000088de888>] rest_init+0x80/0x90
> [    2.247824] [<ffff000008c90b2c>] start_kernel+0x380/0x394
> [    2.248437] [<ffff000008c901d8>] __primary_switched+0x5c/0x64
> [    2.249090] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
> [    2.249784] ---[ end trace ee076ceca10e9295 ]---
> [    2.837878] SMP: failed to stop secondary CPUs 0,2
> [    2.838465] Kernel Offset: disabled
> [    2.838887] Memory Limit: 512 MB
>

Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu

[Greg Gallagher]

Hi,
  To the best of my knowledge we don't run it with qemu often.  Boards
that have been run used in the past have been rpi3 (64-bit mode),
hikey-960 and there are some online resources for the dragon board
410c.  I'm currently testing the raspberry pi3 on the new ARM64 4.14
ipipe branch.  Hopefully i'll have some results in the next two week
or so.

-Greg

On Thu, Jun 21, 2018 at 9:56 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
> Sorry, I discription is not right, the Xenomai branch is next not stable
> 3.x.  Do you ever run it using kvm qemu? thanks a lot
>
>
>
> 发件人：Greg Gallagher
> 收件人：gengdongjiu,
> 抄 送：Jan Kiszka,Pintu Kumar,Philippe Gerum,Xenomai@xenomai.org,
> 时间：2018-06-21 21:48:00
> 主 题：Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu (e1000e)
>
> For ARM64 you need to use the Xenomai next branch not the stable release.
>
> -Greg
>
> On Thu, Jun 21, 2018 at 8:27 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
>> Hi
>>    I use the stable 3.x Xenomai and ipipe-4.9.y, and run it using below
>> qemu commands:
>>  qemu-kvm -enable-kvm -m 1024 -cpu host -machine virt,gic-version=3  -smp
>> 10 -nographic -kernel Image -append "rdinit=/init console=ttyAMA0 mem=512M
>> root=/dev/ram0 earlycon=pl011,0x9000000 rw" -initrd guestfs_new.cpio.gz
>>  it will happen below panic, do you know the reason? thanks
>>
>> [    1.286223] Hardware name: linux,dummy-virt (DT)
>> [    1.286773] I-pipe domain: Linux
>> [    1.287153] task: ffff80001d8d0000 task.stack: ffff80001d8a4000
>> [    1.287840] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.288395] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.288982] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.289818] sp : ffff80001ef12fd0
>> [    1.290201] x29: ffff80001ef12fd0 x28: ffff80001d8a4000
>> [    1.290866] x27: 0000000000000003 x26: ffff80001ef13090
>> [    1.291506] x25: ffff80001ef0f0a0 x24: ffff00000817ad00
>> [    1.292147] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.292787] x21: 0000000000000000 x20: ffff80001ef1df80
>> [    1.293427] x19: ffff80001d804a00 x18: 000000000000000e
>> [    1.294066] x17: 00000000000045bf x16: 0140000000000000
>> [    1.294716] x15: ffff000049f61000 x14: ffff80001effd290
>> [    1.295357] x13: ffff000008e66078 x12: 0000000000000000
>> [    1.295998] x11: 0000000000000000 x10: 000000003b9aca00
>> [    1.296639] x9 : 0000000000000000 x8 : 0000000000000000
>> [    1.297281] x7 : 0000000000000000 x6 : 0000000000000000
>> [    1.297922] x5 : 00008000161bd000 x4 : ffff000009050000
>> [    1.298582] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.299223] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.299865]
>> [    1.300049] Process swapper/0 (pid: 1, stack limit =
>> 0xffff80001d8a4038)
>> [    1.300816] Stack: (0xffff80001ef12fd0 to 0xffff80001d8a8000)
>> [    1.301474] Call trace:
>> [    1.301763] Exception stack(0xffff80001ef12e70 to 0xffff80001ef12fa0)
>> [    1.302500] 2e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.303403] 2e80: ffff80001ef12fd0 ffff0000087899d0 00000000800001c5
>> ffff0000083d5b04
>> [    1.304297] 2ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.305191] 2ec0: ffff000009050000 00008000161bd000 0000000000000000
>> 0000000000000000
>> [    1.306084] 2ee0: 0000000000000000 0000000000000000 000000003b9aca00
>> 0000000000000000
>> [    1.307000] 2f00: 0000000000000000 ffff000008e66078 ffff80001effd290
>> ffff000049f61000
>> [    1.307894] 2f20: 0140000000000000 00000000000045bf 000000000000000e
>> ffff80001d804a00
>> [    1.308789] 2f40: ffff80001ef1df80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.309687] 2f60: ffff00000817ad00 ffff80001ef0f0a0 ffff80001ef13090
>> 0000000000000003
>> [    1.310592] 2f80: ffff80001d8a4000 ffff80001ef12fd0 ffff000008156e18
>> ffff80001ef12fd0
>> [    1.311497] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.312176] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.312873] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.313519] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
>> [    1.314262] 7b00:                                   0000000000000000
>> ffff000008eab1a8
>> [    1.315181] 7b20: 0000000000000000 0000000000000001 ffff000008eab188
>> 00000000ffffffff
>> [    1.316081] 7b40: 0000000000000000 0000000000000000 0000000000000000
>> 0000000000000000
>> [    1.316981] 7b60: 000000003b9aca00 0000000000000000 0000000000000000
>> ffff000008e66078
>> [    1.317880] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000
>> 00000000000045bf
>> [    1.318786] 7ba0: 000000000000000e ffff000008eab000 0000000000000040
>> ffff000008170738
>> [    1.319688] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00
>> 0000000000000001
>> [    1.320587] 7be0: ffff000008ecd000 0000000000000003 0000000000000000
>> ffff80001d8a7c40
>> [    1.321489] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594
>> 0000000020000045
>> [    1.322390] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff
>> 0000000000000040
>> [    1.323306] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.323942] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
>> [    1.324687] 7b00:                                   0000000000000000
>> ffff000008eab1a8
>> [    1.325588] 7b20: 0000000000000000 0000000000000001 ffff000008eab188
>> 00000000ffffffff
>> [    1.326490] 7b40: 0000000000000000 0000000000000000 0000000000000000
>> 0000000000000000
>> [    1.327402] 7b60: 000000003b9aca00 0000000000000000 0000000000000000
>> ffff000008e66078
>> [    1.328308] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000
>> 00000000000045bf
>> [    1.329212] 7ba0: 000000000000000e ffff000008eab000 0000000000000040
>> ffff000008170738
>> [    1.330117] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00
>> 0000000000000001
>> [    1.331035] 7be0: ffff000008ecd000 0000000000000003 0000000000000000
>> ffff80001d8a7c40
>> [    1.331933] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594
>> 0000000020000045
>> [    1.332830] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff
>> 0000000000000040
>> [    1.333730] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.334295] [<ffff000008155594>] ipipe_critical_exit+0x9c/0xa8
>> [    1.334976] [<ffff0000081576a0>] ipipe_timer_start+0x90/0x1d0
>> [    1.335654] [<ffff00000817b184>] xntimer_grab_hardware+0xfc/0x380
>> [    1.336368] [<ffff000008ca2660>] xenomai_init+0x3c8/0x4e0
>> [    1.336997] [<ffff000008083b08>] do_one_initcall+0x38/0x128
>> [    1.337654] [<ffff000008c90cf8>] kernel_init_freeable+0x1b8/0x254
>> [    1.338367] [<ffff0000088de8a8>] kernel_init+0x10/0x100
>> [    1.338991] [<ffff0000080837e4>] ret_from_fork+0x14/0x30
>> [    1.339609] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.340325] Internal error: undefined instruction: 0 [#2] PREEMPT SMP
>> [    1.340337] ---[ end trace ee076ceca10e9294 ]---
>> [    1.340351] Kernel panic - not syncing: Attempted to kill init!
>> exitcode=0x0000000b
>> [    1.340351]
>> [    1.340352] SMP: stopping secondary CPUs
>> [    1.343408] Modules linked in:
>> [    1.343819] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.345016] Hardware name: linux,dummy-virt (DT)
>> [    1.345596] I-pipe domain: Linux
>> [    1.346007] task: ffff80001d923e80 task.stack: ffff80001d93c000
>> [    1.346768] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.347372] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.348018] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.348936] sp : ffff80001ef48fd0
>> [    1.349357] x29: ffff80001ef48fd0 x28: ffff80001d93c000
>> [    1.350057] x27: ffff80001d93c000 x26: ffff80001ef49090
>> [    1.350779] x25: ffff80001ef450a0 x24: ffff000008d60d20
>> [    1.351482] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.352186] x21: 0000000000000000 x20: ffff80001ef53f80
>> [    1.352885] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.353585] x17: 000000000000000e x16: 0000000000000001
>> [    1.354284] x15: 0000000000000019 x14: 0000000000000033
>> [    1.355005] x13: 000000000000004c x12: 0000000000000068
>> [    1.355710] x11: 0000000000000000 x10: ffff80001d923e88
>> [    1.356411] x9 : ffff80001d93c000 x8 : ffff80001d924760
>> [    1.357114] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.357818] x5 : 00008000161f3000 x4 : ffff000009090000
>> [    1.358521] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.359239] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.359940]
>> [    1.360141] Process swapper/4 (pid: 0, stack limit =
>> 0xffff80001d93c038)
>> [    1.360979] Stack: (0xffff80001ef48fd0 to 0xffff80001d940000)
>> [    1.361699] Call trace:
>> [    1.362018] Exception stack(0xffff80001ef48e70 to 0xffff80001ef48fa0)
>> [    1.362842] 8e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.363823] 8e80: ffff80001ef48fd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.364801] 8ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.365780] 8ec0: ffff000009090000 00008000161f3000 003aa6a600000000
>> 0000000000000001
>> [    1.366774] 8ee0: ffff80001d924760 ffff80001d93c000 ffff80001d923e88
>> 0000000000000000
>> [    1.367752] 8f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.368730] 8f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.369707] 8f40: ffff80001ef53f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.370740] 8f60: ffff000008d60d20 ffff80001ef450a0 ffff80001ef49090
>> ffff80001d93c000
>> [    1.371723] 8f80: ffff80001d93c000 ffff80001ef48fd0 ffff000008156e18
>> ffff80001ef48fd0
>> [    1.372706] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.373442] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.374197] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.374916] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
>> [    1.375726] fe00:                                   0000000000000000
>> 00008000161f3000
>> [    1.376708] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000
>> 00008000161f3000
>> [    1.377688] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760
>> ffff80001d93c000
>> [    1.378685] fe60: ffff80001d923e88 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.379668] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.380648] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000010
>> [    1.381630] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d93c000
>> [    1.382627] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000
>> ffff80001d93ff40
>> [    1.383609] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c
>> 0000000060000045
>> [    1.384590] ff20: 0000000000000010 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.385572] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.386263] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
>> [    1.387089] fe00:                                   0000000000000000
>> 00008000161f3000
>> [    1.388069] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000
>> 00008000161f3000
>> [    1.389049] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760
>> ffff80001d93c000
>> [    1.390029] fe60: ffff80001d923e88 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.391025] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.392005] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000010
>> [    1.392986] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d93c000
>> [    1.393967] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000
>> ffff80001d93ff40
>> [    1.394962] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c
>> 0000000060000045
>> [    1.395941] ff20: 0000000000000010 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.396921] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.397538] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.398263] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.398958] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.399679] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.400421] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.401211] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.401745] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.402516] Internal error: undefined instruction: 0 [#3] PREEMPT SMP
>> [    1.403371] Modules linked in:
>> [    1.403780] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.404974] Hardware name: linux,dummy-virt (DT)
>> [    1.405555] I-pipe domain: Linux
>> [    1.405965] task: ffff80001d925780 task.stack: ffff80001d94c000
>> [    1.406727] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.407329] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.407972] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.408889] sp : ffff80001ef7efd0
>> [    1.409311] x29: ffff80001ef7efd0 x28: ffff80001d94c000
>> [    1.410010] x27: ffff80001d94c000 x26: ffff80001ef7f090
>> [    1.410726] x25: ffff80001ef7b0a0 x24: ffff000008d60d20
>> [    1.411425] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.412122] x21: 0000000000000000 x20: ffff80001ef89f80
>> [    1.412819] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.413516] x17: 000000000000000e x16: 0000000000000001
>> [    1.414214] x15: 0000000000000019 x14: 0000000000000033
>> [    1.414929] x13: 000000000000004c x12: 0000000000000068
>> [    1.415623] x11: 0000000000000000 x10: ffff80001d925788
>> [    1.416317] x9 : ffff80001d94c000 x8 : ffff80001d926060
>> [    1.417011] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.417703] x5 : 0000800016229000 x4 : ffff0000090d0000
>> [    1.418396] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.419107] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.419799]
>> [    1.419996] Process swapper/6 (pid: 0, stack limit =
>> 0xffff80001d94c038)
>> [    1.420823] Stack: (0xffff80001ef7efd0 to 0xffff80001d950000)
>> [    1.421533] Call trace:
>> [    1.421846] Exception stack(0xffff80001ef7ee70 to 0xffff80001ef7efa0)
>> [    1.422680] ee60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.423641] ee80: ffff80001ef7efd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.424601] eea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.425559] eec0: ffff0000090d0000 0000800016229000 003aa6a600000000
>> 0000000000000001
>> [    1.426518] eee0: ffff80001d926060 ffff80001d94c000 ffff80001d925788
>> 0000000000000000
>> [    1.427497] ef00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.428456] ef20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.429415] ef40: ffff80001ef89f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.430375] ef60: ffff000008d60d20 ffff80001ef7b0a0 ffff80001ef7f090
>> ffff80001d94c000
>> [    1.431356] ef80: ffff80001d94c000 ffff80001ef7efd0 ffff000008156e18
>> ffff80001ef7efd0
>> [    1.432324] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.433050] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.433796] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.434488] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
>> [    1.435312] fe00:                                   0000000000000000
>> 0000800016229000
>> [    1.436272] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000
>> 0000800016229000
>> [    1.437234] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060
>> ffff80001d94c000
>> [    1.438194] fe60: ffff80001d925788 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.439174] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.440137] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000040
>> [    1.441098] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d94c000
>> [    1.442058] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000
>> ffff80001d94ff40
>> [    1.443038] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c
>> 0000000060000045
>> [    1.443999] ff20: 0000000000000040 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.444959] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.445638] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
>> [    1.446431] fe00:                                   0000000000000000
>> 0000800016229000
>> [    1.447407] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000
>> 0000800016229000
>> [    1.448368] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060
>> ffff80001d94c000
>> [    1.449328] fe60: ffff80001d925788 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.450288] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.451266] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000040
>> [    1.452226] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d94c000
>> [    1.453186] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000
>> ffff80001d94ff40
>> [    1.454146] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c
>> 0000000060000045
>> [    1.455122] ff20: 0000000000000040 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.456083] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.456689] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.457399] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.458059] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.458808] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.459535] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.460310] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.460833] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.461589] Internal error: undefined instruction: 0 [#4] PREEMPT SMP
>> [    1.462402] Modules linked in:
>> [    1.462822] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.464019] Hardware name: linux,dummy-virt (DT)
>> [    1.464599] I-pipe domain: Linux
>> [    1.465010] task: ffff80001d921900 task.stack: ffff80001d930000
>> [    1.465748] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.466347] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.467020] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.467929] sp : ffff80001eef7fd0
>> [    1.468345] x29: ffff80001eef7fd0 x28: ffff80001d930000
>> [    1.469038] x27: ffff80001d930000 x26: ffff80001eef8090
>> [    1.469729] x25: ffff80001eef40a0 x24: ffff000008d60d20
>> [    1.470420] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.471134] x21: 0000000000000000 x20: ffff80001ef02f80
>> [    1.471828] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.472522] x17: 000000000000000e x16: 0000000000000001
>> [    1.473215] x15: 0000000000000019 x14: 0000000000000033
>> [    1.473908] x13: 000000000000004c x12: 0000000000000068
>> [    1.474634] x11: 0000000000000000 x10: ffff80001d921908
>> [    1.475328] x9 : ffff80001d930000 x8 : ffff80001d9221e0
>> [    1.476022] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.476714] x5 : 00008000161a2000 x4 : ffff000009030000
>> [    1.477407] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.478100] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.478822]
>> [    1.479021] Process swapper/1 (pid: 0, stack limit =
>> 0xffff80001d930038)
>> [    1.479851] Stack: (0xffff80001eef7fd0 to 0xffff80001d934000)
>> [    1.480563] Call trace:
>> [    1.480877] Exception stack(0xffff80001eef7e70 to 0xffff80001eef7fa0)
>> [    1.481675] 7e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.482676] 7e80: ffff80001eef7fd0 ffff0000087899d0 00000000800001c5
>> ffff000008128070
>> [    1.483640] 7ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.484603] 7ec0: ffff000009030000 00008000161a2000 003aa6a600000000
>> 0000000000000001
>> [    1.485565] 7ee0: ffff80001d9221e0 ffff80001d930000 ffff80001d921908
>> 0000000000000000
>> [    1.486526] 7f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.487516] 7f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.488480] 7f40: ffff80001ef02f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.489445] 7f60: ffff000008d60d20 ffff80001eef40a0 ffff80001eef8090
>> ffff80001d930000
>> [    1.490411] 7f80: ffff80001d930000 ffff80001eef7fd0 ffff000008156e18
>> ffff80001eef7fd0
>> [    1.491399] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.492124] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.492869] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.493562] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
>> [    1.494360] 3e00:                                   0000000000000000
>> 00008000161a2000
>> [    1.495344] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000
>> 00008000161a2000
>> [    1.496310] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0
>> ffff80001d930000
>> [    1.497275] 3e60: ffff80001d921908 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.498240] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.499221] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000002
>> [    1.500189] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d930000
>> [    1.501156] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000
>> ffff80001d933f40
>> [    1.502122] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c
>> 0000000060000045
>> [    1.503107] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.504074] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.504756] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
>> [    1.505555] 3e00:                                   0000000000000000
>> 00008000161a2000
>> [    1.506522] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000
>> 00008000161a2000
>> [    1.507518] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0
>> ffff80001d930000
>> [    1.508488] 3e60: ffff80001d921908 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.509457] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.510426] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000002
>> [    1.511427] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d930000
>> [    1.512398] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000
>> ffff80001d933f40
>> [    1.513366] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c
>> 0000000060000045
>> [    1.514334] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.515321] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.515930] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.516645] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.517309] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.518014] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.518769] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.519549] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.520076] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.520837] Internal error: undefined instruction: 0 [#5] PREEMPT SMP
>> [    1.521669] Modules linked in:
>> [    1.522075] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.523272] Hardware name: linux,dummy-virt (DT)
>> [    1.523850] I-pipe domain: Linux
>> [    1.524260] task: ffff80001d926400 task.stack: ffff80001d950000
>> [    1.524999] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.525600] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.526242] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.527176] sp : ffff80001ef99fd0
>> [    1.527595] x29: ffff80001ef99fd0 x28: ffff80001d950000
>> [    1.528291] x27: ffff80001d950000 x26: ffff80001ef9a090
>> [    1.528988] x25: ffff80001ef960a0 x24: ffff000008d60d20
>> [    1.529685] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.530381] x21: 0000000000000000 x20: ffff80001efa4f80
>> [    1.531143] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.531842] x17: 000000000000000e x16: 0000000000000001
>> [    1.532540] x15: 0000000000000019 x14: 0000000000000033
>> [    1.533236] x13: 000000000000004c x12: 0000000000000068
>> [    1.533933] x11: 0000000000000000 x10: ffff80001d926408
>> [    1.534655] x9 : ffff80001d950000 x8 : ffff80001d926ce0
>> [    1.535354] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.536052] x5 : 0000800016244000 x4 : ffff0000090f0000
>> [    1.536750] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.537449] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.538146]
>> [    1.538347] Process swapper/7 (pid: 0, stack limit =
>> 0xffff80001d950038)
>> [    1.539207] Stack: (0xffff80001ef99fd0 to 0xffff80001d954000)
>> [    1.539926] Call trace:
>> [    1.540242] Exception stack(0xffff80001ef99e70 to 0xffff80001ef99fa0)
>> [    1.541046] 9e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.542019] 9e80: ffff80001ef99fd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.543010] 9ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.543987] 9ec0: ffff0000090f0000 0000800016244000 003aa6a600000000
>> 0000000000000001
>> [    1.544963] 9ee0: ffff80001d926ce0 ffff80001d950000 ffff80001d926408
>> 0000000000000000
>> [    1.545940] 9f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.546947] 9f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.547905] 9f40: ffff80001efa4f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.548862] 9f60: ffff000008d60d20 ffff80001ef960a0 ffff80001ef9a090
>> ffff80001d950000
>> [    1.549820] 9f80: ffff80001d950000 ffff80001ef99fd0 ffff000008156e18
>> ffff80001ef99fd0
>> [    1.550799] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.551520] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.552260] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.552947] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
>> [    1.553739] 3e00:                                   0000000000000000
>> 0000800016244000
>> [    1.554715] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000
>> 0000800016244000
>> [    1.555676] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0
>> ffff80001d950000
>> [    1.556636] 3e60: ffff80001d926408 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.557597] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.558573] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000080
>> [    1.559537] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d950000
>> [    1.560496] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000
>> ffff80001d953f40
>> [    1.561454] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c
>> 0000000060000045
>> [    1.562412] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.563391] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.564068] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
>> [    1.564859] 3e00:                                   0000000000000000
>> 0000800016244000
>> [    1.565816] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000
>> 0000800016244000
>> [    1.566797] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0
>> ffff80001d950000
>> [    1.567758] 3e60: ffff80001d926408 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.568718] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.569679] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000080
>> [    1.570657] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d950000
>> [    1.571617] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000
>> ffff80001d953f40
>> [    1.572579] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c
>> 0000000060000045
>> [    1.573540] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.574502] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.575127] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.575837] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.576497] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.577198] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.577919] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.578716] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.579238] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.579992] Internal error: undefined instruction: 0 [#6] PREEMPT SMP
>> [    1.580808] Modules linked in:
>> [    1.581208] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.582377] Hardware name: linux,dummy-virt (DT)
>> [    1.582954] I-pipe domain: Linux
>> [    1.583356] task: ffff80001d923200 task.stack: ffff80001d938000
>> [    1.584081] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.584671] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.585303] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.586203] sp : ffff80001ef2dfd0
>> [    1.586633] x29: ffff80001ef2dfd0 x28: ffff80001d938000
>> [    1.587320] x27: ffff80001d938000 x26: ffff80001ef2e090
>> [    1.588006] x25: ffff80001ef2a0a0 x24: ffff000008d60d20
>> [    1.588691] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.589377] x21: 0000000000000000 x20: ffff80001ef38f80
>> [    1.590061] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.590763] x17: 000000000000000e x16: 0000000000000001
>> [    1.591448] x15: 0000000000000019 x14: ffffffffffffffff
>> [    1.592133] x13: ffffffffffffffff x12: 0000000000000000
>> [    1.592817] x11: 0000000000000001 x10: ffff80001d923208
>> [    1.593502] x9 : ffff80001d938000 x8 : ffff80001d923ae0
>> [    1.594186] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.594888] x5 : 00008000161d8000 x4 : ffff000009070000
>> [    1.595574] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.596259] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.596945]
>> [    1.597140] Process swapper/3 (pid: 0, stack limit =
>> 0xffff80001d938038)
>> [    1.597960] Stack: (0xffff80001ef2dfd0 to 0xffff80001d93c000)
>> [    1.598680] Call trace:
>> [    1.598991] Exception stack(0xffff80001ef2de70 to 0xffff80001ef2dfa0)
>> [    1.599781] de60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.600737] de80: ffff80001ef2dfd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.601693] dea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.602663] dec0: ffff000009070000 00008000161d8000 003aa6a600000000
>> 0000000000000001
>> [    1.603618] dee0: ffff80001d923ae0 ffff80001d938000 ffff80001d923208
>> 0000000000000001
>> [    1.604573] df00: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    1.605527] df20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.606483] df40: ffff80001ef38f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.607453] df60: ffff000008d60d20 ffff80001ef2a0a0 ffff80001ef2e090
>> ffff80001d938000
>> [    1.608408] df80: ffff80001d938000 ffff80001ef2dfd0 ffff000008156e18
>> ffff80001ef2dfd0
>> [    1.609363] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.610078] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.610831] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.611515] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
>> [    1.612302] be00:                                   0000000000000000
>> 00008000161d8000
>> [    1.613256] be20: 0000000000000000 ffff80001ef318e8 0000000000000000
>> 00008000161d8000
>> [    1.614209] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0
>> ffff80001d938000
>> [    1.615179] be60: ffff80001d923208 0000000000000001 0000000000000000
>> ffffffffffffffff
>> [    1.616134] be80: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.617088] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000008
>> [    1.618085] bec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d938000
>> [    1.619047] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000
>> ffff80001d93bf40
>> [    1.620003] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c
>> 0000000060000045
>> [    1.620957] bf20: 0000000000000008 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.621912] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.622608] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
>> [    1.623394] be00:                                   0000000000000000
>> 00008000161d8000
>> [    1.624342] be20: 0000000000000000 ffff80001ef318e8 0000000000000000
>> 00008000161d8000
>> [    1.625291] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0
>> ffff80001d938000
>> [    1.626240] be60: ffff80001d923208 0000000000000001 0000000000000000
>> ffffffffffffffff
>> [    1.627208] be80: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.628156] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000008
>> [    1.629103] bec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d938000
>> [    1.630049] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000
>> ffff80001d93bf40
>> [    1.631013] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c
>> 0000000060000045
>> [    1.631960] bf20: 0000000000000008 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.632908] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.633505] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.634206] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.634894] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.635595] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.636314] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.637082] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.637602] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.638352] Internal error: undefined instruction: 0 [#7] PREEMPT SMP
>> [    2.174474] Modules linked in:
>> [    2.174895] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    2.175975] Hardware name: linux,dummy-virt (DT)
>> [    2.176500] I-pipe domain: Linux
>> [    2.176873] task: ffff000008da0080 task.stack: ffff000008d90000
>> [    2.177549] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    2.178097] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    2.178698] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    2.179532] sp : ffff80001eedcda0
>> [    2.179914] x29: ffff80001eedcda0 x28: ffff000008d90000
>> [    2.180548] x27: ffff000008d90000 x26: 0000000000000002
>> [    2.181181] x25: ffff80001eeddbb4 x24: ffff000008d60d20
>> [    2.181814] x23: ffff000008db2000 x22: 0000000000000003
>> [    2.182447] x21: 0000000000000000 x20: ffff80001eee7f80
>> [    2.183096] x19: ffff80001d804a00 x18: 0000000000000007
>> [    2.183728] x17: 000000000000000e x16: 0000000000000001
>> [    2.184361] x15: 0000000000000019 x14: ffffffffffffffff
>> [    2.184994] x13: ffffffffffffffff x12: 0000000000000000
>> [    2.185628] x11: 0000000000000004 x10: ffff80001eee0968
>> [    2.186260] x9 : ffff80001eee08f0 x8 : ffff80001eee0978
>> [    2.186909] x7 : ffff80001eee08e8 x6 : 0000000000000000
>> [    2.187543] x5 : 0000800016187000 x4 : ffff000009010000
>> [    2.188176] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    2.188810] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    2.189443]
>> [    2.189625] Process swapper/0 (pid: 0, stack limit =
>> 0xffff000008d90038)
>> [    2.190382] Stack: (0xffff80001eedcda0 to 0xffff000008d94000)
>> [    2.191047] Call trace:
>> [    2.191334] Exception stack(0xffff80001eedcc40 to 0xffff80001eedcd70)
>> [    2.192063] cc40: ffff80001d804a00 0001000000000000 ffff80001eedcda0
>> ffff0000087899d0
>> [    2.192945] cc60: 00000000800001c5 0000000000000400 ffff0000087899d0
>> ffff0000083d4ed8
>> [    2.193827] cc80: ffff0000083d5ae8 0000000008000000 ffff000009010000
>> 0000800016187000
>> [    2.194726] cca0: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.195607] ccc0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.196491] cce0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.197373] cd00: 0000000000000007 ffff80001d804a00 ffff80001eee7f80
>> 0000000000000000
>> [    2.198255] cd20: 0000000000000003 ffff000008db2000 ffff000008d60d20
>> ffff80001eeddbb4
>> [    2.199152] cd40: 0000000000000002 ffff000008d90000 ffff000008d90000
>> ffff80001eedcda0
>> [    2.200033] cd60: ffff000008156e18 ffff80001eedcda0
>> [    2.200589] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    2.201251] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    2.201932] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    2.202579] Exception stack(0xffff80001eedce10 to 0xffff80001eedcf40)
>> [    2.203307] ce00:                                   0000000000001fff
>> ffff000008118c28
>> [    2.204188] ce20: ffff80001eedcf80 ffff000008082fc8 ffff80001eedce50
>> 0001000000000000
>> [    2.205069] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402
>> ffff000008eab880
>> [    2.205949] ce60: 0000000000000000 0000000000000002 0000000000010000
>> 0000000000000000
>> [    2.206867] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.207743] cea0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.208618] cec0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.209494] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080
>> 0000000000000001
>> [    2.210369] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20
>> ffff80001eed90a0
>> [    2.211269] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000
>> ffff80001eedcf80
>> [    2.212147] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    2.212766] Exception stack(0xffff80001eedce30 to 0xffff80001eedcf60)
>> [    2.213490] ce20:                                   ffff80001eedce50
>> 0001000000000000
>> [    2.214368] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402
>> ffff000008eab880
>> [    2.215262] ce60: 0000000000000000 0000000000000002 0000000000010000
>> 0000000000000000
>> [    2.216138] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.217015] cea0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.217892] cec0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.218785] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080
>> 0000000000000001
>> [    2.219664] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20
>> ffff80001eed90a0
>> [    2.220543] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000
>> ffff80001eedcf80
>> [    2.221422] cf40: ffff000008155fd4 ffff80001eedcf80 ffff000008155e6c
>> 0000000020000145
>> [    2.222300] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    2.222881] [<ffff000008155fd4>] __ipipe_do_sync_pipeline+0xa4/0xb8
>> [    2.223587] [<ffff000008156730>] __ipipe_dispatch_irq+0x170/0x1d8
>> [    2.224274] [<ffff000008081274>] __ipipe_grab_ipi+0x1c/0x30
>> [    2.224902] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
>> [    2.225626] 3dc0: 0000000000000000 0000800016187000 0000000000000000
>> ffff80001eee08e8
>> [    2.226501] 3de0: 0000000000000000 0000000000000020 003aa6a600000000
>> 0000000000000001
>> [    2.227442] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088
>> 0000000000000001
>> [    2.228329] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    2.229215] 3e40: 0000000000000001 000000000000000e 0000000000000007
>> ffff000008d97258
>> [    2.230100] 3e60: ffff000008d972e8 0000000000000001 0000000000000000
>> 0000000000000000
>> [    2.231007] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555
>> ffff000008d90000
>> [    2.231895] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28
>> ffff000008d93ef0
>> [    2.232782] 3ec0: ffff00000815601c 0000000060000045 0000000000000001
>> 0000000000000000
>> [    2.233669] 3ee0: ffffffffffffffff ffff000008d60d20
>> [    2.234228] [<ffff00000808178c>] gic_handle_irq+0x64/0x100
>> [    2.234885] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
>> [    2.235610] 3dc0: 0000000000000000 0000800016187000 0000000000000000
>> ffff80001eee08e8
>> [    2.236488] 3de0: 0000000000000000 0000000000000020 003aa6a600000000
>> 0000000000000001
>> [    2.237365] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088
>> 0000000000000001
>> [    2.238241] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    2.239151] 3e40: 0000000000000001 000000000000000e 0000000000000007
>> ffff000008d97258
>> [    2.240028] 3e60: ffff000008d972e8 0000000000000001 0000000000000000
>> 0000000000000000
>> [    2.240906] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555
>> ffff000008d90000
>> [    2.241783] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28
>> ffff000008d93ef0
>> [    2.242686] 3ec0: ffff00000815601c 0000000060000045 0000000000000001
>> 0000000000000000
>> [    2.243565] 3ee0: ffffffffffffffff ffff000008d60d20
>> [    2.244120] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    2.244675] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    2.245326] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    2.245932] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    2.246594] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    2.247256] [<ffff0000088de888>] rest_init+0x80/0x90
>> [    2.247824] [<ffff000008c90b2c>] start_kernel+0x380/0x394
>> [    2.248437] [<ffff000008c901d8>] __primary_switched+0x5c/0x64
>> [    2.249090] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    2.249784] ---[ end trace ee076ceca10e9295 ]---
>> [    2.837878] SMP: failed to stop secondary CPUs 0,2
>> [    2.838465] Kernel Offset: disabled
>> [    2.838887] Memory Limit: 512 MB
>>

Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu

[gengdongjiu]

Thanks the reply. using I-pipe 4.9 branch, if not apply Xenomai patches, it can boot up normal using qemu. But if apply the Xenomai patch, it will happen this panic, do not know the reason.
发件人：Greg Gallagher
收件人：gengdongjiu,
抄 送：Jan Kiszka,Pintu Kumar,Philippe Gerum,Xenomai@xenomai.org,
时间：2018-06-21 22:04:34
主 题：Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu

Hi,
  To the best of my knowledge we don't run it with qemu often.  Boards
that have been run used in the past have been rpi3 (64-bit mode),
hikey-960 and there are some online resources for the dragon board
410c.  I'm currently testing the raspberry pi3 on the new ARM64 4.14
ipipe branch.  Hopefully i'll have some results in the next two week
or so.

-Greg

On Thu, Jun 21, 2018 at 9:56 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
> Sorry, I discription is not right, the Xenomai branch is next not stable
> 3.x.  Do you ever run it using kvm qemu? thanks a lot
>
>
>
> 发件人：Greg Gallagher
> 收件人：gengdongjiu,
> 抄 送：Jan Kiszka,Pintu Kumar,Philippe Gerum,Xenomai@xenomai.org,
> 时间：2018-06-21 21:48:00
> 主 题：Re: [Xenomai] Kernel panic during kernel bootup on arm64 qemu (e1000e)
>
> For ARM64 you need to use the Xenomai next branch not the stable release.
>
> -Greg
>
> On Thu, Jun 21, 2018 at 8:27 AM, gengdongjiu <gengdongjiu@huawei.com> wrote:
>> Hi
>>    I use the stable 3.x Xenomai and ipipe-4.9.y, and run it using below
>> qemu commands:
>>  qemu-kvm -enable-kvm -m 1024 -cpu host -machine virt,gic-version=3  -smp
>> 10 -nographic -kernel Image -append "rdinit=/init console=ttyAMA0 mem=512M
>> root=/dev/ram0 earlycon=pl011,0x9000000 rw" -initrd guestfs_new.cpio.gz
>>  it will happen below panic, do you know the reason? thanks
>>
>> [    1.286223] Hardware name: linux,dummy-virt (DT)
>> [    1.286773] I-pipe domain: Linux
>> [    1.287153] task: ffff80001d8d0000 task.stack: ffff80001d8a4000
>> [    1.287840] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.288395] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.288982] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.289818] sp : ffff80001ef12fd0
>> [    1.290201] x29: ffff80001ef12fd0 x28: ffff80001d8a4000
>> [    1.290866] x27: 0000000000000003 x26: ffff80001ef13090
>> [    1.291506] x25: ffff80001ef0f0a0 x24: ffff00000817ad00
>> [    1.292147] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.292787] x21: 0000000000000000 x20: ffff80001ef1df80
>> [    1.293427] x19: ffff80001d804a00 x18: 000000000000000e
>> [    1.294066] x17: 00000000000045bf x16: 0140000000000000
>> [    1.294716] x15: ffff000049f61000 x14: ffff80001effd290
>> [    1.295357] x13: ffff000008e66078 x12: 0000000000000000
>> [    1.295998] x11: 0000000000000000 x10: 000000003b9aca00
>> [    1.296639] x9 : 0000000000000000 x8 : 0000000000000000
>> [    1.297281] x7 : 0000000000000000 x6 : 0000000000000000
>> [    1.297922] x5 : 00008000161bd000 x4 : ffff000009050000
>> [    1.298582] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.299223] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.299865]
>> [    1.300049] Process swapper/0 (pid: 1, stack limit =
>> 0xffff80001d8a4038)
>> [    1.300816] Stack: (0xffff80001ef12fd0 to 0xffff80001d8a8000)
>> [    1.301474] Call trace:
>> [    1.301763] Exception stack(0xffff80001ef12e70 to 0xffff80001ef12fa0)
>> [    1.302500] 2e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.303403] 2e80: ffff80001ef12fd0 ffff0000087899d0 00000000800001c5
>> ffff0000083d5b04
>> [    1.304297] 2ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.305191] 2ec0: ffff000009050000 00008000161bd000 0000000000000000
>> 0000000000000000
>> [    1.306084] 2ee0: 0000000000000000 0000000000000000 000000003b9aca00
>> 0000000000000000
>> [    1.307000] 2f00: 0000000000000000 ffff000008e66078 ffff80001effd290
>> ffff000049f61000
>> [    1.307894] 2f20: 0140000000000000 00000000000045bf 000000000000000e
>> ffff80001d804a00
>> [    1.308789] 2f40: ffff80001ef1df80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.309687] 2f60: ffff00000817ad00 ffff80001ef0f0a0 ffff80001ef13090
>> 0000000000000003
>> [    1.310592] 2f80: ffff80001d8a4000 ffff80001ef12fd0 ffff000008156e18
>> ffff80001ef12fd0
>> [    1.311497] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.312176] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.312873] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.313519] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
>> [    1.314262] 7b00:                                   0000000000000000
>> ffff000008eab1a8
>> [    1.315181] 7b20: 0000000000000000 0000000000000001 ffff000008eab188
>> 00000000ffffffff
>> [    1.316081] 7b40: 0000000000000000 0000000000000000 0000000000000000
>> 0000000000000000
>> [    1.316981] 7b60: 000000003b9aca00 0000000000000000 0000000000000000
>> ffff000008e66078
>> [    1.317880] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000
>> 00000000000045bf
>> [    1.318786] 7ba0: 000000000000000e ffff000008eab000 0000000000000040
>> ffff000008170738
>> [    1.319688] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00
>> 0000000000000001
>> [    1.320587] 7be0: ffff000008ecd000 0000000000000003 0000000000000000
>> ffff80001d8a7c40
>> [    1.321489] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594
>> 0000000020000045
>> [    1.322390] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff
>> 0000000000000040
>> [    1.323306] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.323942] Exception stack(0xffff80001d8a7b10 to 0xffff80001d8a7c40)
>> [    1.324687] 7b00:                                   0000000000000000
>> ffff000008eab1a8
>> [    1.325588] 7b20: 0000000000000000 0000000000000001 ffff000008eab188
>> 00000000ffffffff
>> [    1.326490] 7b40: 0000000000000000 0000000000000000 0000000000000000
>> 0000000000000000
>> [    1.327402] 7b60: 000000003b9aca00 0000000000000000 0000000000000000
>> ffff000008e66078
>> [    1.328308] 7b80: ffff80001effd290 ffff000049f61000 0140000000000000
>> 00000000000045bf
>> [    1.329212] 7ba0: 000000000000000e ffff000008eab000 0000000000000040
>> ffff000008170738
>> [    1.330117] 7bc0: 0000000000000040 ffff00000817adf8 ffff00000817ad00
>> 0000000000000001
>> [    1.331035] 7be0: ffff000008ecd000 0000000000000003 0000000000000000
>> ffff80001d8a7c40
>> [    1.331933] 7c00: ffff00000815558c ffff80001d8a7c40 ffff000008155594
>> 0000000020000045
>> [    1.332830] 7c20: ffff80001ef3dc80 ffff80001ef38f80 ffffffffffffffff
>> 0000000000000040
>> [    1.333730] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.334295] [<ffff000008155594>] ipipe_critical_exit+0x9c/0xa8
>> [    1.334976] [<ffff0000081576a0>] ipipe_timer_start+0x90/0x1d0
>> [    1.335654] [<ffff00000817b184>] xntimer_grab_hardware+0xfc/0x380
>> [    1.336368] [<ffff000008ca2660>] xenomai_init+0x3c8/0x4e0
>> [    1.336997] [<ffff000008083b08>] do_one_initcall+0x38/0x128
>> [    1.337654] [<ffff000008c90cf8>] kernel_init_freeable+0x1b8/0x254
>> [    1.338367] [<ffff0000088de8a8>] kernel_init+0x10/0x100
>> [    1.338991] [<ffff0000080837e4>] ret_from_fork+0x14/0x30
>> [    1.339609] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.340325] Internal error: undefined instruction: 0 [#2] PREEMPT SMP
>> [    1.340337] ---[ end trace ee076ceca10e9294 ]---
>> [    1.340351] Kernel panic - not syncing: Attempted to kill init!
>> exitcode=0x0000000b
>> [    1.340351]
>> [    1.340352] SMP: stopping secondary CPUs
>> [    1.343408] Modules linked in:
>> [    1.343819] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.345016] Hardware name: linux,dummy-virt (DT)
>> [    1.345596] I-pipe domain: Linux
>> [    1.346007] task: ffff80001d923e80 task.stack: ffff80001d93c000
>> [    1.346768] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.347372] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.348018] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.348936] sp : ffff80001ef48fd0
>> [    1.349357] x29: ffff80001ef48fd0 x28: ffff80001d93c000
>> [    1.350057] x27: ffff80001d93c000 x26: ffff80001ef49090
>> [    1.350779] x25: ffff80001ef450a0 x24: ffff000008d60d20
>> [    1.351482] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.352186] x21: 0000000000000000 x20: ffff80001ef53f80
>> [    1.352885] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.353585] x17: 000000000000000e x16: 0000000000000001
>> [    1.354284] x15: 0000000000000019 x14: 0000000000000033
>> [    1.355005] x13: 000000000000004c x12: 0000000000000068
>> [    1.355710] x11: 0000000000000000 x10: ffff80001d923e88
>> [    1.356411] x9 : ffff80001d93c000 x8 : ffff80001d924760
>> [    1.357114] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.357818] x5 : 00008000161f3000 x4 : ffff000009090000
>> [    1.358521] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.359239] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.359940]
>> [    1.360141] Process swapper/4 (pid: 0, stack limit =
>> 0xffff80001d93c038)
>> [    1.360979] Stack: (0xffff80001ef48fd0 to 0xffff80001d940000)
>> [    1.361699] Call trace:
>> [    1.362018] Exception stack(0xffff80001ef48e70 to 0xffff80001ef48fa0)
>> [    1.362842] 8e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.363823] 8e80: ffff80001ef48fd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.364801] 8ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.365780] 8ec0: ffff000009090000 00008000161f3000 003aa6a600000000
>> 0000000000000001
>> [    1.366774] 8ee0: ffff80001d924760 ffff80001d93c000 ffff80001d923e88
>> 0000000000000000
>> [    1.367752] 8f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.368730] 8f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.369707] 8f40: ffff80001ef53f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.370740] 8f60: ffff000008d60d20 ffff80001ef450a0 ffff80001ef49090
>> ffff80001d93c000
>> [    1.371723] 8f80: ffff80001d93c000 ffff80001ef48fd0 ffff000008156e18
>> ffff80001ef48fd0
>> [    1.372706] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.373442] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.374197] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.374916] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
>> [    1.375726] fe00:                                   0000000000000000
>> 00008000161f3000
>> [    1.376708] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000
>> 00008000161f3000
>> [    1.377688] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760
>> ffff80001d93c000
>> [    1.378685] fe60: ffff80001d923e88 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.379668] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.380648] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000010
>> [    1.381630] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d93c000
>> [    1.382627] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000
>> ffff80001d93ff40
>> [    1.383609] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c
>> 0000000060000045
>> [    1.384590] ff20: 0000000000000010 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.385572] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.386263] Exception stack(0xffff80001d93fe10 to 0xffff80001d93ff40)
>> [    1.387089] fe00:                                   0000000000000000
>> 00008000161f3000
>> [    1.388069] fe20: 0000000000000000 ffff80001ef4c8e8 0000000000000000
>> 00008000161f3000
>> [    1.389049] fe40: 003aa6a600000000 0000000000000001 ffff80001d924760
>> ffff80001d93c000
>> [    1.390029] fe60: ffff80001d923e88 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.391025] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.392005] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000010
>> [    1.392986] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d93c000
>> [    1.393967] fee0: ffff000008e68555 ffff80001d93c000 ffff000008d97000
>> ffff80001d93ff40
>> [    1.394962] ff00: ffff000008085d28 ffff80001d93ff40 ffff00000815601c
>> 0000000060000045
>> [    1.395941] ff20: 0000000000000010 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.396921] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.397538] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.398263] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.398958] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.399679] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.400421] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.401211] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.401745] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.402516] Internal error: undefined instruction: 0 [#3] PREEMPT SMP
>> [    1.403371] Modules linked in:
>> [    1.403780] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.404974] Hardware name: linux,dummy-virt (DT)
>> [    1.405555] I-pipe domain: Linux
>> [    1.405965] task: ffff80001d925780 task.stack: ffff80001d94c000
>> [    1.406727] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.407329] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.407972] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.408889] sp : ffff80001ef7efd0
>> [    1.409311] x29: ffff80001ef7efd0 x28: ffff80001d94c000
>> [    1.410010] x27: ffff80001d94c000 x26: ffff80001ef7f090
>> [    1.410726] x25: ffff80001ef7b0a0 x24: ffff000008d60d20
>> [    1.411425] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.412122] x21: 0000000000000000 x20: ffff80001ef89f80
>> [    1.412819] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.413516] x17: 000000000000000e x16: 0000000000000001
>> [    1.414214] x15: 0000000000000019 x14: 0000000000000033
>> [    1.414929] x13: 000000000000004c x12: 0000000000000068
>> [    1.415623] x11: 0000000000000000 x10: ffff80001d925788
>> [    1.416317] x9 : ffff80001d94c000 x8 : ffff80001d926060
>> [    1.417011] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.417703] x5 : 0000800016229000 x4 : ffff0000090d0000
>> [    1.418396] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.419107] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.419799]
>> [    1.419996] Process swapper/6 (pid: 0, stack limit =
>> 0xffff80001d94c038)
>> [    1.420823] Stack: (0xffff80001ef7efd0 to 0xffff80001d950000)
>> [    1.421533] Call trace:
>> [    1.421846] Exception stack(0xffff80001ef7ee70 to 0xffff80001ef7efa0)
>> [    1.422680] ee60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.423641] ee80: ffff80001ef7efd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.424601] eea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.425559] eec0: ffff0000090d0000 0000800016229000 003aa6a600000000
>> 0000000000000001
>> [    1.426518] eee0: ffff80001d926060 ffff80001d94c000 ffff80001d925788
>> 0000000000000000
>> [    1.427497] ef00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.428456] ef20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.429415] ef40: ffff80001ef89f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.430375] ef60: ffff000008d60d20 ffff80001ef7b0a0 ffff80001ef7f090
>> ffff80001d94c000
>> [    1.431356] ef80: ffff80001d94c000 ffff80001ef7efd0 ffff000008156e18
>> ffff80001ef7efd0
>> [    1.432324] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.433050] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.433796] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.434488] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
>> [    1.435312] fe00:                                   0000000000000000
>> 0000800016229000
>> [    1.436272] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000
>> 0000800016229000
>> [    1.437234] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060
>> ffff80001d94c000
>> [    1.438194] fe60: ffff80001d925788 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.439174] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.440137] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000040
>> [    1.441098] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d94c000
>> [    1.442058] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000
>> ffff80001d94ff40
>> [    1.443038] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c
>> 0000000060000045
>> [    1.443999] ff20: 0000000000000040 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.444959] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.445638] Exception stack(0xffff80001d94fe10 to 0xffff80001d94ff40)
>> [    1.446431] fe00:                                   0000000000000000
>> 0000800016229000
>> [    1.447407] fe20: 0000000000000000 ffff80001ef828e8 0000000000000000
>> 0000800016229000
>> [    1.448368] fe40: 003aa6a600000000 0000000000000001 ffff80001d926060
>> ffff80001d94c000
>> [    1.449328] fe60: ffff80001d925788 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.450288] fe80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.451266] fea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000040
>> [    1.452226] fec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d94c000
>> [    1.453186] fee0: ffff000008e68555 ffff80001d94c000 ffff000008d97000
>> ffff80001d94ff40
>> [    1.454146] ff00: ffff000008085d28 ffff80001d94ff40 ffff00000815601c
>> 0000000060000045
>> [    1.455122] ff20: 0000000000000040 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.456083] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.456689] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.457399] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.458059] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.458808] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.459535] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.460310] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.460833] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.461589] Internal error: undefined instruction: 0 [#4] PREEMPT SMP
>> [    1.462402] Modules linked in:
>> [    1.462822] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.464019] Hardware name: linux,dummy-virt (DT)
>> [    1.464599] I-pipe domain: Linux
>> [    1.465010] task: ffff80001d921900 task.stack: ffff80001d930000
>> [    1.465748] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.466347] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.467020] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.467929] sp : ffff80001eef7fd0
>> [    1.468345] x29: ffff80001eef7fd0 x28: ffff80001d930000
>> [    1.469038] x27: ffff80001d930000 x26: ffff80001eef8090
>> [    1.469729] x25: ffff80001eef40a0 x24: ffff000008d60d20
>> [    1.470420] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.471134] x21: 0000000000000000 x20: ffff80001ef02f80
>> [    1.471828] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.472522] x17: 000000000000000e x16: 0000000000000001
>> [    1.473215] x15: 0000000000000019 x14: 0000000000000033
>> [    1.473908] x13: 000000000000004c x12: 0000000000000068
>> [    1.474634] x11: 0000000000000000 x10: ffff80001d921908
>> [    1.475328] x9 : ffff80001d930000 x8 : ffff80001d9221e0
>> [    1.476022] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.476714] x5 : 00008000161a2000 x4 : ffff000009030000
>> [    1.477407] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.478100] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.478822]
>> [    1.479021] Process swapper/1 (pid: 0, stack limit =
>> 0xffff80001d930038)
>> [    1.479851] Stack: (0xffff80001eef7fd0 to 0xffff80001d934000)
>> [    1.480563] Call trace:
>> [    1.480877] Exception stack(0xffff80001eef7e70 to 0xffff80001eef7fa0)
>> [    1.481675] 7e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.482676] 7e80: ffff80001eef7fd0 ffff0000087899d0 00000000800001c5
>> ffff000008128070
>> [    1.483640] 7ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.484603] 7ec0: ffff000009030000 00008000161a2000 003aa6a600000000
>> 0000000000000001
>> [    1.485565] 7ee0: ffff80001d9221e0 ffff80001d930000 ffff80001d921908
>> 0000000000000000
>> [    1.486526] 7f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.487516] 7f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.488480] 7f40: ffff80001ef02f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.489445] 7f60: ffff000008d60d20 ffff80001eef40a0 ffff80001eef8090
>> ffff80001d930000
>> [    1.490411] 7f80: ffff80001d930000 ffff80001eef7fd0 ffff000008156e18
>> ffff80001eef7fd0
>> [    1.491399] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.492124] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.492869] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.493562] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
>> [    1.494360] 3e00:                                   0000000000000000
>> 00008000161a2000
>> [    1.495344] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000
>> 00008000161a2000
>> [    1.496310] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0
>> ffff80001d930000
>> [    1.497275] 3e60: ffff80001d921908 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.498240] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.499221] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000002
>> [    1.500189] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d930000
>> [    1.501156] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000
>> ffff80001d933f40
>> [    1.502122] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c
>> 0000000060000045
>> [    1.503107] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.504074] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.504756] Exception stack(0xffff80001d933e10 to 0xffff80001d933f40)
>> [    1.505555] 3e00:                                   0000000000000000
>> 00008000161a2000
>> [    1.506522] 3e20: 0000000000000000 ffff80001eefb8e8 0000000000000000
>> 00008000161a2000
>> [    1.507518] 3e40: 003aa6a600000000 0000000000000001 ffff80001d9221e0
>> ffff80001d930000
>> [    1.508488] 3e60: ffff80001d921908 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.509457] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.510426] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000002
>> [    1.511427] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d930000
>> [    1.512398] 3ee0: ffff000008e68555 ffff80001d930000 ffff000008d97000
>> ffff80001d933f40
>> [    1.513366] 3f00: ffff000008085d28 ffff80001d933f40 ffff00000815601c
>> 0000000060000045
>> [    1.514334] 3f20: 0000000000000002 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.515321] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.515930] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.516645] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.517309] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.518014] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.518769] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.519549] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.520076] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.520837] Internal error: undefined instruction: 0 [#5] PREEMPT SMP
>> [    1.521669] Modules linked in:
>> [    1.522075] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.523272] Hardware name: linux,dummy-virt (DT)
>> [    1.523850] I-pipe domain: Linux
>> [    1.524260] task: ffff80001d926400 task.stack: ffff80001d950000
>> [    1.524999] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.525600] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.526242] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.527176] sp : ffff80001ef99fd0
>> [    1.527595] x29: ffff80001ef99fd0 x28: ffff80001d950000
>> [    1.528291] x27: ffff80001d950000 x26: ffff80001ef9a090
>> [    1.528988] x25: ffff80001ef960a0 x24: ffff000008d60d20
>> [    1.529685] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.530381] x21: 0000000000000000 x20: ffff80001efa4f80
>> [    1.531143] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.531842] x17: 000000000000000e x16: 0000000000000001
>> [    1.532540] x15: 0000000000000019 x14: 0000000000000033
>> [    1.533236] x13: 000000000000004c x12: 0000000000000068
>> [    1.533933] x11: 0000000000000000 x10: ffff80001d926408
>> [    1.534655] x9 : ffff80001d950000 x8 : ffff80001d926ce0
>> [    1.535354] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.536052] x5 : 0000800016244000 x4 : ffff0000090f0000
>> [    1.536750] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.537449] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.538146]
>> [    1.538347] Process swapper/7 (pid: 0, stack limit =
>> 0xffff80001d950038)
>> [    1.539207] Stack: (0xffff80001ef99fd0 to 0xffff80001d954000)
>> [    1.539926] Call trace:
>> [    1.540242] Exception stack(0xffff80001ef99e70 to 0xffff80001ef99fa0)
>> [    1.541046] 9e60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.542019] 9e80: ffff80001ef99fd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.543010] 9ea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.543987] 9ec0: ffff0000090f0000 0000800016244000 003aa6a600000000
>> 0000000000000001
>> [    1.544963] 9ee0: ffff80001d926ce0 ffff80001d950000 ffff80001d926408
>> 0000000000000000
>> [    1.545940] 9f00: 0000000000000068 000000000000004c 0000000000000033
>> 0000000000000019
>> [    1.546947] 9f20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.547905] 9f40: ffff80001efa4f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.548862] 9f60: ffff000008d60d20 ffff80001ef960a0 ffff80001ef9a090
>> ffff80001d950000
>> [    1.549820] 9f80: ffff80001d950000 ffff80001ef99fd0 ffff000008156e18
>> ffff80001ef99fd0
>> [    1.550799] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.551520] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.552260] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.552947] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
>> [    1.553739] 3e00:                                   0000000000000000
>> 0000800016244000
>> [    1.554715] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000
>> 0000800016244000
>> [    1.555676] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0
>> ffff80001d950000
>> [    1.556636] 3e60: ffff80001d926408 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.557597] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.558573] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000080
>> [    1.559537] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d950000
>> [    1.560496] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000
>> ffff80001d953f40
>> [    1.561454] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c
>> 0000000060000045
>> [    1.562412] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.563391] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.564068] Exception stack(0xffff80001d953e10 to 0xffff80001d953f40)
>> [    1.564859] 3e00:                                   0000000000000000
>> 0000800016244000
>> [    1.565816] 3e20: 0000000000000000 ffff80001ef9d8e8 0000000000000000
>> 0000800016244000
>> [    1.566797] 3e40: 003aa6a600000000 0000000000000001 ffff80001d926ce0
>> ffff80001d950000
>> [    1.567758] 3e60: ffff80001d926408 0000000000000000 0000000000000068
>> 000000000000004c
>> [    1.568718] 3e80: 0000000000000033 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.569679] 3ea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000080
>> [    1.570657] 3ec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d950000
>> [    1.571617] 3ee0: ffff000008e68555 ffff80001d950000 ffff000008d97000
>> ffff80001d953f40
>> [    1.572579] 3f00: ffff000008085d28 ffff80001d953f40 ffff00000815601c
>> 0000000060000045
>> [    1.573540] 3f20: 0000000000000080 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.574502] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.575127] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.575837] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.576497] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.577198] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.577919] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.578716] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.579238] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.579992] Internal error: undefined instruction: 0 [#6] PREEMPT SMP
>> [    1.580808] Modules linked in:
>> [    1.581208] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    1.582377] Hardware name: linux,dummy-virt (DT)
>> [    1.582954] I-pipe domain: Linux
>> [    1.583356] task: ffff80001d923200 task.stack: ffff80001d938000
>> [    1.584081] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    1.584671] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    1.585303] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    1.586203] sp : ffff80001ef2dfd0
>> [    1.586633] x29: ffff80001ef2dfd0 x28: ffff80001d938000
>> [    1.587320] x27: ffff80001d938000 x26: ffff80001ef2e090
>> [    1.588006] x25: ffff80001ef2a0a0 x24: ffff000008d60d20
>> [    1.588691] x23: ffff000008db2000 x22: 0000000000000003
>> [    1.589377] x21: 0000000000000000 x20: ffff80001ef38f80
>> [    1.590061] x19: ffff80001d804a00 x18: 0000000000000007
>> [    1.590763] x17: 000000000000000e x16: 0000000000000001
>> [    1.591448] x15: 0000000000000019 x14: ffffffffffffffff
>> [    1.592133] x13: ffffffffffffffff x12: 0000000000000000
>> [    1.592817] x11: 0000000000000001 x10: ffff80001d923208
>> [    1.593502] x9 : ffff80001d938000 x8 : ffff80001d923ae0
>> [    1.594186] x7 : 0000000000000001 x6 : 003aa6a600000000
>> [    1.594888] x5 : 00008000161d8000 x4 : ffff000009070000
>> [    1.595574] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    1.596259] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    1.596945]
>> [    1.597140] Process swapper/3 (pid: 0, stack limit =
>> 0xffff80001d938038)
>> [    1.597960] Stack: (0xffff80001ef2dfd0 to 0xffff80001d93c000)
>> [    1.598680] Call trace:
>> [    1.598991] Exception stack(0xffff80001ef2de70 to 0xffff80001ef2dfa0)
>> [    1.599781] de60:                                   ffff80001d804a00
>> 0001000000000000
>> [    1.600737] de80: ffff80001ef2dfd0 ffff0000087899d0 00000000800001c5
>> ffff00000811e52c
>> [    1.601693] dea0: ffff0000087899d0 ffff0000083d4ed8 ffff0000083d5ae8
>> 0000000008000000
>> [    1.602663] dec0: ffff000009070000 00008000161d8000 003aa6a600000000
>> 0000000000000001
>> [    1.603618] dee0: ffff80001d923ae0 ffff80001d938000 ffff80001d923208
>> 0000000000000001
>> [    1.604573] df00: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    1.605527] df20: 0000000000000001 000000000000000e 0000000000000007
>> ffff80001d804a00
>> [    1.606483] df40: ffff80001ef38f80 0000000000000000 0000000000000003
>> ffff000008db2000
>> [    1.607453] df60: ffff000008d60d20 ffff80001ef2a0a0 ffff80001ef2e090
>> ffff80001d938000
>> [    1.608408] df80: ffff80001d938000 ffff80001ef2dfd0 ffff000008156e18
>> ffff80001ef2dfd0
>> [    1.609363] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    1.610078] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    1.610831] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    1.611515] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
>> [    1.612302] be00:                                   0000000000000000
>> 00008000161d8000
>> [    1.613256] be20: 0000000000000000 ffff80001ef318e8 0000000000000000
>> 00008000161d8000
>> [    1.614209] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0
>> ffff80001d938000
>> [    1.615179] be60: ffff80001d923208 0000000000000001 0000000000000000
>> ffffffffffffffff
>> [    1.616134] be80: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.617088] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000008
>> [    1.618085] bec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d938000
>> [    1.619047] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000
>> ffff80001d93bf40
>> [    1.620003] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c
>> 0000000060000045
>> [    1.620957] bf20: 0000000000000008 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.621912] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    1.622608] Exception stack(0xffff80001d93be10 to 0xffff80001d93bf40)
>> [    1.623394] be00:                                   0000000000000000
>> 00008000161d8000
>> [    1.624342] be20: 0000000000000000 ffff80001ef318e8 0000000000000000
>> 00008000161d8000
>> [    1.625291] be40: 003aa6a600000000 0000000000000001 ffff80001d923ae0
>> ffff80001d938000
>> [    1.626240] be60: ffff80001d923208 0000000000000001 0000000000000000
>> ffffffffffffffff
>> [    1.627208] be80: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    1.628156] bea0: 0000000000000007 ffff000008d97258 ffff000008d972e8
>> 0000000000000008
>> [    1.629103] bec0: 0000000000000000 0000000000000000 ffff000008d60d20
>> ffff80001d938000
>> [    1.630049] bee0: ffff000008e68555 ffff80001d938000 ffff000008d97000
>> ffff80001d93bf40
>> [    1.631013] bf00: ffff000008085d28 ffff80001d93bf40 ffff00000815601c
>> 0000000060000045
>> [    1.631960] bf20: 0000000000000008 0000000000000000 ffffffffffffffff
>> ffff000008d60d20
>> [    1.632908] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    1.633505] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    1.634206] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    1.634894] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    1.635595] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    1.636314] [<ffff00000808f0cc>] secondary_start_kernel+0x164/0x1a0
>> [    1.637082] [<00000000408ea1a4>] 0x408ea1a4
>> [    1.637602] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    1.638352] Internal error: undefined instruction: 0 [#7] PREEMPT SMP
>> [    2.174474] Modules linked in:
>> [    2.174895] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D
>> 4.9.92-ipipe-00107-ge02d0928-dirty #10
>> [    2.175975] Hardware name: linux,dummy-virt (DT)
>> [    2.176500] I-pipe domain: Linux
>> [    2.176873] task: ffff000008da0080 task.stack: ffff000008d90000
>> [    2.177549] PC is at arch_itimer_ack_phys+0x0/0x18
>> [    2.178097] LR is at __ipipe_ack_hrtimer_irq+0x38/0x68
>> [    2.178698] pc : [<ffff0000087899d0>] lr : [<ffff000008156e18>] pstate:
>> 800001c5
>> [    2.179532] sp : ffff80001eedcda0
>> [    2.179914] x29: ffff80001eedcda0 x28: ffff000008d90000
>> [    2.180548] x27: ffff000008d90000 x26: 0000000000000002
>> [    2.181181] x25: ffff80001eeddbb4 x24: ffff000008d60d20
>> [    2.181814] x23: ffff000008db2000 x22: 0000000000000003
>> [    2.182447] x21: 0000000000000000 x20: ffff80001eee7f80
>> [    2.183096] x19: ffff80001d804a00 x18: 0000000000000007
>> [    2.183728] x17: 000000000000000e x16: 0000000000000001
>> [    2.184361] x15: 0000000000000019 x14: ffffffffffffffff
>> [    2.184994] x13: ffffffffffffffff x12: 0000000000000000
>> [    2.185628] x11: 0000000000000004 x10: ffff80001eee0968
>> [    2.186260] x9 : ffff80001eee08f0 x8 : ffff80001eee0978
>> [    2.186909] x7 : ffff80001eee08e8 x6 : 0000000000000000
>> [    2.187543] x5 : 0000800016187000 x4 : ffff000009010000
>> [    2.188176] x3 : 0000000008000000 x2 : ffff0000083d5ae8
>> [    2.188810] x1 : ffff0000083d4ed8 x0 : ffff0000087899d0
>> [    2.189443]
>> [    2.189625] Process swapper/0 (pid: 0, stack limit =
>> 0xffff000008d90038)
>> [    2.190382] Stack: (0xffff80001eedcda0 to 0xffff000008d94000)
>> [    2.191047] Call trace:
>> [    2.191334] Exception stack(0xffff80001eedcc40 to 0xffff80001eedcd70)
>> [    2.192063] cc40: ffff80001d804a00 0001000000000000 ffff80001eedcda0
>> ffff0000087899d0
>> [    2.192945] cc60: 00000000800001c5 0000000000000400 ffff0000087899d0
>> ffff0000083d4ed8
>> [    2.193827] cc80: ffff0000083d5ae8 0000000008000000 ffff000009010000
>> 0000800016187000
>> [    2.194726] cca0: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.195607] ccc0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.196491] cce0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.197373] cd00: 0000000000000007 ffff80001d804a00 ffff80001eee7f80
>> 0000000000000000
>> [    2.198255] cd20: 0000000000000003 ffff000008db2000 ffff000008d60d20
>> ffff80001eeddbb4
>> [    2.199152] cd40: 0000000000000002 ffff000008d90000 ffff000008d90000
>> ffff80001eedcda0
>> [    2.200033] cd60: ffff000008156e18 ffff80001eedcda0
>> [    2.200589] [<ffff0000087899d0>] arch_itimer_ack_phys+0x0/0x18
>> [    2.201251] [<ffff00000815663c>] __ipipe_dispatch_irq+0x7c/0x1d8
>> [    2.201932] [<ffff000008081308>] __ipipe_grab_irq+0x80/0x98
>> [    2.202579] Exception stack(0xffff80001eedce10 to 0xffff80001eedcf40)
>> [    2.203307] ce00:                                   0000000000001fff
>> ffff000008118c28
>> [    2.204188] ce20: ffff80001eedcf80 ffff000008082fc8 ffff80001eedce50
>> 0001000000000000
>> [    2.205069] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402
>> ffff000008eab880
>> [    2.205949] ce60: 0000000000000000 0000000000000002 0000000000010000
>> 0000000000000000
>> [    2.206867] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.207743] cea0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.208618] cec0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.209494] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080
>> 0000000000000001
>> [    2.210369] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20
>> ffff80001eed90a0
>> [    2.211269] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000
>> ffff80001eedcf80
>> [    2.212147] [<ffff0000080817d0>] gic_handle_irq+0xa8/0x100
>> [    2.212766] Exception stack(0xffff80001eedce30 to 0xffff80001eedcf60)
>> [    2.213490] ce20:                                   ffff80001eedce50
>> 0001000000000000
>> [    2.214368] ce40: ffff80001eedcf80 ffff000008155e6c 0000000000000402
>> ffff000008eab880
>> [    2.215262] ce60: 0000000000000000 0000000000000002 0000000000010000
>> 0000000000000000
>> [    2.216138] ce80: 0000000000000000 ffff80001eee08e8 ffff80001eee0978
>> ffff80001eee08f0
>> [    2.217015] cea0: ffff80001eee0968 0000000000000004 0000000000000000
>> ffffffffffffffff
>> [    2.217892] cec0: ffffffffffffffff 0000000000000019 0000000000000001
>> 000000000000000e
>> [    2.218785] cee0: 0000000000000007 ffff000008d5df48 ffff000008e89080
>> 0000000000000001
>> [    2.219664] cf00: 0000000000000402 ffff000008db2000 ffff000008d60d20
>> ffff80001eed90a0
>> [    2.220543] cf20: ffff80001eedd090 ffff000008d90000 ffff000008d90000
>> ffff80001eedcf80
>> [    2.221422] cf40: ffff000008155fd4 ffff80001eedcf80 ffff000008155e6c
>> 0000000020000145
>> [    2.222300] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    2.222881] [<ffff000008155fd4>] __ipipe_do_sync_pipeline+0xa4/0xb8
>> [    2.223587] [<ffff000008156730>] __ipipe_dispatch_irq+0x170/0x1d8
>> [    2.224274] [<ffff000008081274>] __ipipe_grab_ipi+0x1c/0x30
>> [    2.224902] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
>> [    2.225626] 3dc0: 0000000000000000 0000800016187000 0000000000000000
>> ffff80001eee08e8
>> [    2.226501] 3de0: 0000000000000000 0000000000000020 003aa6a600000000
>> 0000000000000001
>> [    2.227442] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088
>> 0000000000000001
>> [    2.228329] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    2.229215] 3e40: 0000000000000001 000000000000000e 0000000000000007
>> ffff000008d97258
>> [    2.230100] 3e60: ffff000008d972e8 0000000000000001 0000000000000000
>> 0000000000000000
>> [    2.231007] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555
>> ffff000008d90000
>> [    2.231895] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28
>> ffff000008d93ef0
>> [    2.232782] 3ec0: ffff00000815601c 0000000060000045 0000000000000001
>> 0000000000000000
>> [    2.233669] 3ee0: ffffffffffffffff ffff000008d60d20
>> [    2.234228] [<ffff00000808178c>] gic_handle_irq+0x64/0x100
>> [    2.234885] Exception stack(0xffff000008d93dc0 to 0xffff000008d93ef0)
>> [    2.235610] 3dc0: 0000000000000000 0000800016187000 0000000000000000
>> ffff80001eee08e8
>> [    2.236488] 3de0: 0000000000000000 0000000000000020 003aa6a600000000
>> 0000000000000001
>> [    2.237365] 3e00: ffff000008da0960 ffff000008d90000 ffff000008da0088
>> 0000000000000001
>> [    2.238241] 3e20: 0000000000000000 ffffffffffffffff ffffffffffffffff
>> 0000000000000019
>> [    2.239151] 3e40: 0000000000000001 000000000000000e 0000000000000007
>> ffff000008d97258
>> [    2.240028] 3e60: ffff000008d972e8 0000000000000001 0000000000000000
>> 0000000000000000
>> [    2.240906] 3e80: ffff000008d60d20 ffff000008d90000 ffff000008e68555
>> ffff000008d90000
>> [    2.241783] 3ea0: ffff000008d97000 ffff000008d93ef0 ffff000008085d28
>> ffff000008d93ef0
>> [    2.242686] 3ec0: ffff00000815601c 0000000060000045 0000000000000001
>> 0000000000000000
>> [    2.243565] 3ee0: ffffffffffffffff ffff000008d60d20
>> [    2.244120] [<ffff000008082fc8>] el1_irq+0xc8/0x108
>> [    2.244675] [<ffff00000815601c>] ipipe_unstall_root+0x34/0x50
>> [    2.245326] [<ffff000008085d28>] arch_cpu_idle+0x48/0x58
>> [    2.245932] [<ffff0000088e3f0c>] default_idle_call+0x34/0x40
>> [    2.246594] [<ffff0000080fac9c>] cpu_startup_entry+0x18c/0x1d0
>> [    2.247256] [<ffff0000088de888>] rest_init+0x80/0x90
>> [    2.247824] [<ffff000008c90b2c>] start_kernel+0x380/0x394
>> [    2.248437] [<ffff000008c901d8>] __primary_switched+0x5c/0x64
>> [    2.249090] Code: 321f0000 d51be320 d5033fdf d65f03c0 (d53be220)
>> [    2.249784] ---[ end trace ee076ceca10e9295 ]---
>> [    2.837878] SMP: failed to stop secondary CPUs 0,2
>> [    2.838465] Kernel Offset: disabled
>> [    2.838887] Memory Limit: 512 MB
>>

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-21 13:55, Jan Kiszka wrote:
> On 2018-06-21 13:20, Pintu Kumar wrote:
>> Dear Jan, Greg,
>>
>> Is there any pointer about this issue?
>> This is blocking my next work..
> 
> Does this solve the issue AND still generate valid UDP checksums (please
> check with wireshark or against a normal networking stack?
> 
> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> index 8e80d3e0b..bb0b0fc12 100644
> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>      if (offset)
>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>  
> -    /* Checksum of the complete data part of the UDP message: */
> -    for (i = 0; i < ufh->iovlen; i++) {
> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> -                                       ufh->wcheck);
> -    }
> -
>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>  			      to + sizeof(struct udphdr),
>  			      fraglen - sizeof(struct udphdr));
>      if (ret)
>  	    return ret;
>  
> +    /* Checksum of the complete data part of the UDP message: */
> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
> +			       fraglen - sizeof(struct udphdr),
> +			       ufh->wcheck);
> +
>      /* Checksum of the udp header: */
>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>  			       sizeof(struct udphdr), ufh->wcheck);
> 

This was definitely wrong. Here is another try:

diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
index 8e80d3e0b..6cf1d369e 100644
--- a/kernel/drivers/net/stack/ipv4/udp/udp.c
+++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
@@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
                           unsigned int offset, unsigned int fraglen)
 {
     struct udpfakehdr *ufh = (struct udpfakehdr *)p;
+    unsigned int datalen = 0;
     int i, ret;
 
 
@@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
     if (offset)
 	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
 
-    /* Checksum of the complete data part of the UDP message: */
-    for (i = 0; i < ufh->iovlen; i++) {
-            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
-                                       ufh->wcheck);
-    }
-
     ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
 			      to + sizeof(struct udphdr),
 			      fraglen - sizeof(struct udphdr));
     if (ret)
 	    return ret;
 
+    /* Checksum of the complete data part of the UDP message: */
+    for (i = 0; i < ufh->iovlen; i++)
+	    datalen += ufh->iov[i].iov_len;
+    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
+			       ufh->wcheck);
+
     /* Checksum of the udp header: */
     ufh->wcheck = csum_partial((unsigned char *)ufh,
 			       sizeof(struct udphdr), ufh->wcheck);

Jan

>>
>> If there is already a fix available for it, please let me know.
>> Or, should I need analyze this freshly....
>> I wonder why no body else is facing this issue....
> 
> Users may still have the related memory protection features off which
> surface this issue.
> 
> Jan
> 

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-21 15:41, Jan Kiszka wrote:
> On 2018-06-21 13:55, Jan Kiszka wrote:
>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>> Dear Jan, Greg,
>>>
>>> Is there any pointer about this issue?
>>> This is blocking my next work..
>>
>> Does this solve the issue AND still generate valid UDP checksums (please
>> check with wireshark or against a normal networking stack?
>>
>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> index 8e80d3e0b..bb0b0fc12 100644
>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>      if (offset)
>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>  
>> -    /* Checksum of the complete data part of the UDP message: */
>> -    for (i = 0; i < ufh->iovlen; i++) {
>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>> -                                       ufh->wcheck);
>> -    }
>> -
>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>  			      to + sizeof(struct udphdr),
>>  			      fraglen - sizeof(struct udphdr));
>>      if (ret)
>>  	    return ret;
>>  
>> +    /* Checksum of the complete data part of the UDP message: */
>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>> +			       fraglen - sizeof(struct udphdr),
>> +			       ufh->wcheck);
>> +
>>      /* Checksum of the udp header: */
>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>  			       sizeof(struct udphdr), ufh->wcheck);
>>
> 
> This was definitely wrong. Here is another try:
> 
> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> index 8e80d3e0b..6cf1d369e 100644
> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>                            unsigned int offset, unsigned int fraglen)
>  {
>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
> +    unsigned int datalen = 0;
>      int i, ret;
>  
>  
> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>      if (offset)
>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>  
> -    /* Checksum of the complete data part of the UDP message: */
> -    for (i = 0; i < ufh->iovlen; i++) {
> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> -                                       ufh->wcheck);
> -    }
> -
>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>  			      to + sizeof(struct udphdr),
>  			      fraglen - sizeof(struct udphdr));
>      if (ret)
>  	    return ret;
>  
> +    /* Checksum of the complete data part of the UDP message: */
> +    for (i = 0; i < ufh->iovlen; i++)
> +	    datalen += ufh->iov[i].iov_len;
> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
> +			       ufh->wcheck);
> +
>      /* Checksum of the udp header: */
>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>  			       sizeof(struct udphdr), ufh->wcheck);
> 

OK, this won't work either if fraglen < datalen. We have to rework this
more fundamentally.

As a workaround, run the kernel with nosmap.

Jan

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

> As a workaround, run the kernel with nosmap.
OK, I guessed it correctly and tried with kernel commandline "nosmap".
Because it works normally on virtual box.

With "nosmap" I don't see any kernel oops, but there are no prints either.
I mean in my rtnet-server/client, I have given some print statement.
But I am not seeing those print statement coming on the console now.
The print statements are coming for normal "xeno_task" application.

Is there any clue for this ?


Thanks,
Pintu





On Thu, Jun 21, 2018 at 8:27 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2018-06-21 15:41, Jan Kiszka wrote:
> > On 2018-06-21 13:55, Jan Kiszka wrote:
> >> On 2018-06-21 13:20, Pintu Kumar wrote:
> >>> Dear Jan, Greg,
> >>>
> >>> Is there any pointer about this issue?
> >>> This is blocking my next work..
> >>
> >> Does this solve the issue AND still generate valid UDP checksums (please
> >> check with wireshark or against a normal networking stack?
> >>
> >> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> >> index 8e80d3e0b..bb0b0fc12 100644
> >> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> >> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> >> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> >>      if (offset)
> >>          return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
> >>
> >> -    /* Checksum of the complete data part of the UDP message: */
> >> -    for (i = 0; i < ufh->iovlen; i++) {
> >> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> >> -                                       ufh->wcheck);
> >> -    }
> >> -
> >>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> >>                            to + sizeof(struct udphdr),
> >>                            fraglen - sizeof(struct udphdr));
> >>      if (ret)
> >>          return ret;
> >>
> >> +    /* Checksum of the complete data part of the UDP message: */
> >> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
> >> +                           fraglen - sizeof(struct udphdr),
> >> +                           ufh->wcheck);
> >> +
> >>      /* Checksum of the udp header: */
> >>      ufh->wcheck = csum_partial((unsigned char *)ufh,
> >>                             sizeof(struct udphdr), ufh->wcheck);
> >>
> >
> > This was definitely wrong. Here is another try:
> >
> > diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > index 8e80d3e0b..6cf1d369e 100644
> > --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> > +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> >                            unsigned int offset, unsigned int fraglen)
> >  {
> >      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
> > +    unsigned int datalen = 0;
> >      int i, ret;
> >
> >
> > @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> >      if (offset)
> >           return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
> >
> > -    /* Checksum of the complete data part of the UDP message: */
> > -    for (i = 0; i < ufh->iovlen; i++) {
> > -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> > -                                       ufh->wcheck);
> > -    }
> > -
> >      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> >                             to + sizeof(struct udphdr),
> >                             fraglen - sizeof(struct udphdr));
> >      if (ret)
> >           return ret;
> >
> > +    /* Checksum of the complete data part of the UDP message: */
> > +    for (i = 0; i < ufh->iovlen; i++)
> > +         datalen += ufh->iov[i].iov_len;
> > +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
> > +                            ufh->wcheck);
> > +
> >      /* Checksum of the udp header: */
> >      ufh->wcheck = csum_partial((unsigned char *)ufh,
> >                              sizeof(struct udphdr), ufh->wcheck);
> >
>
> OK, this won't work either if fraglen < datalen. We have to rework this
> more fundamentally.
>
> As a workaround, run the kernel with nosmap.
>
> Jan

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Dear Jan,

Till now I haven't had any success for running my rtnet demo test
either or x86 or arm.
I even upgraded to xenomai-next (both kernel and user space) but still no luck.

Now even the "loopback" test is not working on xenomai-next.

Can you confirm, if any of the below rtnet test works for you with
Kernel 4.9 and xenomai-next.
- xenomai-3-next/testsuite/smokey/net_udp
- url = https://git.code.sf.net/p/rtnet/code  => examples/generic/
- Or a simple UDP client/server program, compiled with xenomai posix skin

I already tried with "nosmap" but there are no response coming (I am
getting no prints on console, but client is terminated).
If there is any work around to make rtnet works, please let me know.


Thanks,
Pintu

On Mon, Jun 25, 2018 at 6:26 PM Pintu Kumar <pintu.ping@gmail.com> wrote:
>
> > As a workaround, run the kernel with nosmap.
> OK, I guessed it correctly and tried with kernel commandline "nosmap".
> Because it works normally on virtual box.
>
> With "nosmap" I don't see any kernel oops, but there are no prints either.
> I mean in my rtnet-server/client, I have given some print statement.
> But I am not seeing those print statement coming on the console now.
> The print statements are coming for normal "xeno_task" application.
>
> Is there any clue for this ?
>
>
> Thanks,
> Pintu
>
>
>
>
>
> On Thu, Jun 21, 2018 at 8:27 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >
> > On 2018-06-21 15:41, Jan Kiszka wrote:
> > > On 2018-06-21 13:55, Jan Kiszka wrote:
> > >> On 2018-06-21 13:20, Pintu Kumar wrote:
> > >>> Dear Jan, Greg,
> > >>>
> > >>> Is there any pointer about this issue?
> > >>> This is blocking my next work..
> > >>
> > >> Does this solve the issue AND still generate valid UDP checksums (please
> > >> check with wireshark or against a normal networking stack?
> > >>
> > >> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > >> index 8e80d3e0b..bb0b0fc12 100644
> > >> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> > >> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > >> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> > >>      if (offset)
> > >>          return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
> > >>
> > >> -    /* Checksum of the complete data part of the UDP message: */
> > >> -    for (i = 0; i < ufh->iovlen; i++) {
> > >> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> > >> -                                       ufh->wcheck);
> > >> -    }
> > >> -
> > >>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> > >>                            to + sizeof(struct udphdr),
> > >>                            fraglen - sizeof(struct udphdr));
> > >>      if (ret)
> > >>          return ret;
> > >>
> > >> +    /* Checksum of the complete data part of the UDP message: */
> > >> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
> > >> +                           fraglen - sizeof(struct udphdr),
> > >> +                           ufh->wcheck);
> > >> +
> > >>      /* Checksum of the udp header: */
> > >>      ufh->wcheck = csum_partial((unsigned char *)ufh,
> > >>                             sizeof(struct udphdr), ufh->wcheck);
> > >>
> > >
> > > This was definitely wrong. Here is another try:
> > >
> > > diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > > index 8e80d3e0b..6cf1d369e 100644
> > > --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> > > +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> > > @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> > >                            unsigned int offset, unsigned int fraglen)
> > >  {
> > >      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
> > > +    unsigned int datalen = 0;
> > >      int i, ret;
> > >
> > >
> > > @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
> > >      if (offset)
> > >           return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
> > >
> > > -    /* Checksum of the complete data part of the UDP message: */
> > > -    for (i = 0; i < ufh->iovlen; i++) {
> > > -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> > > -                                       ufh->wcheck);
> > > -    }
> > > -
> > >      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> > >                             to + sizeof(struct udphdr),
> > >                             fraglen - sizeof(struct udphdr));
> > >      if (ret)
> > >           return ret;
> > >
> > > +    /* Checksum of the complete data part of the UDP message: */
> > > +    for (i = 0; i < ufh->iovlen; i++)
> > > +         datalen += ufh->iov[i].iov_len;
> > > +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
> > > +                            ufh->wcheck);
> > > +
> > >      /* Checksum of the udp header: */
> > >      ufh->wcheck = csum_partial((unsigned char *)ufh,
> > >                              sizeof(struct udphdr), ufh->wcheck);
> > >
> >
> > OK, this won't work either if fraglen < datalen. We have to rework this
> > more fundamentally.
> >
> > As a workaround, run the kernel with nosmap.
> >
> > Jan

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-26 13:08, Pintu Kumar wrote:
> Dear Jan,
> 
> Till now I haven't had any success for running my rtnet demo test
> either or x86 or arm.
> I even upgraded to xenomai-next (both kernel and user space) but still no luck.
> 
> Now even the "loopback" test is not working on xenomai-next.

What means "now"? Did it work before? What was the setup then?

> 
> Can you confirm, if any of the below rtnet test works for you with
> Kernel 4.9 and xenomai-next.
> - xenomai-3-next/testsuite/smokey/net_udp
> - url = https://git.code.sf.net/p/rtnet/code  => examples/generic/
> - Or a simple UDP client/server program, compiled with xenomai posix skin
> 
> I already tried with "nosmap" but there are no response coming (I am
> getting no prints on console, but client is terminated).
> If there is any work around to make rtnet works, please let me know.
> 

I'm afraid there is currently a lack of bandwidth here to dive into the
details. Either you drill deeper, systematically, and report your
findings for discussions and suggestions on the list, or you need to be
patient until someone finds the time to reproduce and resolve the issue(s).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

Dear Jan,

> What means "now"? Did it work before? What was the setup then?
rtnet loopback test is working (even with older kernel) on my Virtual
Box with Ubuntu 32-bit.

But it never worked for me on x86_64 machine.
So, at first I wonder what is the minimum criteria to make rtnet work
on any system.
I am sure it would have definitely worked in the past. No?

> Either you drill deeper, systematically, and report your
> findings for discussions and suggestions on the list
Yes sure, I am ready to do that.
Once I know the little bit of history and current problems with rtnet,
I can definitely start contributing.
At first I assumed that there might be some fix already available.
So, instead of reinventing the wheel, I thought to first check out.

I am really sorry for troubling you!
I will try to look into this issue and see if I can find something.

Thank you!
Pintu

On Wed, Jun 27, 2018 at 3:50 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2018-06-26 13:08, Pintu Kumar wrote:
> > Dear Jan,
> >
> > Till now I haven't had any success for running my rtnet demo test
> > either or x86 or arm.
> > I even upgraded to xenomai-next (both kernel and user space) but still no luck.
> >
> > Now even the "loopback" test is not working on xenomai-next.
>
> What means "now"? Did it work before? What was the setup then?
>
> >
> > Can you confirm, if any of the below rtnet test works for you with
> > Kernel 4.9 and xenomai-next.
> > - xenomai-3-next/testsuite/smokey/net_udp
> > - url = https://git.code.sf.net/p/rtnet/code  => examples/generic/
> > - Or a simple UDP client/server program, compiled with xenomai posix skin
> >
> > I already tried with "nosmap" but there are no response coming (I am
> > getting no prints on console, but client is terminated).
> > If there is any work around to make rtnet works, please let me know.
> >
>
> I'm afraid there is currently a lack of bandwidth here to dive into the
> details. Either you drill deeper, systematically, and report your
> findings for discussions and suggestions on the list, or you need to be
> patient until someone finds the time to reproduce and resolve the issue(s).
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-27 12:56, Pintu Kumar wrote:
> Dear Jan,
> 
>> What means "now"? Did it work before? What was the setup then?
> rtnet loopback test is working (even with older kernel) on my Virtual
> Box with Ubuntu 32-bit.
> 
> But it never worked for me on x86_64 machine.
> So, at first I wonder what is the minimum criteria to make rtnet work
> on any system.
> I am sure it would have definitely worked in the past. No?

It surely worked, but it wasn't looked after consistently in the (even
not so) recent past since then.

> 
>> Either you drill deeper, systematically, and report your
>> findings for discussions and suggestions on the list
> Yes sure, I am ready to do that.
> Once I know the little bit of history and current problems with rtnet,
> I can definitely start contributing.
> At first I assumed that there might be some fix already available.
> So, instead of reinventing the wheel, I thought to first check out.

The main recent problem was (and still is) related to SMAP (supervisor
mode access prevention) which started to bite RTnet due to its sloppy
way of taken data from user space or pushing it back there (not using
proper copy-to/from services). The problem in the UDP code you ran into
is related to the code still doing direct access (checksum generation
over userspace data, rather than over the kernel copy).

With nosmap, that particular issue should no longer occur (at least as
long as we can ask the kernel for this relaxation), so I suspect the
other effect you see now is something else.

Jan

> 
> I am really sorry for troubling you!
> I will try to look into this issue and see if I can find something.
> 
> Thank you!
> Pintu
> 
> On Wed, Jun 27, 2018 at 3:50 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>> On 2018-06-26 13:08, Pintu Kumar wrote:
>>> Dear Jan,
>>>
>>> Till now I haven't had any success for running my rtnet demo test
>>> either or x86 or arm.
>>> I even upgraded to xenomai-next (both kernel and user space) but still no luck.
>>>
>>> Now even the "loopback" test is not working on xenomai-next.
>>
>> What means "now"? Did it work before? What was the setup then?
>>
>>>
>>> Can you confirm, if any of the below rtnet test works for you with
>>> Kernel 4.9 and xenomai-next.
>>> - xenomai-3-next/testsuite/smokey/net_udp
>>> - url = https://git.code.sf.net/p/rtnet/code  => examples/generic/
>>> - Or a simple UDP client/server program, compiled with xenomai posix skin
>>>
>>> I already tried with "nosmap" but there are no response coming (I am
>>> getting no prints on console, but client is terminated).
>>> If there is any work around to make rtnet works, please let me know.
>>>
>>
>> I'm afraid there is currently a lack of bandwidth here to dive into the
>> details. Either you drill deeper, systematically, and report your
>> findings for discussions and suggestions on the list, or you need to be
>> patient until someone finds the time to reproduce and resolve the issue(s).
>>
>> Jan
>>
>> --
>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> Corporate Competence Center Embedded Linux


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

> With nosmap, that particular issue should no longer occur (at least as
> long as we can ask the kernel for this relaxation), so I suspect the
> other effect you see now is something else.

Just for the information, that issue occurred even on Beagle bone, and
there is no smap on arm.
However, it works on virtual box. how ?

This is the back trace seen on ARM Beagle bone:
{{{
[  970.717610] Unhandled fault: page domain fault (0x01b) at 0xbe926678
[  970.724030] pgd = da830000
[  970.726755] [be926678] *pgd=9cda7831, *pte=9a61634f, *ppte=9a61683f
[  970.733107] Internal error: : 1b [#1] PREEMPT SMP ARM
[  970.738188] Modules linked in: rt_loopback rtpacket rtudp rtipv4 rtnet
[  970.744831] CPU: 0 PID: 564 Comm: rtnet-client Not tainted 4.9.51-scel-beagle
bone #6
[  970.752611] Hardware name: Generic AM33XX (Flattened Device Tree)
[  970.758733] I-pipe domain: Linux
[  970.761979] task: dce0d740 task.stack: dce08000
[  970.766567] PC is at rt_udp_getfrag+0x48/0x118 [rtudp]
[  970.771796] LR is at rt_ip_build_xmit+0x230/0x2f4 [rtipv4]
[  970.777313] pc : [<bf026be0>]    lr : [<bf019688>]    psr: 20060013
[  970.777313] sp : dce09cf0  ip : bf026bb0  fp : dce09d14
[  970.788846] r10: dcb35824  r9 : 00000400  r8 : c12050a0
[  970.794099] r7 : dcd9e364  r6 : 0000000a  r5 : dce09db4  r4 : 00000000
[  970.800659] r3 : be926678  r2 : 00000000  r1 : be926678  r0 : 00000000
[  970.807221] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  970.814392] Control: 10c5387d  Table: 9a830019  DAC: 00000051
[  970.820168] Process rtnet-client (pid: 564, stack limit = 0xdce08220)
[  970.826642] Stack: (0xdce09cf0 to 0xdce0a000)
[  970.831025] 9ce0:                                     dce09d14 dce09d00 dcc0b
600 dcd9e2c0
[  970.839249] 9d00: 00000040 dcd9e364 dce09d6c dce09d18 bf019688 bf026ba4 c1205
0a0 00000000
[  970.847470] 9d20: dce09d6c dce09d30 bf017d4c bf01c13c 0000000a dce09db4 bf026
b98 0000000f
[  970.855692] 9d40: 00000000 c120418c dce09e64 00000002 dce09d98 dcb35800 dce09
dd4 00000000
[  970.863912] 9d60: dce09ebc dce09d70 bf027098 bf019464 dce09e64 00000000 00000
000 0000e015
[  970.872133] 9d80: 0100007f c1199b50 c12050a0 00000000 c0173fd0 dce09dd4 be926
6cc 00000010
[  970.880354] 9da0: be926678 00000001 00000000 00000000 00000000 e015a816 00000
a00 0100007f
[  970.888576] 9dc0: 0100007f dcb35800 be926678 00000001 00000000 be926714 00000
002 c0179034
[  970.896798] 9de0: c12095fc 00000002 c1200000 c010332c c0103340 c015ef90 c0103
32c ffffffff
[  970.905020] 9e00: c1209f90 00000002 c1200000 c0112abc c0112ad0 c015ef90 dce09
e4c dce09e28
[  970.913240] 9e20: 00000000 00000000 00000000 dce08000 c01ab530 c015f2d4 00000
000 00000000
[  970.921461] 9e40: 00000003 00000001 c020cdec c027ea78 c1199b50 e0150002 01000
07f 00000000
[  970.929682] 9e60: 00000000 0100007f 00000000 00000000 00000000 00000000 00000
000 00000000
[  970.937904] 9e80: 00000000 00000000 dcc0b600 00040933 dce09ebc dcb35800 c119a
cf0 40060013
[  970.946124] 9ea0: 00000000 00000003 dce09efc c11989b0 dce09ef4 dce09ec0 c027f
6f0 bf026cbc
[  970.954346] 9ec0: 00000052 dce0d740 00000000 00000003 00000000 00000051 00000
001 e0580008
[  970.962568] 9ee0: c0285920 c11989b0 dce09f34 dce09ef8 c02859b0 c027f65c e0580
008 be9266cc
[  970.970789] 9f00: 00000010 be926678 00000001 00000000 00000000 00000000 dce09
fb0 dce09fb0
[  970.979010] 9f20: 00000052 c12050a0 dce09f6c dce09f38 c029351c c028592c 00000
000 dce09f48
[  970.987231] 9f40: c02d1f68 df9429b0 df9419b0 c12050a0 20060013 c13f7600 c13f7
600 c11989b0
[  970.995452] 9f60: dce09fac dce09f70 c020d5a0 c029340c c11989b0 c119acf0 c13f7
600 dce09fb0
[  971.003675] 9f80: c02d8de4 00022080 00000000 be926680 000f0042 c0109288 dce08
000 00000002
[  971.011896] 9fa0: 00000000 dce09fb0 c01091d4 c020d4a8 10000054 00000003 be926
680 00000000
[  971.020119] 9fc0: 00022080 00000000 be926680 000f0042 00000003 00000002 00001
5e0 00000000
[  971.028340] 9fe0: b6f482d0 be926650 00000000 b6f2f044 00060030 10000054 00000
000 00000000
[  971.036617] [<bf026be0>] (rt_udp_getfrag [rtudp]) from [<bf019688>] (rt_ip_bu
ild_xmit+0x230/0x2f4 [rtipv4])
[  971.046445] [<bf019688>] (rt_ip_build_xmit [rtipv4]) from [<bf027098>] (rt_ud
p_sendmsg+0x3e8/0x45c [rtudp])
[  971.056256] [<bf027098>] (rt_udp_sendmsg [rtudp]) from [<c027f6f0>] (rtdm_fd_
sendmsg+0xa0/0x248)
[  971.065095] [<c027f6f0>] (rtdm_fd_sendmsg) from [<c02859b0>] (CoBaLt_sendmsg+
0x90/0x98)
[  971.073156] [<c02859b0>] (CoBaLt_sendmsg) from [<c029351c>] (ipipe_syscall_ho
ok+0x11c/0x400)
[  971.081647] [<c029351c>] (ipipe_syscall_hook) from [<c020d5a0>] (__ipipe_noti
fy_syscall+0x104/0x1d4)
[  971.090841] [<c020d5a0>] (__ipipe_notify_syscall) from [<c01091d4>] (pipeline
_syscall+0x8/0x24)
[  971.099591] Code: da00000a e5953014 e1a02000 e0831184 (e7930184)
[  971.105740] ---[ end trace 0e2d3ec12e870a84 ]---
}}}

I will try to debug this issue if my time permits.

Thanks,
Pintu
On Wed, Jun 27, 2018 at 4:44 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2018-06-27 12:56, Pintu Kumar wrote:
> > Dear Jan,
> >
> >> What means "now"? Did it work before? What was the setup then?
> > rtnet loopback test is working (even with older kernel) on my Virtual
> > Box with Ubuntu 32-bit.
> >
> > But it never worked for me on x86_64 machine.
> > So, at first I wonder what is the minimum criteria to make rtnet work
> > on any system.
> > I am sure it would have definitely worked in the past. No?
>
> It surely worked, but it wasn't looked after consistently in the (even
> not so) recent past since then.
>
> >
> >> Either you drill deeper, systematically, and report your
> >> findings for discussions and suggestions on the list
> > Yes sure, I am ready to do that.
> > Once I know the little bit of history and current problems with rtnet,
> > I can definitely start contributing.
> > At first I assumed that there might be some fix already available.
> > So, instead of reinventing the wheel, I thought to first check out.
>
> The main recent problem was (and still is) related to SMAP (supervisor
> mode access prevention) which started to bite RTnet due to its sloppy
> way of taken data from user space or pushing it back there (not using
> proper copy-to/from services). The problem in the UDP code you ran into
> is related to the code still doing direct access (checksum generation
> over userspace data, rather than over the kernel copy).
>
> With nosmap, that particular issue should no longer occur (at least as
> long as we can ask the kernel for this relaxation), so I suspect the
> other effect you see now is something else.
>
> Jan
>
> >
> > I am really sorry for troubling you!
> > I will try to look into this issue and see if I can find something.
> >
> > Thank you!
> > Pintu
> >
> > On Wed, Jun 27, 2018 at 3:50 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >>
> >> On 2018-06-26 13:08, Pintu Kumar wrote:
> >>> Dear Jan,
> >>>
> >>> Till now I haven't had any success for running my rtnet demo test
> >>> either or x86 or arm.
> >>> I even upgraded to xenomai-next (both kernel and user space) but still no luck.
> >>>
> >>> Now even the "loopback" test is not working on xenomai-next.
> >>
> >> What means "now"? Did it work before? What was the setup then?
> >>
> >>>
> >>> Can you confirm, if any of the below rtnet test works for you with
> >>> Kernel 4.9 and xenomai-next.
> >>> - xenomai-3-next/testsuite/smokey/net_udp
> >>> - url = https://git.code.sf.net/p/rtnet/code  => examples/generic/
> >>> - Or a simple UDP client/server program, compiled with xenomai posix skin
> >>>
> >>> I already tried with "nosmap" but there are no response coming (I am
> >>> getting no prints on console, but client is terminated).
> >>> If there is any work around to make rtnet works, please let me know.
> >>>
> >>
> >> I'm afraid there is currently a lack of bandwidth here to dive into the
> >> details. Either you drill deeper, systematically, and report your
> >> findings for discussions and suggestions on the list, or you need to be
> >> patient until someone finds the time to reproduce and resolve the issue(s).
> >>
> >> Jan
> >>
> >> --
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
>
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 2018-06-27 16:12, Pintu Kumar wrote:
>> With nosmap, that particular issue should no longer occur (at least as
>> long as we can ask the kernel for this relaxation), so I suspect the
>> other effect you see now is something else.
> 
> Just for the information, that issue occurred even on Beagle bone, and
> there is no smap on arm.
> However, it works on virtual box. how ?

Does it work when you go back to Xenomai 3.0.6? Then it would be a
regression of the copy-to/from-userspace improvements done after that
release, you could start bisecting which commit introduced it.

But I'm not sure right now if there isn't something similar to smap
(then just without a knob to turn it off) in the ARM architecture as well...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

On Wed, Jun 27, 2018 at 9:47 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2018-06-27 16:12, Pintu Kumar wrote:
> >> With nosmap, that particular issue should no longer occur (at least as
> >> long as we can ask the kernel for this relaxation), so I suspect the
> >> other effect you see now is something else.
> >
> > Just for the information, that issue occurred even on Beagle bone, and
> > there is no smap on arm.
> > However, it works on virtual box. how ?
>
> Does it work when you go back to Xenomai 3.0.6? Then it would be a
> regression of the copy-to/from-userspace improvements done after that
> release, you could start bisecting which commit introduced it.
>

Nope, I never had success with rtnet on arm architecture (mainly
beagle bone), even with the old 3.0.6.
At least earlier, on x86_64 (SkyLake), I saw the loopback part working
with 3.0.6 and my old xenomai kernel 4.9.51 (without those rtnet
fixes) and with "nosmap" in commandline.

But now it looks like the situation becomes more worse.
I will have to freshly look into it, but since I use prepare_kernel to
apply as single patch, it might be painful to bisect it.
Will try to find the root cause though....

Hoping for the best...

> But I'm not sure right now if there isn't something similar to smap
> (then just without a knob to turn it off) in the ARM architecture as well...
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Greg Gallagher]

I don't think Beaglebone is supported currently in RTNet.

On Wed, Jun 27, 2018 at 1:13 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> On Wed, Jun 27, 2018 at 9:47 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>> On 2018-06-27 16:12, Pintu Kumar wrote:
>> >> With nosmap, that particular issue should no longer occur (at least as
>> >> long as we can ask the kernel for this relaxation), so I suspect the
>> >> other effect you see now is something else.
>> >
>> > Just for the information, that issue occurred even on Beagle bone, and
>> > there is no smap on arm.
>> > However, it works on virtual box. how ?
>>
>> Does it work when you go back to Xenomai 3.0.6? Then it would be a
>> regression of the copy-to/from-userspace improvements done after that
>> release, you could start bisecting which commit introduced it.
>>
>
> Nope, I never had success with rtnet on arm architecture (mainly
> beagle bone), even with the old 3.0.6.
> At least earlier, on x86_64 (SkyLake), I saw the loopback part working
> with 3.0.6 and my old xenomai kernel 4.9.51 (without those rtnet
> fixes) and with "nosmap" in commandline.
>
> But now it looks like the situation becomes more worse.
> I will have to freshly look into it, but since I use prepare_kernel to
> apply as single patch, it might be painful to bisect it.
> Will try to find the root cause though....
>
> Hoping for the best...
>
>> But I'm not sure right now if there isn't something similar to smap
>> (then just without a knob to turn it off) in the ARM architecture as well...
>>
>> Jan
>>
>> --
>> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
>> Corporate Competence Center Embedded Linux
>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

On Wed, Jun 27, 2018 at 10:47 PM Greg Gallagher <greg@embeddedgreg.com> wrote:
>
> I don't think Beaglebone is supported currently in RTNet.
>

But I am just checking loopback interface with 127.0.0.1
I hope that should work on all boards and arch, even without
installing the net driver.

> On Wed, Jun 27, 2018 at 1:13 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > On Wed, Jun 27, 2018 at 9:47 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >>
> >> On 2018-06-27 16:12, Pintu Kumar wrote:
> >> >> With nosmap, that particular issue should no longer occur (at least as
> >> >> long as we can ask the kernel for this relaxation), so I suspect the
> >> >> other effect you see now is something else.
> >> >
> >> > Just for the information, that issue occurred even on Beagle bone, and
> >> > there is no smap on arm.
> >> > However, it works on virtual box. how ?
> >>
> >> Does it work when you go back to Xenomai 3.0.6? Then it would be a
> >> regression of the copy-to/from-userspace improvements done after that
> >> release, you could start bisecting which commit introduced it.
> >>
> >
> > Nope, I never had success with rtnet on arm architecture (mainly
> > beagle bone), even with the old 3.0.6.
> > At least earlier, on x86_64 (SkyLake), I saw the loopback part working
> > with 3.0.6 and my old xenomai kernel 4.9.51 (without those rtnet
> > fixes) and with "nosmap" in commandline.
> >
> > But now it looks like the situation becomes more worse.
> > I will have to freshly look into it, but since I use prepare_kernel to
> > apply as single patch, it might be painful to bisect it.
> > Will try to find the root cause though....
> >
> > Hoping for the best...
> >
> >> But I'm not sure right now if there isn't something similar to smap
> >> (then just without a knob to turn it off) in the ARM architecture as well...
> >>
> >> Jan
> >>
> >> --
> >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> >> Corporate Competence Center Embedded Linux
> >
> > _______________________________________________
> > Xenomai mailing list
> > Xenomai@xenomai.org
> > https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 06/27/2018 07:29 PM, Pintu Kumar wrote:
> On Wed, Jun 27, 2018 at 10:47 PM Greg Gallagher <greg@embeddedgreg.com> wrote:
>>
>> I don't think Beaglebone is supported currently in RTNet.
>>
> 
> But I am just checking loopback interface with 127.0.0.1
> I hope that should work on all boards and arch, even without
> installing the net driver.
> 

You must be the first and only user giving any feedback about rtnet over
ARM if my memory serves me well. So either all others have been hiding
behind for all these years, or you may be the only specimen of that species.

FWIW, all of my rtnet fixes that went in since 3.0.6 have been tested on
x86 exclusively.

-- 
Philippe.

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Pintu Kumar]

> FWIW, all of my rtnet fixes that went in since 3.0.6 have been tested on
> x86 exclusively.

For us, even the xeno-test/net_udp fails on both x86/arm.
I also verified the udp test (loopback) that exists on rtnet repo:
https://git.code.sf.net/p/rtnet/code
Under: examples/generic/

Then, I also developed a common rtnet udp server/client using posix skin.
When I build and verify it on normal kernel it works fine.
But when I build it with Xenomai posix skin, I get the kernel oops on
both x86/arm.

Now it fails with xenomai-3/next (latest) and with Kernel 4.9, on x86.
Earlier (1.5 months ago), I remember, rtnet loopback worked on x86_64
with 3.0.6, with "nosmap".
 If I run the same on Virtual Box Ubuntu 32-bit with same Kernel 4.9, it works.

So, I wonder, normally how do you verify rtnet sample application.
Please share the test applications and the steps you follow.


Thanks,
Pintu

On Wed, Jun 27, 2018 at 11:40 PM Philippe Gerum <rpm@xenomai.org> wrote:
>
> On 06/27/2018 07:29 PM, Pintu Kumar wrote:
> > On Wed, Jun 27, 2018 at 10:47 PM Greg Gallagher <greg@embeddedgreg.com> wrote:
> >>
> >> I don't think Beaglebone is supported currently in RTNet.
> >>
> >
> > But I am just checking loopback interface with 127.0.0.1
> > I hope that should work on all boards and arch, even without
> > installing the net driver.
> >
>
> You must be the first and only user giving any feedback about rtnet over
> ARM if my memory serves me well. So either all others have been hiding
> behind for all these years, or you may be the only specimen of that species.
>
> FWIW, all of my rtnet fixes that went in since 3.0.6 have been tested on
> x86 exclusively.
>
> --
> Philippe.

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 06/28/2018 12:40 PM, Pintu Kumar wrote:
>> FWIW, all of my rtnet fixes that went in since 3.0.6 have been tested on
>> x86 exclusively.
> 
> For us, even the xeno-test/net_udp fails on both x86/arm.
> I also verified the udp test (loopback) that exists on rtnet repo:
> https://git.code.sf.net/p/rtnet/code
> Under: examples/generic/
> 
> Then, I also developed a common rtnet udp server/client using posix skin.
> When I build and verify it on normal kernel it works fine.
> But when I build it with Xenomai posix skin, I get the kernel oops on
> both x86/arm.
> > Now it fails with xenomai-3/next (latest) and with Kernel 4.9, on x86.
> Earlier (1.5 months ago), I remember, rtnet loopback worked on x86_64
> with 3.0.6, with "nosmap".
>  If I run the same on Virtual Box Ubuntu 32-bit with same Kernel 4.9, it works.
> 
> So, I wonder, normally how do you verify rtnet sample application.

Sorry, I don't maintain rtnet.

> Please share the test applications and the steps you follow.
> 

Run the smokey test suite over real x86 hw and a real ethernet
interface, not the loopback. These were the tests conditions for the
changes addressing the smap issue.

I have no bandwidth to invest in rtnet at the moment, so you may want to
confront and debug the issue directly. I'm sure that any proposal for a
fix once the bug is spotted would be welcome, reviewed and commented on
this list.

-- 
Philippe.

Re: [Xenomai] Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 06/27/2018 06:17 PM, Jan Kiszka wrote:
> On 2018-06-27 16:12, Pintu Kumar wrote:
>>> With nosmap, that particular issue should no longer occur (at least as
>>> long as we can ask the kernel for this relaxation), so I suspect the
>>> other effect you see now is something else.
>>
>> Just for the information, that issue occurred even on Beagle bone, and
>> there is no smap on arm.
>> However, it works on virtual box. how ?
> 
> Does it work when you go back to Xenomai 3.0.6? Then it would be a
> regression of the copy-to/from-userspace improvements done after that
> release, you could start bisecting which commit introduced it.
> 
> But I'm not sure right now if there isn't something similar to smap
> (then just without a knob to turn it off) in the ARM architecture as well...
> 

AFAIR, use of protection domains is built in depending on a selection by
the CPU support code.

-- 
Philippe.

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 6/21/18 4:57 PM, Jan Kiszka wrote:
> On 2018-06-21 15:41, Jan Kiszka wrote:
>> On 2018-06-21 13:55, Jan Kiszka wrote:
>>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>>> Dear Jan, Greg,
>>>>
>>>> Is there any pointer about this issue?
>>>> This is blocking my next work..
>>>
>>> Does this solve the issue AND still generate valid UDP checksums (please
>>> check with wireshark or against a normal networking stack?
>>>
>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> index 8e80d3e0b..bb0b0fc12 100644
>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>      if (offset)
>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>  
>>> -    /* Checksum of the complete data part of the UDP message: */
>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>> -                                       ufh->wcheck);
>>> -    }
>>> -
>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>  			      to + sizeof(struct udphdr),
>>>  			      fraglen - sizeof(struct udphdr));
>>>      if (ret)
>>>  	    return ret;
>>>  
>>> +    /* Checksum of the complete data part of the UDP message: */
>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>>> +			       fraglen - sizeof(struct udphdr),
>>> +			       ufh->wcheck);
>>> +
>>>      /* Checksum of the udp header: */
>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>
>>
>> This was definitely wrong. Here is another try:
>>
>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> index 8e80d3e0b..6cf1d369e 100644
>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>                            unsigned int offset, unsigned int fraglen)
>>  {
>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>> +    unsigned int datalen = 0;
>>      int i, ret;
>>  
>>  
>> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>      if (offset)
>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>  
>> -    /* Checksum of the complete data part of the UDP message: */
>> -    for (i = 0; i < ufh->iovlen; i++) {
>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>> -                                       ufh->wcheck);
>> -    }
>> -
>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>  			      to + sizeof(struct udphdr),
>>  			      fraglen - sizeof(struct udphdr));
>>      if (ret)
>>  	    return ret;
>>  
>> +    /* Checksum of the complete data part of the UDP message: */
>> +    for (i = 0; i < ufh->iovlen; i++)
>> +	    datalen += ufh->iov[i].iov_len;
>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
>> +			       ufh->wcheck);
>> +
>>      /* Checksum of the udp header: */
>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>  			       sizeof(struct udphdr), ufh->wcheck);
>>
> 
> OK, this won't work either if fraglen < datalen. We have to rework this
> more fundamentally.
> 
> As a workaround, run the kernel with nosmap.
> 
> Jan
> 

There is no point in debugging the splash with UDP until this one is addressed since running with nosmap won't be acceptable in many cases, so I took a stab at fixing this issue first. This fix is somewhat ugly, but a clean solution would involve reworking the entire handling of fragmented packets, so let's see if the logic is right first. Comments and testing welcome:

diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
index 8e80d3e0b..44616269f 100644
--- a/kernel/drivers/net/stack/ipv4/udp/udp.c
+++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
@@ -538,6 +538,7 @@ struct udpfakehdr
     struct iovec *iov;
     int iovlen;
     u32 wcheck;
+    size_t rawlen;
 };
 
 
@@ -549,24 +550,42 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
                           unsigned int offset, unsigned int fraglen)
 {
     struct udpfakehdr *ufh = (struct udpfakehdr *)p;
-    int i, ret;
-
+    void *ckdata, *tmp = NULL;
+    size_t cklen, len, iovsz;
+    struct iovec *tmpiov;
+    int ret;
 
     // We should optimize this function a bit (copy+csum...)!
     if (offset)
-	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
-
-    /* Checksum of the complete data part of the UDP message: */
-    for (i = 0; i < ufh->iovlen; i++) {
-            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
-                                       ufh->wcheck);
+	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
+				       to, fraglen);
+
+    len = fraglen - sizeof(struct udphdr);
+    if (len < ufh->rawlen) {
+	    iovsz = sizeof(*tmpiov) * ufh->iovlen;
+	    tmp = xnmalloc(ufh->rawlen + iovsz);
+	    if (tmp == NULL)
+		    return -ENOMEM;
+	    tmpiov = tmp;
+	    memcpy(tmpiov, ufh->iov, iovsz);
+	    ckdata = tmp + iovsz;
+	    cklen = ufh->rawlen;
+	    ret = rtnet_read_from_iov(ufh->fd, tmpiov, ufh->iovlen,
+				      ckdata, cklen);
+	    if (ret)
+		    goto out;
+	    memcpy(to + sizeof(struct udphdr), ckdata, len);
+    } else {
+	    ckdata = to + sizeof(struct udphdr);
+	    cklen = len;
+	    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
+				      ckdata, cklen);
+	    if (ret)
+		    goto out;
     }
 
-    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
-			      to + sizeof(struct udphdr),
-			      fraglen - sizeof(struct udphdr));
-    if (ret)
-	    return ret;
+    /* Checksum of the complete data part of the UDP message: */
+    ufh->wcheck = csum_partial(ckdata, cklen, ufh->wcheck);
 
     /* Checksum of the udp header: */
     ufh->wcheck = csum_partial((unsigned char *)ufh,
@@ -580,7 +599,11 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
 
     memcpy(to, ufh, sizeof(struct udphdr));
 
-    return 0;
+out:
+    if (tmp)
+	    xnfree(tmp);
+
+    return ret;
 }
 
 
@@ -682,9 +705,10 @@ ssize_t rt_udp_sendmsg(struct rtdm_fd *fd, const struct user_msghdr *msg, int ms
     ufh.uh.len    = htons(ulen);
     ufh.uh.check  = 0;
     ufh.fd        = fd;
-    ufh.iov       = msg->msg_iov;
+    ufh.iov       = iov;
     ufh.iovlen    = msg->msg_iovlen;
     ufh.wcheck    = 0;
+    ufh.rawlen    = len;
 
     err = rt_ip_build_xmit(sock, rt_udp_getfrag, &ufh, ulen, &rt, msg_flags);


-- 
Philippe.

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 04.11.18 17:17, Philippe Gerum wrote:
> On 6/21/18 4:57 PM, Jan Kiszka wrote:
>> On 2018-06-21 15:41, Jan Kiszka wrote:
>>> On 2018-06-21 13:55, Jan Kiszka wrote:
>>>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>>>> Dear Jan, Greg,
>>>>>
>>>>> Is there any pointer about this issue?
>>>>> This is blocking my next work..
>>>>
>>>> Does this solve the issue AND still generate valid UDP checksums (please
>>>> check with wireshark or against a normal networking stack?
>>>>
>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> index 8e80d3e0b..bb0b0fc12 100644
>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>      if (offset)
>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>  
>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>> -                                       ufh->wcheck);
>>>> -    }
>>>> -
>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>  			      to + sizeof(struct udphdr),
>>>>  			      fraglen - sizeof(struct udphdr));
>>>>      if (ret)
>>>>  	    return ret;
>>>>  
>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>>>> +			       fraglen - sizeof(struct udphdr),
>>>> +			       ufh->wcheck);
>>>> +
>>>>      /* Checksum of the udp header: */
>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>
>>>
>>> This was definitely wrong. Here is another try:
>>>
>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> index 8e80d3e0b..6cf1d369e 100644
>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>                            unsigned int offset, unsigned int fraglen)
>>>  {
>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>> +    unsigned int datalen = 0;
>>>      int i, ret;
>>>  
>>>  
>>> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>      if (offset)
>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>  
>>> -    /* Checksum of the complete data part of the UDP message: */
>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>> -                                       ufh->wcheck);
>>> -    }
>>> -
>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>  			      to + sizeof(struct udphdr),
>>>  			      fraglen - sizeof(struct udphdr));
>>>      if (ret)
>>>  	    return ret;
>>>  
>>> +    /* Checksum of the complete data part of the UDP message: */
>>> +    for (i = 0; i < ufh->iovlen; i++)
>>> +	    datalen += ufh->iov[i].iov_len;
>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
>>> +			       ufh->wcheck);
>>> +
>>>      /* Checksum of the udp header: */
>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>
>>
>> OK, this won't work either if fraglen < datalen. We have to rework this
>> more fundamentally.
>>
>> As a workaround, run the kernel with nosmap.
>>
>> Jan
>>
> 
> There is no point in debugging the splash with UDP until this one is addressed since running with nosmap won't be acceptable in many cases, so I took a stab at fixing this issue first. This fix is somewhat ugly, but a clean solution would involve reworking the entire handling of fragmented packets, so let's see if the logic is right first. Comments and testing welcome:
> 
> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
> index 8e80d3e0b..44616269f 100644
> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
> @@ -538,6 +538,7 @@ struct udpfakehdr
>      struct iovec *iov;
>      int iovlen;
>      u32 wcheck;
> +    size_t rawlen;
>  };
>  
>  
> @@ -549,24 +550,42 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>                            unsigned int offset, unsigned int fraglen)
>  {
>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
> -    int i, ret;
> -
> +    void *ckdata, *tmp = NULL;
> +    size_t cklen, len, iovsz;
> +    struct iovec *tmpiov;
> +    int ret;
>  
>      // We should optimize this function a bit (copy+csum...)!
>      if (offset)
> -	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
> -
> -    /* Checksum of the complete data part of the UDP message: */
> -    for (i = 0; i < ufh->iovlen; i++) {
> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
> -                                       ufh->wcheck);
> +	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> +				       to, fraglen);
> +
> +    len = fraglen - sizeof(struct udphdr);
> +    if (len < ufh->rawlen) {
> +	    iovsz = sizeof(*tmpiov) * ufh->iovlen;
> +	    tmp = xnmalloc(ufh->rawlen + iovsz);
> +	    if (tmp == NULL)
> +		    return -ENOMEM;
> +	    tmpiov = tmp;
> +	    memcpy(tmpiov, ufh->iov, iovsz);
> +	    ckdata = tmp + iovsz;
> +	    cklen = ufh->rawlen;
> +	    ret = rtnet_read_from_iov(ufh->fd, tmpiov, ufh->iovlen,
> +				      ckdata, cklen);
> +	    if (ret)
> +		    goto out;
> +	    memcpy(to + sizeof(struct udphdr), ckdata, len);
> +    } else {
> +	    ckdata = to + sizeof(struct udphdr);
> +	    cklen = len;
> +	    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> +				      ckdata, cklen);
> +	    if (ret)
> +		    goto out;
>      }
>  
> -    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
> -			      to + sizeof(struct udphdr),
> -			      fraglen - sizeof(struct udphdr));
> -    if (ret)
> -	    return ret;
> +    /* Checksum of the complete data part of the UDP message: */
> +    ufh->wcheck = csum_partial(ckdata, cklen, ufh->wcheck);
>  
>      /* Checksum of the udp header: */
>      ufh->wcheck = csum_partial((unsigned char *)ufh,
> @@ -580,7 +599,11 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>  
>      memcpy(to, ufh, sizeof(struct udphdr));
>  
> -    return 0;
> +out:
> +    if (tmp)
> +	    xnfree(tmp);
> +
> +    return ret;
>  }
>  
>  
> @@ -682,9 +705,10 @@ ssize_t rt_udp_sendmsg(struct rtdm_fd *fd, const struct user_msghdr *msg, int ms
>      ufh.uh.len    = htons(ulen);
>      ufh.uh.check  = 0;
>      ufh.fd        = fd;
> -    ufh.iov       = msg->msg_iov;
> +    ufh.iov       = iov;
>      ufh.iovlen    = msg->msg_iovlen;
>      ufh.wcheck    = 0;
> +    ufh.rawlen    = len;
>  
>      err = rt_ip_build_xmit(sock, rt_udp_getfrag, &ufh, ulen, &rt, msg_flags);
> 
> 

Nasty, very nasty. That dynamic allocation could easily cause a performance
regression for existing users.

For my memory: The smap-trigger is doing csum_partial on a userspace buffer,
right? So we would need a user-safe csum_partial. I think the kernel does that...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 11/6/18 7:57 PM, Jan Kiszka wrote:
> On 04.11.18 17:17, Philippe Gerum wrote:
>> On 6/21/18 4:57 PM, Jan Kiszka wrote:
>>> On 2018-06-21 15:41, Jan Kiszka wrote:
>>>> On 2018-06-21 13:55, Jan Kiszka wrote:
>>>>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>>>>> Dear Jan, Greg,
>>>>>>
>>>>>> Is there any pointer about this issue?
>>>>>> This is blocking my next work..
>>>>>
>>>>> Does this solve the issue AND still generate valid UDP checksums (please
>>>>> check with wireshark or against a normal networking stack?
>>>>>
>>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> index 8e80d3e0b..bb0b0fc12 100644
>>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>      if (offset)
>>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>>  
>>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>>> -                                       ufh->wcheck);
>>>>> -    }
>>>>> -
>>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>>  			      to + sizeof(struct udphdr),
>>>>>  			      fraglen - sizeof(struct udphdr));
>>>>>      if (ret)
>>>>>  	    return ret;
>>>>>  
>>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>>>>> +			       fraglen - sizeof(struct udphdr),
>>>>> +			       ufh->wcheck);
>>>>> +
>>>>>      /* Checksum of the udp header: */
>>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>>
>>>>
>>>> This was definitely wrong. Here is another try:
>>>>
>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> index 8e80d3e0b..6cf1d369e 100644
>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>                            unsigned int offset, unsigned int fraglen)
>>>>  {
>>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>>> +    unsigned int datalen = 0;
>>>>      int i, ret;
>>>>  
>>>>  
>>>> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>      if (offset)
>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>  
>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>> -                                       ufh->wcheck);
>>>> -    }
>>>> -
>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>  			      to + sizeof(struct udphdr),
>>>>  			      fraglen - sizeof(struct udphdr));
>>>>      if (ret)
>>>>  	    return ret;
>>>>  
>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>> +    for (i = 0; i < ufh->iovlen; i++)
>>>> +	    datalen += ufh->iov[i].iov_len;
>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
>>>> +			       ufh->wcheck);
>>>> +
>>>>      /* Checksum of the udp header: */
>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>
>>>
>>> OK, this won't work either if fraglen < datalen. We have to rework this
>>> more fundamentally.
>>>
>>> As a workaround, run the kernel with nosmap.
>>>
>>> Jan
>>>
>>
>> There is no point in debugging the splash with UDP until this one is addressed since running with nosmap won't be acceptable in many cases, so I took a stab at fixing this issue first. This fix is somewhat ugly, but a clean solution would involve reworking the entire handling of fragmented packets, so let's see if the logic is right first. Comments and testing welcome:
>>
>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> index 8e80d3e0b..44616269f 100644
>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>> @@ -538,6 +538,7 @@ struct udpfakehdr
>>      struct iovec *iov;
>>      int iovlen;
>>      u32 wcheck;
>> +    size_t rawlen;
>>  };
>>  
>>  
>> @@ -549,24 +550,42 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>                            unsigned int offset, unsigned int fraglen)
>>  {
>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>> -    int i, ret;
>> -
>> +    void *ckdata, *tmp = NULL;
>> +    size_t cklen, len, iovsz;
>> +    struct iovec *tmpiov;
>> +    int ret;
>>  
>>      // We should optimize this function a bit (copy+csum...)!
>>      if (offset)
>> -	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>> -
>> -    /* Checksum of the complete data part of the UDP message: */
>> -    for (i = 0; i < ufh->iovlen; i++) {
>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>> -                                       ufh->wcheck);
>> +	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>> +				       to, fraglen);
>> +
>> +    len = fraglen - sizeof(struct udphdr);
>> +    if (len < ufh->rawlen) {
>> +	    iovsz = sizeof(*tmpiov) * ufh->iovlen;
>> +	    tmp = xnmalloc(ufh->rawlen + iovsz);
>> +	    if (tmp == NULL)
>> +		    return -ENOMEM;
>> +	    tmpiov = tmp;
>> +	    memcpy(tmpiov, ufh->iov, iovsz);
>> +	    ckdata = tmp + iovsz;
>> +	    cklen = ufh->rawlen;
>> +	    ret = rtnet_read_from_iov(ufh->fd, tmpiov, ufh->iovlen,
>> +				      ckdata, cklen);
>> +	    if (ret)
>> +		    goto out;
>> +	    memcpy(to + sizeof(struct udphdr), ckdata, len);
>> +    } else {
>> +	    ckdata = to + sizeof(struct udphdr);
>> +	    cklen = len;
>> +	    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>> +				      ckdata, cklen);
>> +	    if (ret)
>> +		    goto out;
>>      }
>>  
>> -    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>> -			      to + sizeof(struct udphdr),
>> -			      fraglen - sizeof(struct udphdr));
>> -    if (ret)
>> -	    return ret;
>> +    /* Checksum of the complete data part of the UDP message: */
>> +    ufh->wcheck = csum_partial(ckdata, cklen, ufh->wcheck);
>>  
>>      /* Checksum of the udp header: */
>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>> @@ -580,7 +599,11 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>  
>>      memcpy(to, ufh, sizeof(struct udphdr));
>>  
>> -    return 0;
>> +out:
>> +    if (tmp)
>> +	    xnfree(tmp);
>> +
>> +    return ret;
>>  }
>>  
>>  
>> @@ -682,9 +705,10 @@ ssize_t rt_udp_sendmsg(struct rtdm_fd *fd, const struct user_msghdr *msg, int ms
>>      ufh.uh.len    = htons(ulen);
>>      ufh.uh.check  = 0;
>>      ufh.fd        = fd;
>> -    ufh.iov       = msg->msg_iov;
>> +    ufh.iov       = iov;
>>      ufh.iovlen    = msg->msg_iovlen;
>>      ufh.wcheck    = 0;
>> +    ufh.rawlen    = len;
>>  
>>      err = rt_ip_build_xmit(sock, rt_udp_getfrag, &ufh, ulen, &rt, msg_flags);
>>
>>
> 
> Nasty, very nasty. That dynamic allocation could easily cause a performance
> regression for existing users.
> 

Yes, this is my version of ugliness.

> For my memory: The smap-trigger is doing csum_partial on a userspace buffer,
> right?

Right, dereferencing the iov_base pointer(s).

 So we would need a user-safe csum_partial. I think the kernel does that...
> 

Would be nice. How would that cope with rt_ip_build_xmit_slow() in case
the packet size exceeds the MTU? AFAICT, the header and payload data are
sent over the wire on the fly as they are collected by the getfrag()
callback.

-- 
Philippe.

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 07.11.18 11:04, Philippe Gerum wrote:
> On 11/6/18 7:57 PM, Jan Kiszka wrote:
>> On 04.11.18 17:17, Philippe Gerum wrote:
>>> On 6/21/18 4:57 PM, Jan Kiszka wrote:
>>>> On 2018-06-21 15:41, Jan Kiszka wrote:
>>>>> On 2018-06-21 13:55, Jan Kiszka wrote:
>>>>>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>>>>>> Dear Jan, Greg,
>>>>>>>
>>>>>>> Is there any pointer about this issue?
>>>>>>> This is blocking my next work..
>>>>>>
>>>>>> Does this solve the issue AND still generate valid UDP checksums (please
>>>>>> check with wireshark or against a normal networking stack?
>>>>>>
>>>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> index 8e80d3e0b..bb0b0fc12 100644
>>>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>>      if (offset)
>>>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>>>  
>>>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>>>> -                                       ufh->wcheck);
>>>>>> -    }
>>>>>> -
>>>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>>>  			      to + sizeof(struct udphdr),
>>>>>>  			      fraglen - sizeof(struct udphdr));
>>>>>>      if (ret)
>>>>>>  	    return ret;
>>>>>>  
>>>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>>>>>> +			       fraglen - sizeof(struct udphdr),
>>>>>> +			       ufh->wcheck);
>>>>>> +
>>>>>>      /* Checksum of the udp header: */
>>>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>>>
>>>>>
>>>>> This was definitely wrong. Here is another try:
>>>>>
>>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> index 8e80d3e0b..6cf1d369e 100644
>>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>                            unsigned int offset, unsigned int fraglen)
>>>>>  {
>>>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>>>> +    unsigned int datalen = 0;
>>>>>      int i, ret;
>>>>>  
>>>>>  
>>>>> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>      if (offset)
>>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>>  
>>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>>> -                                       ufh->wcheck);
>>>>> -    }
>>>>> -
>>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>>  			      to + sizeof(struct udphdr),
>>>>>  			      fraglen - sizeof(struct udphdr));
>>>>>      if (ret)
>>>>>  	    return ret;
>>>>>  
>>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>>> +    for (i = 0; i < ufh->iovlen; i++)
>>>>> +	    datalen += ufh->iov[i].iov_len;
>>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
>>>>> +			       ufh->wcheck);
>>>>> +
>>>>>      /* Checksum of the udp header: */
>>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>>
>>>>
>>>> OK, this won't work either if fraglen < datalen. We have to rework this
>>>> more fundamentally.
>>>>
>>>> As a workaround, run the kernel with nosmap.
>>>>
>>>> Jan
>>>>
>>>
>>> There is no point in debugging the splash with UDP until this one is addressed since running with nosmap won't be acceptable in many cases, so I took a stab at fixing this issue first. This fix is somewhat ugly, but a clean solution would involve reworking the entire handling of fragmented packets, so let's see if the logic is right first. Comments and testing welcome:
>>>
>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> index 8e80d3e0b..44616269f 100644
>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>> @@ -538,6 +538,7 @@ struct udpfakehdr
>>>      struct iovec *iov;
>>>      int iovlen;
>>>      u32 wcheck;
>>> +    size_t rawlen;
>>>  };
>>>  
>>>  
>>> @@ -549,24 +550,42 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>                            unsigned int offset, unsigned int fraglen)
>>>  {
>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>> -    int i, ret;
>>> -
>>> +    void *ckdata, *tmp = NULL;
>>> +    size_t cklen, len, iovsz;
>>> +    struct iovec *tmpiov;
>>> +    int ret;
>>>  
>>>      // We should optimize this function a bit (copy+csum...)!
>>>      if (offset)
>>> -	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>> -
>>> -    /* Checksum of the complete data part of the UDP message: */
>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>> -                                       ufh->wcheck);
>>> +	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>> +				       to, fraglen);
>>> +
>>> +    len = fraglen - sizeof(struct udphdr);
>>> +    if (len < ufh->rawlen) {
>>> +	    iovsz = sizeof(*tmpiov) * ufh->iovlen;
>>> +	    tmp = xnmalloc(ufh->rawlen + iovsz);
>>> +	    if (tmp == NULL)
>>> +		    return -ENOMEM;
>>> +	    tmpiov = tmp;
>>> +	    memcpy(tmpiov, ufh->iov, iovsz);
>>> +	    ckdata = tmp + iovsz;
>>> +	    cklen = ufh->rawlen;
>>> +	    ret = rtnet_read_from_iov(ufh->fd, tmpiov, ufh->iovlen,
>>> +				      ckdata, cklen);
>>> +	    if (ret)
>>> +		    goto out;
>>> +	    memcpy(to + sizeof(struct udphdr), ckdata, len);
>>> +    } else {
>>> +	    ckdata = to + sizeof(struct udphdr);
>>> +	    cklen = len;
>>> +	    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>> +				      ckdata, cklen);
>>> +	    if (ret)
>>> +		    goto out;
>>>      }
>>>  
>>> -    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>> -			      to + sizeof(struct udphdr),
>>> -			      fraglen - sizeof(struct udphdr));
>>> -    if (ret)
>>> -	    return ret;
>>> +    /* Checksum of the complete data part of the UDP message: */
>>> +    ufh->wcheck = csum_partial(ckdata, cklen, ufh->wcheck);
>>>  
>>>      /* Checksum of the udp header: */
>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>> @@ -580,7 +599,11 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>  
>>>      memcpy(to, ufh, sizeof(struct udphdr));
>>>  
>>> -    return 0;
>>> +out:
>>> +    if (tmp)
>>> +	    xnfree(tmp);
>>> +
>>> +    return ret;
>>>  }
>>>  
>>>  
>>> @@ -682,9 +705,10 @@ ssize_t rt_udp_sendmsg(struct rtdm_fd *fd, const struct user_msghdr *msg, int ms
>>>      ufh.uh.len    = htons(ulen);
>>>      ufh.uh.check  = 0;
>>>      ufh.fd        = fd;
>>> -    ufh.iov       = msg->msg_iov;
>>> +    ufh.iov       = iov;
>>>      ufh.iovlen    = msg->msg_iovlen;
>>>      ufh.wcheck    = 0;
>>> +    ufh.rawlen    = len;
>>>  
>>>      err = rt_ip_build_xmit(sock, rt_udp_getfrag, &ufh, ulen, &rt, msg_flags);
>>>
>>>
>>
>> Nasty, very nasty. That dynamic allocation could easily cause a performance
>> regression for existing users.
>>
> 
> Yes, this is my version of ugliness.
> 
>> For my memory: The smap-trigger is doing csum_partial on a userspace buffer,
>> right?
> 
> Right, dereferencing the iov_base pointer(s).
> 
>  So we would need a user-safe csum_partial. I think the kernel does that...
>>
> 
> Would be nice. How would that cope with rt_ip_build_xmit_slow() in case
> the packet size exceeds the MTU? AFAICT, the header and payload data are
> sent over the wire on the fly as they are collected by the getfrag()
> callback.

What exactly is the problem there? Sorry, this code structure is no longer in my
cache.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 11/7/18 11:19 AM, Jan Kiszka wrote:
> On 07.11.18 11:04, Philippe Gerum wrote:
>> On 11/6/18 7:57 PM, Jan Kiszka wrote:
>>> On 04.11.18 17:17, Philippe Gerum wrote:
>>>> On 6/21/18 4:57 PM, Jan Kiszka wrote:
>>>>> On 2018-06-21 15:41, Jan Kiszka wrote:
>>>>>> On 2018-06-21 13:55, Jan Kiszka wrote:
>>>>>>> On 2018-06-21 13:20, Pintu Kumar wrote:
>>>>>>>> Dear Jan, Greg,
>>>>>>>>
>>>>>>>> Is there any pointer about this issue?
>>>>>>>> This is blocking my next work..
>>>>>>>
>>>>>>> Does this solve the issue AND still generate valid UDP checksums (please
>>>>>>> check with wireshark or against a normal networking stack?
>>>>>>>
>>>>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>>> index 8e80d3e0b..bb0b0fc12 100644
>>>>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>>> @@ -556,18 +556,17 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>>>      if (offset)
>>>>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>>>>  
>>>>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>>>>> -                                       ufh->wcheck);
>>>>>>> -    }
>>>>>>> -
>>>>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>>>>  			      to + sizeof(struct udphdr),
>>>>>>>  			      fraglen - sizeof(struct udphdr));
>>>>>>>      if (ret)
>>>>>>>  	    return ret;
>>>>>>>  
>>>>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr),
>>>>>>> +			       fraglen - sizeof(struct udphdr),
>>>>>>> +			       ufh->wcheck);
>>>>>>> +
>>>>>>>      /* Checksum of the udp header: */
>>>>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>>>>
>>>>>>
>>>>>> This was definitely wrong. Here is another try:
>>>>>>
>>>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> index 8e80d3e0b..6cf1d369e 100644
>>>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>>>> @@ -549,6 +549,7 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>>                            unsigned int offset, unsigned int fraglen)
>>>>>>  {
>>>>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>>>>> +    unsigned int datalen = 0;
>>>>>>      int i, ret;
>>>>>>  
>>>>>>  
>>>>>> @@ -556,18 +557,18 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>>>      if (offset)
>>>>>>  	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>>>>  
>>>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>>>> -                                       ufh->wcheck);
>>>>>> -    }
>>>>>> -
>>>>>>      ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>>>>  			      to + sizeof(struct udphdr),
>>>>>>  			      fraglen - sizeof(struct udphdr));
>>>>>>      if (ret)
>>>>>>  	    return ret;
>>>>>>  
>>>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>>>> +    for (i = 0; i < ufh->iovlen; i++)
>>>>>> +	    datalen += ufh->iov[i].iov_len;
>>>>>> +    ufh->wcheck = csum_partial(to + sizeof(struct udphdr), datalen,
>>>>>> +			       ufh->wcheck);
>>>>>> +
>>>>>>      /* Checksum of the udp header: */
>>>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>>>>  			       sizeof(struct udphdr), ufh->wcheck);
>>>>>>
>>>>>
>>>>> OK, this won't work either if fraglen < datalen. We have to rework this
>>>>> more fundamentally.
>>>>>
>>>>> As a workaround, run the kernel with nosmap.
>>>>>
>>>>> Jan
>>>>>
>>>>
>>>> There is no point in debugging the splash with UDP until this one is addressed since running with nosmap won't be acceptable in many cases, so I took a stab at fixing this issue first. This fix is somewhat ugly, but a clean solution would involve reworking the entire handling of fragmented packets, so let's see if the logic is right first. Comments and testing welcome:
>>>>
>>>> diff --git a/kernel/drivers/net/stack/ipv4/udp/udp.c b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> index 8e80d3e0b..44616269f 100644
>>>> --- a/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> +++ b/kernel/drivers/net/stack/ipv4/udp/udp.c
>>>> @@ -538,6 +538,7 @@ struct udpfakehdr
>>>>      struct iovec *iov;
>>>>      int iovlen;
>>>>      u32 wcheck;
>>>> +    size_t rawlen;
>>>>  };
>>>>  
>>>>  
>>>> @@ -549,24 +550,42 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>                            unsigned int offset, unsigned int fraglen)
>>>>  {
>>>>      struct udpfakehdr *ufh = (struct udpfakehdr *)p;
>>>> -    int i, ret;
>>>> -
>>>> +    void *ckdata, *tmp = NULL;
>>>> +    size_t cklen, len, iovsz;
>>>> +    struct iovec *tmpiov;
>>>> +    int ret;
>>>>  
>>>>      // We should optimize this function a bit (copy+csum...)!
>>>>      if (offset)
>>>> -	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen, to, fraglen);
>>>> -
>>>> -    /* Checksum of the complete data part of the UDP message: */
>>>> -    for (i = 0; i < ufh->iovlen; i++) {
>>>> -            ufh->wcheck = csum_partial(ufh->iov[i].iov_base, ufh->iov[i].iov_len,
>>>> -                                       ufh->wcheck);
>>>> +	    return rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>> +				       to, fraglen);
>>>> +
>>>> +    len = fraglen - sizeof(struct udphdr);
>>>> +    if (len < ufh->rawlen) {
>>>> +	    iovsz = sizeof(*tmpiov) * ufh->iovlen;
>>>> +	    tmp = xnmalloc(ufh->rawlen + iovsz);
>>>> +	    if (tmp == NULL)
>>>> +		    return -ENOMEM;
>>>> +	    tmpiov = tmp;
>>>> +	    memcpy(tmpiov, ufh->iov, iovsz);
>>>> +	    ckdata = tmp + iovsz;
>>>> +	    cklen = ufh->rawlen;
>>>> +	    ret = rtnet_read_from_iov(ufh->fd, tmpiov, ufh->iovlen,
>>>> +				      ckdata, cklen);
>>>> +	    if (ret)
>>>> +		    goto out;
>>>> +	    memcpy(to + sizeof(struct udphdr), ckdata, len);
>>>> +    } else {
>>>> +	    ckdata = to + sizeof(struct udphdr);
>>>> +	    cklen = len;
>>>> +	    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>> +				      ckdata, cklen);
>>>> +	    if (ret)
>>>> +		    goto out;
>>>>      }
>>>>  
>>>> -    ret = rtnet_read_from_iov(ufh->fd, ufh->iov, ufh->iovlen,
>>>> -			      to + sizeof(struct udphdr),
>>>> -			      fraglen - sizeof(struct udphdr));
>>>> -    if (ret)
>>>> -	    return ret;
>>>> +    /* Checksum of the complete data part of the UDP message: */
>>>> +    ufh->wcheck = csum_partial(ckdata, cklen, ufh->wcheck);
>>>>  
>>>>      /* Checksum of the udp header: */
>>>>      ufh->wcheck = csum_partial((unsigned char *)ufh,
>>>> @@ -580,7 +599,11 @@ static int rt_udp_getfrag(const void *p, unsigned char *to,
>>>>  
>>>>      memcpy(to, ufh, sizeof(struct udphdr));
>>>>  
>>>> -    return 0;
>>>> +out:
>>>> +    if (tmp)
>>>> +	    xnfree(tmp);
>>>> +
>>>> +    return ret;
>>>>  }
>>>>  
>>>>  
>>>> @@ -682,9 +705,10 @@ ssize_t rt_udp_sendmsg(struct rtdm_fd *fd, const struct user_msghdr *msg, int ms
>>>>      ufh.uh.len    = htons(ulen);
>>>>      ufh.uh.check  = 0;
>>>>      ufh.fd        = fd;
>>>> -    ufh.iov       = msg->msg_iov;
>>>> +    ufh.iov       = iov;
>>>>      ufh.iovlen    = msg->msg_iovlen;
>>>>      ufh.wcheck    = 0;
>>>> +    ufh.rawlen    = len;
>>>>  
>>>>      err = rt_ip_build_xmit(sock, rt_udp_getfrag, &ufh, ulen, &rt, msg_flags);
>>>>
>>>>
>>>
>>> Nasty, very nasty. That dynamic allocation could easily cause a performance
>>> regression for existing users.
>>>
>>
>> Yes, this is my version of ugliness.
>>
>>> For my memory: The smap-trigger is doing csum_partial on a userspace buffer,
>>> right?
>>
>> Right, dereferencing the iov_base pointer(s).
>>
>>  So we would need a user-safe csum_partial. I think the kernel does that...
>>>
>>
>> Would be nice. How would that cope with rt_ip_build_xmit_slow() in case
>> the packet size exceeds the MTU? AFAICT, the header and payload data are
>> sent over the wire on the fly as they are collected by the getfrag()
>> callback.
> 
> What exactly is the problem there? Sorry, this code structure is no longer in my
> cache.
> 

The underlying question was: don't we need the checksum value early in
the process of scanning all frags so that we can send it over the wire
into the first packet as part of the header?

If so, the callback mechanism which collects one fragment then
hard-xmits that data at each iteration does not seem to fit this
requirement. Unless we are ok loading the payload twice: once for
building the checksum, twice for sending the data the way it is done now.

-- 
Philippe.

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Jan Kiszka]

On 07.11.18 11:31, Philippe Gerum wrote:
> On 11/7/18 11:19 AM, Jan Kiszka wrote:
>> On 07.11.18 11:04, Philippe Gerum wrote:
>>> Would be nice. How would that cope with rt_ip_build_xmit_slow() in case
>>> the packet size exceeds the MTU? AFAICT, the header and payload data are
>>> sent over the wire on the fly as they are collected by the getfrag()
>>> callback.
>>
>> What exactly is the problem there? Sorry, this code structure is no longer in my
>> cache.
>>
> 
> The underlying question was: don't we need the checksum value early in
> the process of scanning all frags so that we can send it over the wire
> into the first packet as part of the header?
> 
> If so, the callback mechanism which collects one fragment then
> hard-xmits that data at each iteration does not seem to fit this
> requirement. Unless we are ok loading the payload twice: once for
> building the checksum, twice for sending the data the way it is done now.
> 

IIRC, there is no need to have a checksum over the whole data in the first
fragment, only a per-fragment (i.e. per-packet) checksum.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Kernel oops during rtnet loopback usage on x86_64 (e1000e)

[Philippe Gerum]

On 11/7/18 12:19 PM, Jan Kiszka wrote:
> On 07.11.18 11:31, Philippe Gerum wrote:
>> On 11/7/18 11:19 AM, Jan Kiszka wrote:
>>> On 07.11.18 11:04, Philippe Gerum wrote:
>>>> Would be nice. How would that cope with rt_ip_build_xmit_slow() in case
>>>> the packet size exceeds the MTU? AFAICT, the header and payload data are
>>>> sent over the wire on the fly as they are collected by the getfrag()
>>>> callback.
>>>
>>> What exactly is the problem there? Sorry, this code structure is no longer in my
>>> cache.
>>>
>>
>> The underlying question was: don't we need the checksum value early in
>> the process of scanning all frags so that we can send it over the wire
>> into the first packet as part of the header?
>>
>> If so, the callback mechanism which collects one fragment then
>> hard-xmits that data at each iteration does not seem to fit this
>> requirement. Unless we are ok loading the payload twice: once for
>> building the checksum, twice for sending the data the way it is done now.
>>
> 
> IIRC, there is no need to have a checksum over the whole data in the first
> fragment, only a per-fragment (i.e. per-packet) checksum.
> 

Ok, so csum_partial_copy_from_user() would likely do.

-- 
Philippe.