* Re: lttng create call trace on kernel 4.19
[not found] <eee78bd1-4d5c-63e7-dc74-ff5a51426666@windriver.com>
@ 2018-10-11 13:22 ` Jonathan Rajotte-Julien
[not found] ` <20181011132253.GA13655@joraj-alpa>
1 sibling, 0 replies; 9+ messages in thread
From: Jonathan Rajotte-Julien @ 2018-10-11 13:22 UTC (permalink / raw)
To: Dengke Du; +Cc: lttng-dev
Hi,
On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
> Hi all
>
> When I create a local trace session, meeting a call trace:
>
> uname -a:
>
> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
Which version of lttng-modules are you using?
Any modification to the kernel?
>
> lttng create trace_session --live -U net://127.0.0.1
>
> ----------------------------------------------------------------------------------------------------------
>
> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
> [ 223.550725] BUG: unable to handle kernel paging request at
> 0000111400001158
> [ 223.551661] PGD 0 P4D 0
> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
> 4.19.0-rc7-yoctodev-standard #1
> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> 0000111400001158
> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> ffffffffc02f1100
> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> ffff9b3ddec01600
> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> 0000000000000000
> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> 0000000000000002
> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> knlGS:0000000000000000
> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
> 00000000000006f0
> [ 223.566450] Call Trace:
> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
> [ 223.571309] ? 0xffffffffc0309000
> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
> [ 223.573820] ? 0xffffffffc0309000
> [ 223.574237] do_one_initcall+0x53/0x1fe
> [ 223.574682] ? __vunmap+0x8e/0xc0
> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
> [ 223.575684] ? do_init_module+0x27/0x210
> [ 223.576285] do_init_module+0x60/0x210
> [ 223.576702] load_module+0x2406/0x2570
> [ 223.577140] ? vfs_read+0x115/0x140
> [ 223.577611] ? vfs_read+0x115/0x140
> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
> [ 223.579627] do_syscall_64+0x4d/0xf0
> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 223.580837] RIP: 0033:0x7fb049107569
> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
> 3d 01 f0 ff ff 73 01 c3 48 8b 08
> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000139
> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
> 00007fb049107569
> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
> 000000000000000a
> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 000055931b680b70
> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
> 000055931b67734e
> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
> 0000000000000000
> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
> lttng_kretprobes(O) deflate twofish_geneb
> [ 223.597615] CR2: 0000111400001158
> [ 223.614673] ---[ end trace 514884c3611a01af ]---
> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> 0000111400001158
> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> ffffffffc02f1100
> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> ffff9b3ddec01600
> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> 0000000000000000
> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> 0000000000000002
> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> knlGS:0000000000000000
> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
> 00000000000006f0
> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
> paging request at 0000111400001158
> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
> at 0000111400001158
>
> ----------------------------------------------------------------------------------------------------------
>
> When I type that command again:
>
> ----------------------------------------------------------------------------------------------------------
>
> Spawning a session daemon
> Error: Session daemon terminated with an error (exit status: 1)
> Error: Problem occurred while launching session daemon
> (/usr/bin/lttng-sessiond)
> Error: Command error
>
> ----------------------------------------------------------------------------------------------------------
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <20181011132253.GA13655@joraj-alpa>
@ 2018-10-12 1:21 ` Dengke Du
[not found] ` <82561ba6-42f9-d816-0d33-f9906d0affcd@windriver.com>
1 sibling, 0 replies; 9+ messages in thread
From: Dengke Du @ 2018-10-12 1:21 UTC (permalink / raw)
To: Jonathan Rajotte-Julien; +Cc: lttng-dev
On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
> Hi,
>
> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
>> Hi all
>>
>> When I create a local trace session, meeting a call trace:
>>
>> uname -a:
>>
>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
>
> Which version of lttng-modules are you using?
>
> Any modification to the kernel?
Yocto: master branch
Kernel: linux-yocto-dev
phenomenon 1:
linux-yocto was ok (v4.18)
phenomenon 2:
linux-yocto-dev was not ok (v4.19)
Maybe some kernel commit breaks it.
>
>> lttng create trace_session --live -U net://127.0.0.1
>>
>> ----------------------------------------------------------------------------------------------------------
>>
>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
>> [ 223.550725] BUG: unable to handle kernel paging request at
>> 0000111400001158
>> [ 223.551661] PGD 0 P4D 0
>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
>> 4.19.0-rc7-yoctodev-standard #1
>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>> 0000111400001158
>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>> ffffffffc02f1100
>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>> ffff9b3ddec01600
>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>> 0000000000000000
>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>> 0000000000000002
>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>> knlGS:0000000000000000
>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
>> 00000000000006f0
>> [ 223.566450] Call Trace:
>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
>> [ 223.571309] ? 0xffffffffc0309000
>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
>> [ 223.573820] ? 0xffffffffc0309000
>> [ 223.574237] do_one_initcall+0x53/0x1fe
>> [ 223.574682] ? __vunmap+0x8e/0xc0
>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
>> [ 223.575684] ? do_init_module+0x27/0x210
>> [ 223.576285] do_init_module+0x60/0x210
>> [ 223.576702] load_module+0x2406/0x2570
>> [ 223.577140] ? vfs_read+0x115/0x140
>> [ 223.577611] ? vfs_read+0x115/0x140
>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
>> [ 223.579627] do_syscall_64+0x4d/0xf0
>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>> [ 223.580837] RIP: 0033:0x7fb049107569
>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
>> 0000000000000139
>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
>> 00007fb049107569
>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
>> 000000000000000a
>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
>> 000055931b680b70
>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
>> 000055931b67734e
>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
>> 0000000000000000
>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
>> lttng_kretprobes(O) deflate twofish_geneb
>> [ 223.597615] CR2: 0000111400001158
>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>> 0000111400001158
>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>> ffffffffc02f1100
>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>> ffff9b3ddec01600
>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>> 0000000000000000
>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>> 0000000000000002
>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>> knlGS:0000000000000000
>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
>> 00000000000006f0
>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
>> paging request at 0000111400001158
>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
>> at 0000111400001158
>>
>> ----------------------------------------------------------------------------------------------------------
>>
>> When I type that command again:
>>
>> ----------------------------------------------------------------------------------------------------------
>>
>> Spawning a session daemon
>> Error: Session daemon terminated with an error (exit status: 1)
>> Error: Problem occurred while launching session daemon
>> (/usr/bin/lttng-sessiond)
>> Error: Command error
>>
>> ----------------------------------------------------------------------------------------------------------
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <82561ba6-42f9-d816-0d33-f9906d0affcd@windriver.com>
@ 2018-10-12 15:10 ` Mathieu Desnoyers
[not found] ` <2081244858.3133.1539357009809.JavaMail.zimbra@efficios.com>
1 sibling, 0 replies; 9+ messages in thread
From: Mathieu Desnoyers @ 2018-10-12 15:10 UTC (permalink / raw)
To: Dengke Du; +Cc: lttng-dev
----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
> On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
>> Hi,
>>
>> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
>>> Hi all
>>>
>>> When I create a local trace session, meeting a call trace:
>>>
>>> uname -a:
>>>
>>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
>>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
>>
>> Which version of lttng-modules are you using?
Please answer the questions above as well.
Thanks,
Mathieu
>>
>> Any modification to the kernel?
>
> Yocto: master branch
> Kernel: linux-yocto-dev
>
> phenomenon 1:
> linux-yocto was ok (v4.18)
> phenomenon 2:
> linux-yocto-dev was not ok (v4.19)
>
> Maybe some kernel commit breaks it.
>
>>
>>> lttng create trace_session --live -U net://127.0.0.1
>>>
>>> ----------------------------------------------------------------------------------------------------------
>>>
>>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
>>> [ 223.550725] BUG: unable to handle kernel paging request at
>>> 0000111400001158
>>> [ 223.551661] PGD 0 P4D 0
>>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
>>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
>>> 4.19.0-rc7-yoctodev-standard #1
>>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
>>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>> 0000111400001158
>>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>> ffffffffc02f1100
>>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>> ffff9b3ddec01600
>>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>> 0000000000000000
>>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>> 0000000000000002
>>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>> knlGS:0000000000000000
>>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
>>> 00000000000006f0
>>> [ 223.566450] Call Trace:
>>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
>>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
>>> [ 223.571309] ? 0xffffffffc0309000
>>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
>>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
>>> [ 223.573820] ? 0xffffffffc0309000
>>> [ 223.574237] do_one_initcall+0x53/0x1fe
>>> [ 223.574682] ? __vunmap+0x8e/0xc0
>>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
>>> [ 223.575684] ? do_init_module+0x27/0x210
>>> [ 223.576285] do_init_module+0x60/0x210
>>> [ 223.576702] load_module+0x2406/0x2570
>>> [ 223.577140] ? vfs_read+0x115/0x140
>>> [ 223.577611] ? vfs_read+0x115/0x140
>>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
>>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
>>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
>>> [ 223.579627] do_syscall_64+0x4d/0xf0
>>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 223.580837] RIP: 0033:0x7fb049107569
>>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
>>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
>>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
>>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
>>> 0000000000000139
>>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
>>> 00007fb049107569
>>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
>>> 000000000000000a
>>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
>>> 000055931b680b70
>>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
>>> 000055931b67734e
>>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
>>> 0000000000000000
>>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
>>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
>>> lttng_kretprobes(O) deflate twofish_geneb
>>> [ 223.597615] CR2: 0000111400001158
>>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
>>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>> 0000111400001158
>>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>> ffffffffc02f1100
>>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>> ffff9b3ddec01600
>>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>> 0000000000000000
>>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>> 0000000000000002
>>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>> knlGS:0000000000000000
>>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
>>> 00000000000006f0
>>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
>>> paging request at 0000111400001158
>>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
>>> at 0000111400001158
>>>
>>> ----------------------------------------------------------------------------------------------------------
>>>
>>> When I type that command again:
>>>
>>> ----------------------------------------------------------------------------------------------------------
>>>
>>> Spawning a session daemon
>>> Error: Session daemon terminated with an error (exit status: 1)
>>> Error: Problem occurred while launching session daemon
>>> (/usr/bin/lttng-sessiond)
>>> Error: Command error
>>>
>>> ----------------------------------------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <2081244858.3133.1539357009809.JavaMail.zimbra@efficios.com>
@ 2018-10-12 16:21 ` Jonathan Rajotte-Julien
2018-10-15 2:04 ` Dengke Du
[not found] ` <20181012162118.GB13655@joraj-alpa>
2 siblings, 0 replies; 9+ messages in thread
From: Jonathan Rajotte-Julien @ 2018-10-12 16:21 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: lttng-dev
Hi Dengke,
We were able to reproduce it. This is not related to yocto and seems to be
related to a change introduced in the 4.19 kernel.
If you are not interested in kernel tracing you can pass the --no-kernel flag to
lttng-sessiond to prevent the loading of the kernel modules.
We are looking into the root problem and possible fix on either the kernel side
or ours.
Thanks for reporting the issue. We will keep you updated.
Cheers
On Fri, Oct 12, 2018 at 11:10:09AM -0400, Mathieu Desnoyers wrote:
> ----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
>
> > On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
> >> Hi,
> >>
> >> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
> >>> Hi all
> >>>
> >>> When I create a local trace session, meeting a call trace:
> >>>
> >>> uname -a:
> >>>
> >>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
> >>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
> >>
> >> Which version of lttng-modules are you using?
>
> Please answer the questions above as well.
>
> Thanks,
>
> Mathieu
>
>
> >>
> >> Any modification to the kernel?
> >
> > Yocto: master branch
> > Kernel: linux-yocto-dev
> >
> > phenomenon 1:
> > linux-yocto was ok (v4.18)
> > phenomenon 2:
> > linux-yocto-dev was not ok (v4.19)
> >
> > Maybe some kernel commit breaks it.
> >
> >>
> >>> lttng create trace_session --live -U net://127.0.0.1
> >>>
> >>> ----------------------------------------------------------------------------------------------------------
> >>>
> >>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
> >>> [ 223.550725] BUG: unable to handle kernel paging request at
> >>> 0000111400001158
> >>> [ 223.551661] PGD 0 P4D 0
> >>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
> >>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
> >>> 4.19.0-rc7-yoctodev-standard #1
> >>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> >>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
> >>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> >>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> >>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> >>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> >>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> >>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> >>> 0000111400001158
> >>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> >>> ffffffffc02f1100
> >>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> >>> ffff9b3ddec01600
> >>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> >>> 0000000000000000
> >>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> >>> 0000000000000002
> >>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> >>> knlGS:0000000000000000
> >>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
> >>> 00000000000006f0
> >>> [ 223.566450] Call Trace:
> >>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
> >>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
> >>> [ 223.571309] ? 0xffffffffc0309000
> >>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
> >>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
> >>> [ 223.573820] ? 0xffffffffc0309000
> >>> [ 223.574237] do_one_initcall+0x53/0x1fe
> >>> [ 223.574682] ? __vunmap+0x8e/0xc0
> >>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
> >>> [ 223.575684] ? do_init_module+0x27/0x210
> >>> [ 223.576285] do_init_module+0x60/0x210
> >>> [ 223.576702] load_module+0x2406/0x2570
> >>> [ 223.577140] ? vfs_read+0x115/0x140
> >>> [ 223.577611] ? vfs_read+0x115/0x140
> >>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
> >>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
> >>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
> >>> [ 223.579627] do_syscall_64+0x4d/0xf0
> >>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >>> [ 223.580837] RIP: 0033:0x7fb049107569
> >>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
> >>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
> >>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
> >>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
> >>> 0000000000000139
> >>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
> >>> 00007fb049107569
> >>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
> >>> 000000000000000a
> >>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
> >>> 000055931b680b70
> >>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
> >>> 000055931b67734e
> >>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
> >>> 0000000000000000
> >>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
> >>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
> >>> lttng_kretprobes(O) deflate twofish_geneb
> >>> [ 223.597615] CR2: 0000111400001158
> >>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
> >>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> >>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> >>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> >>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> >>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> >>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> >>> 0000111400001158
> >>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> >>> ffffffffc02f1100
> >>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> >>> ffff9b3ddec01600
> >>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> >>> 0000000000000000
> >>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> >>> 0000000000000002
> >>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> >>> knlGS:0000000000000000
> >>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
> >>> 00000000000006f0
> >>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
> >>> paging request at 0000111400001158
> >>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
> >>> at 0000111400001158
> >>>
> >>> ----------------------------------------------------------------------------------------------------------
> >>>
> >>> When I type that command again:
> >>>
> >>> ----------------------------------------------------------------------------------------------------------
> >>>
> >>> Spawning a session daemon
> >>> Error: Session daemon terminated with an error (exit status: 1)
> >>> Error: Problem occurred while launching session daemon
> >>> (/usr/bin/lttng-sessiond)
> >>> Error: Command error
> >>>
> >>> ----------------------------------------------------------------------------------------------------------
> >>>
> >>>
> >>> _______________________________________________
> >>> lttng-dev mailing list
> >>> lttng-dev@lists.lttng.org
> >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
--
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <20181012162118.GB13655@joraj-alpa>
@ 2018-10-15 2:03 ` Dengke Du
2018-10-18 16:10 ` Jonathan Rajotte-Julien
[not found] ` <20181018160801.GA1396@joraj-alpa>
2 siblings, 0 replies; 9+ messages in thread
From: Dengke Du @ 2018-10-15 2:03 UTC (permalink / raw)
To: Jonathan Rajotte-Julien, Mathieu Desnoyers; +Cc: lttng-dev
OK, thanks.
On 2018年10月13日 00:21, Jonathan Rajotte-Julien wrote:
> Hi Dengke,
>
> We were able to reproduce it. This is not related to yocto and seems to be
> related to a change introduced in the 4.19 kernel.
>
> If you are not interested in kernel tracing you can pass the --no-kernel flag to
> lttng-sessiond to prevent the loading of the kernel modules.
>
> We are looking into the root problem and possible fix on either the kernel side
> or ours.
>
> Thanks for reporting the issue. We will keep you updated.
>
> Cheers
>
> On Fri, Oct 12, 2018 at 11:10:09AM -0400, Mathieu Desnoyers wrote:
>> ----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
>>
>>> On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
>>>> Hi,
>>>>
>>>> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
>>>>> Hi all
>>>>>
>>>>> When I create a local trace session, meeting a call trace:
>>>>>
>>>>> uname -a:
>>>>>
>>>>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
>>>>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
>>>>
>>>> Which version of lttng-modules are you using?
>> Please answer the questions above as well.
>>
>> Thanks,
>>
>> Mathieu
>>
>>
>>>> Any modification to the kernel?
>>> Yocto: master branch
>>> Kernel: linux-yocto-dev
>>>
>>> phenomenon 1:
>>> linux-yocto was ok (v4.18)
>>> phenomenon 2:
>>> linux-yocto-dev was not ok (v4.19)
>>>
>>> Maybe some kernel commit breaks it.
>>>
>>>>> lttng create trace_session --live -U net://127.0.0.1
>>>>>
>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>
>>>>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
>>>>> [ 223.550725] BUG: unable to handle kernel paging request at
>>>>> 0000111400001158
>>>>> [ 223.551661] PGD 0 P4D 0
>>>>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
>>>>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
>>>>> 4.19.0-rc7-yoctodev-standard #1
>>>>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>>>>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
>>>>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>>> 0000111400001158
>>>>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>>> ffffffffc02f1100
>>>>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>>> ffff9b3ddec01600
>>>>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>>> 0000000000000000
>>>>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>>> 0000000000000002
>>>>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>>> knlGS:0000000000000000
>>>>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
>>>>> 00000000000006f0
>>>>> [ 223.566450] Call Trace:
>>>>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
>>>>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
>>>>> [ 223.571309] ? 0xffffffffc0309000
>>>>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
>>>>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
>>>>> [ 223.573820] ? 0xffffffffc0309000
>>>>> [ 223.574237] do_one_initcall+0x53/0x1fe
>>>>> [ 223.574682] ? __vunmap+0x8e/0xc0
>>>>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
>>>>> [ 223.575684] ? do_init_module+0x27/0x210
>>>>> [ 223.576285] do_init_module+0x60/0x210
>>>>> [ 223.576702] load_module+0x2406/0x2570
>>>>> [ 223.577140] ? vfs_read+0x115/0x140
>>>>> [ 223.577611] ? vfs_read+0x115/0x140
>>>>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
>>>>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
>>>>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
>>>>> [ 223.579627] do_syscall_64+0x4d/0xf0
>>>>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>>> [ 223.580837] RIP: 0033:0x7fb049107569
>>>>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
>>>>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
>>>>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
>>>>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
>>>>> 0000000000000139
>>>>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
>>>>> 00007fb049107569
>>>>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
>>>>> 000000000000000a
>>>>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
>>>>> 000055931b680b70
>>>>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
>>>>> 000055931b67734e
>>>>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
>>>>> 0000000000000000
>>>>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
>>>>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
>>>>> lttng_kretprobes(O) deflate twofish_geneb
>>>>> [ 223.597615] CR2: 0000111400001158
>>>>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
>>>>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>>> 0000111400001158
>>>>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>>> ffffffffc02f1100
>>>>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>>> ffff9b3ddec01600
>>>>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>>> 0000000000000000
>>>>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>>> 0000000000000002
>>>>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>>> knlGS:0000000000000000
>>>>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
>>>>> 00000000000006f0
>>>>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
>>>>> paging request at 0000111400001158
>>>>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
>>>>> at 0000111400001158
>>>>>
>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>
>>>>> When I type that command again:
>>>>>
>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>
>>>>> Spawning a session daemon
>>>>> Error: Session daemon terminated with an error (exit status: 1)
>>>>> Error: Problem occurred while launching session daemon
>>>>> (/usr/bin/lttng-sessiond)
>>>>> Error: Command error
>>>>>
>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> lttng-dev@lists.lttng.org
>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev@lists.lttng.org
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <2081244858.3133.1539357009809.JavaMail.zimbra@efficios.com>
2018-10-12 16:21 ` Jonathan Rajotte-Julien
@ 2018-10-15 2:04 ` Dengke Du
[not found] ` <20181012162118.GB13655@joraj-alpa>
2 siblings, 0 replies; 9+ messages in thread
From: Dengke Du @ 2018-10-15 2:04 UTC (permalink / raw)
To: Mathieu Desnoyers; +Cc: lttng-dev
On 2018年10月12日 23:10, Mathieu Desnoyers wrote:
> ----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
>
>> On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
>>> Hi,
>>>
>>> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
>>>> Hi all
>>>>
>>>> When I create a local trace session, meeting a call trace:
>>>>
>>>> uname -a:
>>>>
>>>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
>>>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
>>>
>>> Which version of lttng-modules are you using?
2.9.5
> Please answer the questions above as well.
>
> Thanks,
>
> Mathieu
>
>
>>> Any modification to the kernel?
>> Yocto: master branch
>> Kernel: linux-yocto-dev
>>
>> phenomenon 1:
>> linux-yocto was ok (v4.18)
>> phenomenon 2:
>> linux-yocto-dev was not ok (v4.19)
>>
>> Maybe some kernel commit breaks it.
>>
>>>> lttng create trace_session --live -U net://127.0.0.1
>>>>
>>>> ----------------------------------------------------------------------------------------------------------
>>>>
>>>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
>>>> [ 223.550725] BUG: unable to handle kernel paging request at
>>>> 0000111400001158
>>>> [ 223.551661] PGD 0 P4D 0
>>>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
>>>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
>>>> 4.19.0-rc7-yoctodev-standard #1
>>>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>>>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
>>>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>> 0000111400001158
>>>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>> ffffffffc02f1100
>>>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>> ffff9b3ddec01600
>>>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>> 0000000000000000
>>>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>> 0000000000000002
>>>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>> knlGS:0000000000000000
>>>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
>>>> 00000000000006f0
>>>> [ 223.566450] Call Trace:
>>>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
>>>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
>>>> [ 223.571309] ? 0xffffffffc0309000
>>>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
>>>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
>>>> [ 223.573820] ? 0xffffffffc0309000
>>>> [ 223.574237] do_one_initcall+0x53/0x1fe
>>>> [ 223.574682] ? __vunmap+0x8e/0xc0
>>>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
>>>> [ 223.575684] ? do_init_module+0x27/0x210
>>>> [ 223.576285] do_init_module+0x60/0x210
>>>> [ 223.576702] load_module+0x2406/0x2570
>>>> [ 223.577140] ? vfs_read+0x115/0x140
>>>> [ 223.577611] ? vfs_read+0x115/0x140
>>>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
>>>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
>>>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
>>>> [ 223.579627] do_syscall_64+0x4d/0xf0
>>>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>> [ 223.580837] RIP: 0033:0x7fb049107569
>>>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
>>>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
>>>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
>>>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
>>>> 0000000000000139
>>>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
>>>> 00007fb049107569
>>>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
>>>> 000000000000000a
>>>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
>>>> 000055931b680b70
>>>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
>>>> 000055931b67734e
>>>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
>>>> 0000000000000000
>>>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
>>>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
>>>> lttng_kretprobes(O) deflate twofish_geneb
>>>> [ 223.597615] CR2: 0000111400001158
>>>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
>>>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>> 0000111400001158
>>>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>> ffffffffc02f1100
>>>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>> ffff9b3ddec01600
>>>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>> 0000000000000000
>>>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>> 0000000000000002
>>>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>> knlGS:0000000000000000
>>>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
>>>> 00000000000006f0
>>>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
>>>> paging request at 0000111400001158
>>>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
>>>> at 0000111400001158
>>>>
>>>> ----------------------------------------------------------------------------------------------------------
>>>>
>>>> When I type that command again:
>>>>
>>>> ----------------------------------------------------------------------------------------------------------
>>>>
>>>> Spawning a session daemon
>>>> Error: Session daemon terminated with an error (exit status: 1)
>>>> Error: Problem occurred while launching session daemon
>>>> (/usr/bin/lttng-sessiond)
>>>> Error: Command error
>>>>
>>>> ----------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev@lists.lttng.org
>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <20181012162118.GB13655@joraj-alpa>
2018-10-15 2:03 ` Dengke Du
@ 2018-10-18 16:10 ` Jonathan Rajotte-Julien
[not found] ` <20181018160801.GA1396@joraj-alpa>
2 siblings, 0 replies; 9+ messages in thread
From: Jonathan Rajotte-Julien @ 2018-10-18 16:10 UTC (permalink / raw)
To: Mathieu Desnoyers, dengke.du; +Cc: lttng-dev
Hi Dengke,
The fix for the kernel was pulled into linux-master KH[1]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9c0be3f6b5d776dfe3ed249862c244a4486414dc
The corresponding changes for lttng-modules were merged and backported (2.9, 2.10, 2.11). [2]
https://github.com/lttng/lttng-modules/commit/c1b0a62512c0dad3c79f3eb29ed29d3a702f2274
Again thanks again for reporting this issue.
Cheers
On Fri, Oct 12, 2018 at 12:21:18PM -0400, Jonathan Rajotte-Julien wrote:
> Hi Dengke,
>
> We were able to reproduce it. This is not related to yocto and seems to be
> related to a change introduced in the 4.19 kernel.
>
> If you are not interested in kernel tracing you can pass the --no-kernel flag to
> lttng-sessiond to prevent the loading of the kernel modules.
>
> We are looking into the root problem and possible fix on either the kernel side
> or ours.
>
> Thanks for reporting the issue. We will keep you updated.
>
> Cheers
>
> On Fri, Oct 12, 2018 at 11:10:09AM -0400, Mathieu Desnoyers wrote:
> > ----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
> >
> > > On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
> > >> Hi,
> > >>
> > >> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
> > >>> Hi all
> > >>>
> > >>> When I create a local trace session, meeting a call trace:
> > >>>
> > >>> uname -a:
> > >>>
> > >>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
> > >>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> > >> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
> > >>
> > >> Which version of lttng-modules are you using?
> >
> > Please answer the questions above as well.
> >
> > Thanks,
> >
> > Mathieu
> >
> >
> > >>
> > >> Any modification to the kernel?
> > >
> > > Yocto: master branch
> > > Kernel: linux-yocto-dev
> > >
> > > phenomenon 1:
> > > linux-yocto was ok (v4.18)
> > > phenomenon 2:
> > > linux-yocto-dev was not ok (v4.19)
> > >
> > > Maybe some kernel commit breaks it.
> > >
> > >>
> > >>> lttng create trace_session --live -U net://127.0.0.1
> > >>>
> > >>> ----------------------------------------------------------------------------------------------------------
> > >>>
> > >>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
> > >>> [ 223.550725] BUG: unable to handle kernel paging request at
> > >>> 0000111400001158
> > >>> [ 223.551661] PGD 0 P4D 0
> > >>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
> > >>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
> > >>> 4.19.0-rc7-yoctodev-standard #1
> > >>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > >>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
> > >>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> > >>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> > >>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> > >>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> > >>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> > >>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> > >>> 0000111400001158
> > >>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> > >>> ffffffffc02f1100
> > >>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> > >>> ffff9b3ddec01600
> > >>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> > >>> 0000000000000000
> > >>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> > >>> 0000000000000002
> > >>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> > >>> knlGS:0000000000000000
> > >>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > >>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
> > >>> 00000000000006f0
> > >>> [ 223.566450] Call Trace:
> > >>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
> > >>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
> > >>> [ 223.571309] ? 0xffffffffc0309000
> > >>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
> > >>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
> > >>> [ 223.573820] ? 0xffffffffc0309000
> > >>> [ 223.574237] do_one_initcall+0x53/0x1fe
> > >>> [ 223.574682] ? __vunmap+0x8e/0xc0
> > >>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
> > >>> [ 223.575684] ? do_init_module+0x27/0x210
> > >>> [ 223.576285] do_init_module+0x60/0x210
> > >>> [ 223.576702] load_module+0x2406/0x2570
> > >>> [ 223.577140] ? vfs_read+0x115/0x140
> > >>> [ 223.577611] ? vfs_read+0x115/0x140
> > >>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
> > >>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
> > >>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
> > >>> [ 223.579627] do_syscall_64+0x4d/0xf0
> > >>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > >>> [ 223.580837] RIP: 0033:0x7fb049107569
> > >>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
> > >>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
> > >>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
> > >>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
> > >>> 0000000000000139
> > >>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
> > >>> 00007fb049107569
> > >>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
> > >>> 000000000000000a
> > >>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
> > >>> 000055931b680b70
> > >>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
> > >>> 000055931b67734e
> > >>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
> > >>> 0000000000000000
> > >>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
> > >>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
> > >>> lttng_kretprobes(O) deflate twofish_geneb
> > >>> [ 223.597615] CR2: 0000111400001158
> > >>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
> > >>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
> > >>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
> > >>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
> > >>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
> > >>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
> > >>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
> > >>> 0000111400001158
> > >>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
> > >>> ffffffffc02f1100
> > >>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
> > >>> ffff9b3ddec01600
> > >>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
> > >>> 0000000000000000
> > >>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
> > >>> 0000000000000002
> > >>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
> > >>> knlGS:0000000000000000
> > >>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > >>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
> > >>> 00000000000006f0
> > >>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
> > >>> paging request at 0000111400001158
> > >>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
> > >>> at 0000111400001158
> > >>>
> > >>> ----------------------------------------------------------------------------------------------------------
> > >>>
> > >>> When I type that command again:
> > >>>
> > >>> ----------------------------------------------------------------------------------------------------------
> > >>>
> > >>> Spawning a session daemon
> > >>> Error: Session daemon terminated with an error (exit status: 1)
> > >>> Error: Problem occurred while launching session daemon
> > >>> (/usr/bin/lttng-sessiond)
> > >>> Error: Command error
> > >>>
> > >>> ----------------------------------------------------------------------------------------------------------
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> lttng-dev mailing list
> > >>> lttng-dev@lists.lttng.org
> > >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > >
> > > _______________________________________________
> > > lttng-dev mailing list
> > > lttng-dev@lists.lttng.org
> > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > http://www.efficios.com
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lttng create call trace on kernel 4.19
[not found] ` <20181018160801.GA1396@joraj-alpa>
@ 2018-10-19 2:16 ` Dengke Du
0 siblings, 0 replies; 9+ messages in thread
From: Dengke Du @ 2018-10-19 2:16 UTC (permalink / raw)
To: Jonathan Rajotte-Julien, Mathieu Desnoyers; +Cc: lttng-dev
Thanks!
On 2018年10月19日 00:10, Jonathan Rajotte-Julien wrote:
> Hi Dengke,
>
> The fix for the kernel was pulled into linux-master KH[1]
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9c0be3f6b5d776dfe3ed249862c244a4486414dc
>
> The corresponding changes for lttng-modules were merged and backported (2.9, 2.10, 2.11). [2]
>
> https://github.com/lttng/lttng-modules/commit/c1b0a62512c0dad3c79f3eb29ed29d3a702f2274
>
> Again thanks again for reporting this issue.
>
> Cheers
>
> On Fri, Oct 12, 2018 at 12:21:18PM -0400, Jonathan Rajotte-Julien wrote:
>> Hi Dengke,
>>
>> We were able to reproduce it. This is not related to yocto and seems to be
>> related to a change introduced in the 4.19 kernel.
>>
>> If you are not interested in kernel tracing you can pass the --no-kernel flag to
>> lttng-sessiond to prevent the loading of the kernel modules.
>>
>> We are looking into the root problem and possible fix on either the kernel side
>> or ours.
>>
>> Thanks for reporting the issue. We will keep you updated.
>>
>> Cheers
>>
>> On Fri, Oct 12, 2018 at 11:10:09AM -0400, Mathieu Desnoyers wrote:
>>> ----- On Oct 11, 2018, at 9:21 PM, Dengke Du dengke.du@windriver.com wrote:
>>>
>>>> On 2018年10月11日 21:22, Jonathan Rajotte-Julien wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Oct 11, 2018 at 02:00:11PM +0800, Dengke Du wrote:
>>>>>> Hi all
>>>>>>
>>>>>> When I create a local trace session, meeting a call trace:
>>>>>>
>>>>>> uname -a:
>>>>>>
>>>>>> Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
>>>>>> 03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>>>> Which yocto branch, recipes(for kernel, lttng-tools, lttng-modules) and target?
>>>>>
>>>>> Which version of lttng-modules are you using?
>>> Please answer the questions above as well.
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>
>>>>> Any modification to the kernel?
>>>> Yocto: master branch
>>>> Kernel: linux-yocto-dev
>>>>
>>>> phenomenon 1:
>>>> linux-yocto was ok (v4.18)
>>>> phenomenon 2:
>>>> linux-yocto-dev was not ok (v4.19)
>>>>
>>>> Maybe some kernel commit breaks it.
>>>>
>>>>>> lttng create trace_session --live -U net://127.0.0.1
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>>
>>>>>> [ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
>>>>>> [ 223.550725] BUG: unable to handle kernel paging request at
>>>>>> 0000111400001158
>>>>>> [ 223.551661] PGD 0 P4D 0
>>>>>> [ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
>>>>>> [ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
>>>>>> 4.19.0-rc7-yoctodev-standard #1
>>>>>> [ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>>>>>> rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
>>>>>> [ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>>>> [ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>>>> [ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>>>> [ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>>>> 0000111400001158
>>>>>> [ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>>>> ffffffffc02f1100
>>>>>> [ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>>>> ffff9b3ddec01600
>>>>>> [ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>>>> 0000000000000000
>>>>>> [ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>>>> 0000000000000002
>>>>>> [ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>>>> knlGS:0000000000000000
>>>>>> [ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> [ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
>>>>>> 00000000000006f0
>>>>>> [ 223.566450] Call Trace:
>>>>>> [ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
>>>>>> [ 223.569932] register_tracepoint_module_notifier+0x57/0x80
>>>>>> [ 223.571309] ? 0xffffffffc0309000
>>>>>> [ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
>>>>>> [ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
>>>>>> [ 223.573820] ? 0xffffffffc0309000
>>>>>> [ 223.574237] do_one_initcall+0x53/0x1fe
>>>>>> [ 223.574682] ? __vunmap+0x8e/0xc0
>>>>>> [ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
>>>>>> [ 223.575684] ? do_init_module+0x27/0x210
>>>>>> [ 223.576285] do_init_module+0x60/0x210
>>>>>> [ 223.576702] load_module+0x2406/0x2570
>>>>>> [ 223.577140] ? vfs_read+0x115/0x140
>>>>>> [ 223.577611] ? vfs_read+0x115/0x140
>>>>>> [ 223.578034] __se_sys_finit_module+0xc3/0xe0
>>>>>> [ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
>>>>>> [ 223.579135] __x64_sys_finit_module+0x1a/0x20
>>>>>> [ 223.579627] do_syscall_64+0x4d/0xf0
>>>>>> [ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>>>> [ 223.580837] RIP: 0033:0x7fb049107569
>>>>>> [ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48
>>>>>> 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
>>>>>> 3d 01 f0 ff ff 73 01 c3 48 8b 08
>>>>>> [ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
>>>>>> 0000000000000139
>>>>>> [ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
>>>>>> 00007fb049107569
>>>>>> [ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
>>>>>> 000000000000000a
>>>>>> [ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
>>>>>> 000055931b680b70
>>>>>> [ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
>>>>>> 000055931b67734e
>>>>>> [ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
>>>>>> 0000000000000000
>>>>>> [ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
>>>>>> lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
>>>>>> lttng_kretprobes(O) deflate twofish_geneb
>>>>>> [ 223.597615] CR2: 0000111400001158
>>>>>> [ 223.614673] ---[ end trace 514884c3611a01af ]---
>>>>>> [ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
>>>>>> [ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83 c4
>>>>>> 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c c2 <4c>
>>>>>> 8b 31 48 89 4d d0 4c 89 f7 e8 90
>>>>>> [ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
>>>>>> [ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
>>>>>> 0000111400001158
>>>>>> [ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
>>>>>> ffffffffc02f1100
>>>>>> [ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
>>>>>> ffff9b3ddec01600
>>>>>> [ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
>>>>>> 0000000000000000
>>>>>> [ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
>>>>>> 0000000000000002
>>>>>> [ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
>>>>>> knlGS:0000000000000000
>>>>>> [ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> [ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
>>>>>> 00000000000006f0
>>>>>> 2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle kernel
>>>>>> paging request at 0000111400001158
>>>>>> 2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging request
>>>>>> at 0000111400001158
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>>
>>>>>> When I type that command again:
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>>
>>>>>> Spawning a session daemon
>>>>>> Error: Session daemon terminated with an error (exit status: 1)
>>>>>> Error: Problem occurred while launching session daemon
>>>>>> (/usr/bin/lttng-sessiond)
>>>>>> Error: Command error
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> lttng-dev@lists.lttng.org
>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev@lists.lttng.org
>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com
>> --
>> Jonathan Rajotte-Julien
>> EfficiOS
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* lttng create call trace on kernel 4.19
@ 2018-10-11 6:00 Dengke Du
0 siblings, 0 replies; 9+ messages in thread
From: Dengke Du @ 2018-10-11 6:00 UTC (permalink / raw)
To: lttng-dev
Hi all
When I create a local trace session, meeting a call trace:
uname -a:
Linux qemux86-64 4.19.0-rc7-yoctodev-standard #1 SMP PREEMPT Thu Oct 11
03:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
lttng create trace_session --live -U net://127.0.0.1
----------------------------------------------------------------------------------------------------------
[ 222.390601] lttng_kretprobes: loading out-of-tree module taints kernel.
[ 223.550725] BUG: unable to handle kernel paging request at
0000111400001158
[ 223.551661] PGD 0 P4D 0
[ 223.552476] Oops: 0000 [#1] PREEMPT SMP PTI
[ 223.553170] CPU: 0 PID: 500 Comm: modprobe Tainted: G O
4.19.0-rc7-yoctodev-standard #1
[ 223.553892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
[ 223.556619] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
[ 223.557751] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83
c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c
c2 <4c> 8b 31 48 89 4d d0 4c 89 f7 e8 90
[ 223.559769] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
[ 223.560303] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
0000111400001158
[ 223.560959] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
ffffffffc02f1100
[ 223.561579] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
ffff9b3ddec01600
[ 223.562388] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
0000000000000000
[ 223.563008] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
0000000000000002
[ 223.563905] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
knlGS:0000000000000000
[ 223.564941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 223.565501] CR2: 0000111400001158 CR3: 0000000014586000 CR4:
00000000000006f0
[ 223.566450] Call Trace:
[ 223.568367] ? lttng_kernel_tracepoint_add+0x50/0xc0 [lttng_tracer]
[ 223.569932] register_tracepoint_module_notifier+0x57/0x80
[ 223.571309] ? 0xffffffffc0309000
[ 223.572187] lttng_tracepoint_init+0x45/0xbf9 [lttng_tracer]
[ 223.573125] lttng_events_init+0x40/0x1cc [lttng_tracer]
[ 223.573820] ? 0xffffffffc0309000
[ 223.574237] do_one_initcall+0x53/0x1fe
[ 223.574682] ? __vunmap+0x8e/0xc0
[ 223.575196] ? kmem_cache_alloc_trace+0xfc/0x1d0
[ 223.575684] ? do_init_module+0x27/0x210
[ 223.576285] do_init_module+0x60/0x210
[ 223.576702] load_module+0x2406/0x2570
[ 223.577140] ? vfs_read+0x115/0x140
[ 223.577611] ? vfs_read+0x115/0x140
[ 223.578034] __se_sys_finit_module+0xc3/0xe0
[ 223.578581] ? __se_sys_finit_module+0xc3/0xe0
[ 223.579135] __x64_sys_finit_module+0x1a/0x20
[ 223.579627] do_syscall_64+0x4d/0xf0
[ 223.580071] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.580837] RIP: 0033:0x7fb049107569
[ 223.581286] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 08
[ 223.583210] RSP: 002b:00007ffc80620498 EFLAGS: 00000246 ORIG_RAX:
0000000000000139
[ 223.583958] RAX: ffffffffffffffda RBX: 000055931b680b70 RCX:
00007fb049107569
[ 223.584677] RDX: 0000000000000000 RSI: 000055931b67734e RDI:
000000000000000a
[ 223.585372] RBP: 0000000000000000 R08: 0000000000000000 R09:
000055931b680b70
[ 223.586008] R10: 000000000000000a R11: 0000000000000246 R12:
000055931b67734e
[ 223.586696] R13: 0000000000040000 R14: 0000000000000000 R15:
0000000000000000
[ 223.587620] Modules linked in: lttng_tracer(O+) lttng_statedump(O)
lttng_ftrace(O) lttng_kprobes(O) lttng_clock(O) lttng_lib_ring_buffer(O)
lttng_kretprobes(O) deflate twofish_geneb
[ 223.597615] CR2: 0000111400001158
[ 223.614673] ---[ end trace 514884c3611a01af ]---
[ 223.615793] RIP: 0010:lttng_tracepoint_notify+0x16e/0x200 [lttng_tracer]
[ 223.616604] Code: eb bd 48 39 c1 0f 85 a3 00 00 00 49 8b 55 10 41 83
c4 01 44 39 a2 8c 02 00 00 76 a3 48 8b 92 90 02 00 00 49 63 c4 48 8b 0c
c2 <4c> 8b 31 48 89 4d d0 4c 89 f7 e8 90
[ 223.624684] RSP: 0000:ffffa39a00373bc0 EFLAGS: 00000206
[ 223.625177] RAX: 0000000000000000 RBX: ffff9b3dd45a83a0 RCX:
0000111400001158
[ 223.626102] RDX: ffffffffc01b9048 RSI: 0000000000000001 RDI:
ffffffffc02f1100
[ 223.632541] RBP: ffffa39a00373bf0 R08: 0000000000000064 R09:
ffff9b3ddec01600
[ 223.633110] R10: ffffffffc0287411 R11: ffffffff98dc9660 R12:
0000000000000000
[ 223.637690] R13: ffff9b3dd45a83a0 R14: ffffa39a00373e78 R15:
0000000000000002
[ 223.641583] FS: 00007fb049015740(0000) GS:ffff9b3ddf200000(0000)
knlGS:0000000000000000
[ 223.642238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 223.642906] CR2: 00007f6eae3e4b58 CR3: 0000000014586000 CR4:
00000000000006f0
2018 Oct 11 05:37:07 qemux86-64 [ 223.550725] BUG: unable to handle
kernel paging request at 0000111400001158
2018 Oct 11 05:37:07 qemux86-64 BUG: unable to handle kernel paging
request at 0000111400001158
----------------------------------------------------------------------------------------------------------
When I type that command again:
----------------------------------------------------------------------------------------------------------
Spawning a session daemon
Error: Session daemon terminated with an error (exit status: 1)
Error: Problem occurred while launching session daemon
(/usr/bin/lttng-sessiond)
Error: Command error
----------------------------------------------------------------------------------------------------------
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-10-19 2:20 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <eee78bd1-4d5c-63e7-dc74-ff5a51426666@windriver.com>
2018-10-11 13:22 ` lttng create call trace on kernel 4.19 Jonathan Rajotte-Julien
[not found] ` <20181011132253.GA13655@joraj-alpa>
2018-10-12 1:21 ` Dengke Du
[not found] ` <82561ba6-42f9-d816-0d33-f9906d0affcd@windriver.com>
2018-10-12 15:10 ` Mathieu Desnoyers
[not found] ` <2081244858.3133.1539357009809.JavaMail.zimbra@efficios.com>
2018-10-12 16:21 ` Jonathan Rajotte-Julien
2018-10-15 2:04 ` Dengke Du
[not found] ` <20181012162118.GB13655@joraj-alpa>
2018-10-15 2:03 ` Dengke Du
2018-10-18 16:10 ` Jonathan Rajotte-Julien
[not found] ` <20181018160801.GA1396@joraj-alpa>
2018-10-19 2:16 ` Dengke Du
2018-10-11 6:00 Dengke Du
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.