[Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

Hi Philippe,

something got broken with the exception path rework:

[    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
[    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
[    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
[    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
[    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[    2.628911] I-pipe domain: Linux
[    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
[    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
[    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
[    2.631427] Call Trace:
[    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
[    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
[    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
[    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
[    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
[    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
[    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
[    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
[    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
[    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
[    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
[    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
[    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
[    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
[    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
[    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
[    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
[    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
[    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
[    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0

Unless you have some immediate idea, I will debug this tomorrow. Issue
disappears when I revert the related patches in the ipipe queue.

I also have the build fix for 4.4 lined up, basically a backport of
"enable dev_printk() from the head stage" to that kernel. I've pushed
things to my for-upstream/4.4 branch, though they aren't ready yet.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/14/2017 08:28 PM, Jan Kiszka wrote:
> Hi Philippe,
> 
> something got broken with the exception path rework:
> 
> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
> [    2.628911] I-pipe domain: Linux
> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
> [    2.631427] Call Trace:
> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
> 
> Unless you have some immediate idea, I will debug this tomorrow. Issue
> disappears when I revert the related patches in the ipipe queue.
> 

Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
sequence used to be prologue -> handler -> restore_root, it is now
prologue -> restore_root -> handler, which basically means that the
stall bit fixup is no more, ... a fixup.

-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

On 2017-12-14 20:44, Philippe Gerum wrote:
> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>> Hi Philippe,
>>
>> something got broken with the exception path rework:
>>
>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>> [    2.628911] I-pipe domain: Linux
>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>> [    2.631427] Call Trace:
>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>
>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>> disappears when I revert the related patches in the ipipe queue.
>>
> 
> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
> sequence used to be prologue -> handler -> restore_root, it is now
> prologue -> restore_root -> handler, which basically means that the
> stall bit fixup is no more, ... a fixup.

Right, this is broken. The previous code was well matured, it just
excluded the case int3 for dyn-ftrace, kprobe etc. so far.

While gaining that support would be valuable, I would still recommend to
revert the changes for now and redesign the int3 handling carefully on top.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/15/2017 03:24 PM, Jan Kiszka wrote:
> On 2017-12-14 20:44, Philippe Gerum wrote:
>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>> Hi Philippe,
>>>
>>> something got broken with the exception path rework:
>>>
>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>> [    2.628911] I-pipe domain: Linux
>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>> [    2.631427] Call Trace:
>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>
>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>> disappears when I revert the related patches in the ipipe queue.
>>>
>>
>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>> sequence used to be prologue -> handler -> restore_root, it is now
>> prologue -> restore_root -> handler, which basically means that the
>> stall bit fixup is no more, ... a fixup.
> 
> Right, this is broken. The previous code was well matured, it just
> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
> 

Actually, the code was broken with int3 as issued by ftrace. This is
what triggered the change.

> While gaining that support would be valuable, I would still recommend to
> revert the changes for now and redesign the int3 handling carefully on top.
> 

We should not be that far from a proper implementation. I'd like to try
the kvm config that shows the issue. Is there any way to reproduce this
easily?

-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

On 2017-12-15 15:44, Philippe Gerum wrote:
> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>> Hi Philippe,
>>>>
>>>> something got broken with the exception path rework:
>>>>
>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>> [    2.628911] I-pipe domain: Linux
>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>> [    2.631427] Call Trace:
>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>
>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>> disappears when I revert the related patches in the ipipe queue.
>>>>
>>>
>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>> sequence used to be prologue -> handler -> restore_root, it is now
>>> prologue -> restore_root -> handler, which basically means that the
>>> stall bit fixup is no more, ... a fixup.
>>
>> Right, this is broken. The previous code was well matured, it just
>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>
> 
> Actually, the code was broken with int3 as issued by ftrace. This is
> what triggered the change.

Nope, you just had to turn those things off. We may have missed some
Kconfig dependencies, true, but we never supported code patching like
for dyn-ftrace, jump labels, or similar things. We should, but it's not
trivial as we see.

> 
>> While gaining that support would be valuable, I would still recommend to
>> revert the changes for now and redesign the int3 handling carefully on top.
>>
> 
> We should not be that far from a proper implementation. I'd like to try
> the kvm config that shows the issue. Is there any way to reproduce this
> easily?

CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
must run *after* the Linux handler. This is now broken.

Look, I've debugged that stuff more than once, and it took quite some
effort and multiple attempts until we ended up with the current stable
version. We should work on top of that to enable code patching via int3,
very carefully and with good explanations (you can find my attempts for
the current version also in git). And that's why we should revert first.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.xz
Type: application/x-xz
Size: 24444 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20171215/435212cd/attachment.bin>

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/15/2017 04:02 PM, Jan Kiszka wrote:
> On 2017-12-15 15:44, Philippe Gerum wrote:
>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>> Hi Philippe,
>>>>>
>>>>> something got broken with the exception path rework:
>>>>>
>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>> [    2.628911] I-pipe domain: Linux
>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>> [    2.631427] Call Trace:
>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>
>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>
>>>>
>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>> prologue -> restore_root -> handler, which basically means that the
>>>> stall bit fixup is no more, ... a fixup.
>>>
>>> Right, this is broken. The previous code was well matured, it just
>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>
>>
>> Actually, the code was broken with int3 as issued by ftrace. This is
>> what triggered the change.
> 
> Nope, you just had to turn those things off. We may have missed some
> Kconfig dependencies, true, but we never supported code patching like
> for dyn-ftrace, jump labels, or similar things. We should, but it's not
> trivial as we see.
> 
>>
>>> While gaining that support would be valuable, I would still recommend to
>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>
>>
>> We should not be that far from a proper implementation. I'd like to try
>> the kvm config that shows the issue. Is there any way to reproduce this
>> easily?
> 
> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
> must run *after* the Linux handler. This is now broken.
> 
> Look, I've debugged that stuff more than once, and it took quite some
> effort and multiple attempts until we ended up with the current stable
> version. We should work on top of that to enable code patching via int3,
> very carefully and with good explanations (you can find my attempts for
> the current version also in git). And that's why we should revert first.
> 

Absolutely no issue with that, I'll revert those patches and let the
I-pipe/x86 maintainers tackle that stuff. I can live with a separate
branch anyway.

-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/15/2017 05:45 PM, Philippe Gerum wrote:
> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>> Hi Philippe,
>>>>>>
>>>>>> something got broken with the exception path rework:
>>>>>>
>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>> [    2.631427] Call Trace:
>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>
>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>
>>>>>
>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>> stall bit fixup is no more, ... a fixup.
>>>>
>>>> Right, this is broken. The previous code was well matured, it just
>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>
>>>
>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>> what triggered the change.
>>
>> Nope, you just had to turn those things off. We may have missed some
>> Kconfig dependencies, true, but we never supported code patching like
>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>> trivial as we see.
>>
>>>
>>>> While gaining that support would be valuable, I would still recommend to
>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>
>>>
>>> We should not be that far from a proper implementation. I'd like to try
>>> the kvm config that shows the issue. Is there any way to reproduce this
>>> easily?
>>
>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>> must run *after* the Linux handler. This is now broken.
>>
>> Look, I've debugged that stuff more than once, and it took quite some
>> effort and multiple attempts until we ended up with the current stable
>> version. We should work on top of that to enable code patching via int3,
>> very carefully and with good explanations (you can find my attempts for
>> the current version also in git). And that's why we should revert first.
>>
> 
> Absolutely no issue with that, I'll revert those patches and let the
> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
> branch anyway.
> 

Ok, done. I'll try to fix the issue I hinted at [1] in a separate
branch, checking with your config if things get better. I'll keep you
posted.

[1] http://xenomai.org/pipermail/xenomai/2017-December/038119.html

-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

On 2017-12-15 17:45, Philippe Gerum wrote:
> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>> Hi Philippe,
>>>>>>
>>>>>> something got broken with the exception path rework:
>>>>>>
>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>> [    2.631427] Call Trace:
>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>
>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>
>>>>>
>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>> stall bit fixup is no more, ... a fixup.
>>>>
>>>> Right, this is broken. The previous code was well matured, it just
>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>
>>>
>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>> what triggered the change.
>>
>> Nope, you just had to turn those things off. We may have missed some
>> Kconfig dependencies, true, but we never supported code patching like
>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>> trivial as we see.
>>
>>>
>>>> While gaining that support would be valuable, I would still recommend to
>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>
>>>
>>> We should not be that far from a proper implementation. I'd like to try
>>> the kvm config that shows the issue. Is there any way to reproduce this
>>> easily?
>>
>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>> must run *after* the Linux handler. This is now broken.
>>
>> Look, I've debugged that stuff more than once, and it took quite some
>> effort and multiple attempts until we ended up with the current stable
>> version. We should work on top of that to enable code patching via int3,
>> very carefully and with good explanations (you can find my attempts for
>> the current version also in git). And that's why we should revert first.
>>
> 
> Absolutely no issue with that, I'll revert those patches and let the
> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
> branch anyway.

Thanks. But you probably don't need a branch of your own for using
ftrace, just this for now:

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 834e6117f5f8..b9f6121c8530 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -83,7 +83,7 @@ config X86
 	select HAVE_AOUT			if X86_32
 	select HAVE_ARCH_AUDITSYSCALL
 	select HAVE_ARCH_HUGE_VMAP		if X86_64 || X86_PAE
-	select HAVE_ARCH_JUMP_LABEL
+	select HAVE_ARCH_JUMP_LABEL		if !IPIPE
 	select HAVE_ARCH_KASAN			if X86_64 && SPARSEMEM_VMEMMAP
 	select HAVE_ARCH_KGDB
 	select HAVE_ARCH_KMEMCHECK


That was (effectively) what I had to explain internally as well a while
ago but forgot to push it.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

On 2017-12-15 18:27, Philippe Gerum wrote:
> On 12/15/2017 05:45 PM, Philippe Gerum wrote:
>> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>>> Hi Philippe,
>>>>>>>
>>>>>>> something got broken with the exception path rework:
>>>>>>>
>>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>>> [    2.631427] Call Trace:
>>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>>
>>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>>
>>>>>>
>>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>>> stall bit fixup is no more, ... a fixup.
>>>>>
>>>>> Right, this is broken. The previous code was well matured, it just
>>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>>
>>>>
>>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>>> what triggered the change.
>>>
>>> Nope, you just had to turn those things off. We may have missed some
>>> Kconfig dependencies, true, but we never supported code patching like
>>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>>> trivial as we see.
>>>
>>>>
>>>>> While gaining that support would be valuable, I would still recommend to
>>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>>
>>>>
>>>> We should not be that far from a proper implementation. I'd like to try
>>>> the kvm config that shows the issue. Is there any way to reproduce this
>>>> easily?
>>>
>>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>>> must run *after* the Linux handler. This is now broken.
>>>
>>> Look, I've debugged that stuff more than once, and it took quite some
>>> effort and multiple attempts until we ended up with the current stable
>>> version. We should work on top of that to enable code patching via int3,
>>> very carefully and with good explanations (you can find my attempts for
>>> the current version also in git). And that's why we should revert first.
>>>
>>
>> Absolutely no issue with that, I'll revert those patches and let the
>> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
>> branch anyway.
>>
> 
> Ok, done. I'll try to fix the issue I hinted at [1] in a separate
> branch, checking with your config if things get better. I'll keep you
> posted.
> 
> [1] http://xenomai.org/pipermail/xenomai/2017-December/038119.html
> 

Another thing to look into carefully is what is actually run then over
any domain after a change like "cope with int3 triggering kernel
probes". Just like with current tracepoint locking, also that code may
take locks and trigger effects in Linux that may not always work over
the head domain. IIRC, that was mostly what held me back a while a ago
to enable dyn-ftrace, e.g., for I-pipe, but I need to refresh my memory
on that.

Finally we need to retest KGDB. I didn't do this in a while, and maybe
it was broken long ago. Can check this, the setup is very similar to gdb
via QEMU/KVM.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/15/2017 06:53 PM, Jan Kiszka wrote:
> On 2017-12-15 17:45, Philippe Gerum wrote:
>> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>>> Hi Philippe,
>>>>>>>
>>>>>>> something got broken with the exception path rework:
>>>>>>>
>>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>>> [    2.631427] Call Trace:
>>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>>
>>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>>
>>>>>>
>>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>>> stall bit fixup is no more, ... a fixup.
>>>>>
>>>>> Right, this is broken. The previous code was well matured, it just
>>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>>
>>>>
>>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>>> what triggered the change.
>>>
>>> Nope, you just had to turn those things off. We may have missed some
>>> Kconfig dependencies, true, but we never supported code patching like
>>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>>> trivial as we see.
>>>
>>>>
>>>>> While gaining that support would be valuable, I would still recommend to
>>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>>
>>>>
>>>> We should not be that far from a proper implementation. I'd like to try
>>>> the kvm config that shows the issue. Is there any way to reproduce this
>>>> easily?
>>>
>>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>>> must run *after* the Linux handler. This is now broken.
>>>
>>> Look, I've debugged that stuff more than once, and it took quite some
>>> effort and multiple attempts until we ended up with the current stable
>>> version. We should work on top of that to enable code patching via int3,
>>> very carefully and with good explanations (you can find my attempts for
>>> the current version also in git). And that's why we should revert first.
>>>
>>
>> Absolutely no issue with that, I'll revert those patches and let the
>> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
>> branch anyway.
> 
> Thanks. But you probably don't need a branch of your own for using
> ftrace, just this for now:
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 834e6117f5f8..b9f6121c8530 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -83,7 +83,7 @@ config X86
>  	select HAVE_AOUT			if X86_32
>  	select HAVE_ARCH_AUDITSYSCALL
>  	select HAVE_ARCH_HUGE_VMAP		if X86_64 || X86_PAE
> -	select HAVE_ARCH_JUMP_LABEL
> +	select HAVE_ARCH_JUMP_LABEL		if !IPIPE
>  	select HAVE_ARCH_KASAN			if X86_64 && SPARSEMEM_VMEMMAP
>  	select HAVE_ARCH_KGDB
>  	select HAVE_ARCH_KMEMCHECK
> 
> 
> That was (effectively) what I had to explain internally as well a while
> ago but forgot to push it.
> 

Ok, I'll try that. Thanks.


-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Philippe Gerum]

On 12/15/2017 06:53 PM, Jan Kiszka wrote:
> On 2017-12-15 17:45, Philippe Gerum wrote:
>> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>>> Hi Philippe,
>>>>>>>
>>>>>>> something got broken with the exception path rework:
>>>>>>>
>>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>>> [    2.631427] Call Trace:
>>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>>
>>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>>
>>>>>>
>>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>>> stall bit fixup is no more, ... a fixup.
>>>>>
>>>>> Right, this is broken. The previous code was well matured, it just
>>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>>
>>>>
>>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>>> what triggered the change.
>>>
>>> Nope, you just had to turn those things off. We may have missed some
>>> Kconfig dependencies, true, but we never supported code patching like
>>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>>> trivial as we see.
>>>
>>>>
>>>>> While gaining that support would be valuable, I would still recommend to
>>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>>
>>>>
>>>> We should not be that far from a proper implementation. I'd like to try
>>>> the kvm config that shows the issue. Is there any way to reproduce this
>>>> easily?
>>>
>>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>>> must run *after* the Linux handler. This is now broken.
>>>
>>> Look, I've debugged that stuff more than once, and it took quite some
>>> effort and multiple attempts until we ended up with the current stable
>>> version. We should work on top of that to enable code patching via int3,
>>> very carefully and with good explanations (you can find my attempts for
>>> the current version also in git). And that's why we should revert first.
>>>
>>
>> Absolutely no issue with that, I'll revert those patches and let the
>> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
>> branch anyway.
> 
> Thanks. But you probably don't need a branch of your own for using
> ftrace, just this for now:
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 834e6117f5f8..b9f6121c8530 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -83,7 +83,7 @@ config X86
>  	select HAVE_AOUT			if X86_32
>  	select HAVE_ARCH_AUDITSYSCALL
>  	select HAVE_ARCH_HUGE_VMAP		if X86_64 || X86_PAE
> -	select HAVE_ARCH_JUMP_LABEL
> +	select HAVE_ARCH_JUMP_LABEL		if !IPIPE
>  	select HAVE_ARCH_KASAN			if X86_64 && SPARSEMEM_VMEMMAP
>  	select HAVE_ARCH_KGDB
>  	select HAVE_ARCH_KMEMCHECK
> 
> 
> That was (effectively) what I had to explain internally as well a while
> ago but forgot to push it.
> 

Could you elaborate a bit on the reason for this patch? Is this because
a portion of the jump label machinery might run over the head stage with
dyn-ftracing?

-- 
Philippe.

Re: [Xenomai] ipipe: 4.4 BUG on x86, maybe on 4.9 as well

[Jan Kiszka]

On 2017-12-15 22:29, Philippe Gerum wrote:
> On 12/15/2017 06:53 PM, Jan Kiszka wrote:
>> On 2017-12-15 17:45, Philippe Gerum wrote:
>>> On 12/15/2017 04:02 PM, Jan Kiszka wrote:
>>>> On 2017-12-15 15:44, Philippe Gerum wrote:
>>>>> On 12/15/2017 03:24 PM, Jan Kiszka wrote:
>>>>>> On 2017-12-14 20:44, Philippe Gerum wrote:
>>>>>>> On 12/14/2017 08:28 PM, Jan Kiszka wrote:
>>>>>>>> Hi Philippe,
>>>>>>>>
>>>>>>>> something got broken with the exception path rework:
>>>>>>>>
>>>>>>>> [    2.619458] debug: unmapping init [mem 0xffffffff81b43000-0xffffffff81ee5fff]
>>>>>>>> [    2.622011] BUG: using __this_cpu_read() in preemptible [00000000] code: init/1
>>>>>>>> [    2.623446] caller is __this_cpu_preempt_check+0x13/0x20
>>>>>>>> [    2.624724] CPU: 2 PID: 1 Comm: init Not tainted 4.4.105+ #689
>>>>>>>> [    2.626073] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
>>>>>>>> [    2.628911] I-pipe domain: Linux
>>>>>>>> [    2.630083]  ffffffff81993439 ffff88003fb43c10 ffffffff813afa45 0000000000000002
>>>>>>>> [    2.631427]  ffff88003fb38000 ffff88003fb43c50 ffffffff813db5eb ffffffff82c57340
>>>>>>>> [    2.631427]  ffff88003fb43c98 0000000000000002 ffff88003fb44000 0000000000000000
>>>>>>>> [    2.631427] Call Trace:
>>>>>>>> [    2.631427]  [<ffffffff813afa45>] dump_stack+0xb2/0xdd
>>>>>>>> [    2.631427]  [<ffffffff813db5eb>] check_preemption_disabled+0x17b/0x1c0
>>>>>>>> [    2.631427]  [<ffffffff813db663>] __this_cpu_preempt_check+0x13/0x20
>>>>>>>> [    2.631427]  [<ffffffff8104eb1f>] do_page_fault+0x3f/0x60
>>>>>>>> [    2.631427]  [<ffffffff8170f8b7>] page_fault+0x27/0x60
>>>>>>>> [    2.631427]  [<ffffffff813bcef2>] ? __clear_user+0x42/0x70
>>>>>>>> [    2.631427]  [<ffffffff813bcf69>] clear_user+0x49/0x80
>>>>>>>> [    2.631427]  [<ffffffff812727d4>] padzero+0x24/0x40
>>>>>>>> [    2.631427]  [<ffffffff81274bfe>] load_elf_binary+0x8fe/0x10f0
>>>>>>>> [    2.631427]  [<ffffffff81116ee8>] ? ipipe_unstall_root+0x58/0x90
>>>>>>>> [    2.631427]  [<ffffffff812222c7>] search_binary_handler+0x97/0x1d0
>>>>>>>> [    2.631427]  [<ffffffff81223b86>] do_execveat_common.isra.40+0x5d6/0x7e0
>>>>>>>> [    2.631427]  [<ffffffff81223aff>] ? do_execveat_common.isra.40+0x54f/0x7e0
>>>>>>>> [    2.631427]  [<ffffffff8122bb00>] ? path_openat+0x1440/0x14e0
>>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>>> [    2.631427]  [<ffffffff81223dbc>] do_execve+0x2c/0x30
>>>>>>>> [    2.631427]  [<ffffffff810002eb>] run_init_process+0x2b/0x30
>>>>>>>> [    2.631427]  [<ffffffff816fff5d>] kernel_init+0x3d/0xe0
>>>>>>>> [    2.631427]  [<ffffffff8170d4c0>] ret_from_fork+0x40/0x70
>>>>>>>> [    2.631427]  [<ffffffff816fff20>] ? rest_init+0xd0/0xd0
>>>>>>>>
>>>>>>>> Unless you have some immediate idea, I will debug this tomorrow. Issue
>>>>>>>> disappears when I revert the related patches in the ipipe queue.
>>>>>>>>
>>>>>>>
>>>>>>> Issue may be in the changes affecting IPIPE_DO_TRAP() [4e180c151]. The
>>>>>>> sequence used to be prologue -> handler -> restore_root, it is now
>>>>>>> prologue -> restore_root -> handler, which basically means that the
>>>>>>> stall bit fixup is no more, ... a fixup.
>>>>>>
>>>>>> Right, this is broken. The previous code was well matured, it just
>>>>>> excluded the case int3 for dyn-ftrace, kprobe etc. so far.
>>>>>>
>>>>>
>>>>> Actually, the code was broken with int3 as issued by ftrace. This is
>>>>> what triggered the change.
>>>>
>>>> Nope, you just had to turn those things off. We may have missed some
>>>> Kconfig dependencies, true, but we never supported code patching like
>>>> for dyn-ftrace, jump labels, or similar things. We should, but it's not
>>>> trivial as we see.
>>>>
>>>>>
>>>>>> While gaining that support would be valuable, I would still recommend to
>>>>>> revert the changes for now and redesign the int3 handling carefully on top.
>>>>>>
>>>>>
>>>>> We should not be that far from a proper implementation. I'd like to try
>>>>> the kvm config that shows the issue. Is there any way to reproduce this
>>>>> easily?
>>>>
>>>> CONFIG_DEBUG_*. I've attached mine. One key aspect is that the fixup
>>>> must run *after* the Linux handler. This is now broken.
>>>>
>>>> Look, I've debugged that stuff more than once, and it took quite some
>>>> effort and multiple attempts until we ended up with the current stable
>>>> version. We should work on top of that to enable code patching via int3,
>>>> very carefully and with good explanations (you can find my attempts for
>>>> the current version also in git). And that's why we should revert first.
>>>>
>>>
>>> Absolutely no issue with that, I'll revert those patches and let the
>>> I-pipe/x86 maintainers tackle that stuff. I can live with a separate
>>> branch anyway.
>>
>> Thanks. But you probably don't need a branch of your own for using
>> ftrace, just this for now:
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 834e6117f5f8..b9f6121c8530 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -83,7 +83,7 @@ config X86
>>  	select HAVE_AOUT			if X86_32
>>  	select HAVE_ARCH_AUDITSYSCALL
>>  	select HAVE_ARCH_HUGE_VMAP		if X86_64 || X86_PAE
>> -	select HAVE_ARCH_JUMP_LABEL
>> +	select HAVE_ARCH_JUMP_LABEL		if !IPIPE
>>  	select HAVE_ARCH_KASAN			if X86_64 && SPARSEMEM_VMEMMAP
>>  	select HAVE_ARCH_KGDB
>>  	select HAVE_ARCH_KMEMCHECK
>>
>>
>> That was (effectively) what I had to explain internally as well a while
>> ago but forgot to push it.
>>
> 
> Could you elaborate a bit on the reason for this patch? Is this because
> a portion of the jump label machinery might run over the head stage with
> dyn-ftracing?
> 

Jump labels modify if conditions to always take or skip a branch, i.e.
they patch the code statically. But self-modifying code is a complex
thing on x86. To ensure no stale obcodes are picked by the CPU during
the transition, an int3 is injected temporarily (text_poke_bp). If we
hit this with the current exception code path of I-pipe (apparently,
that's easy while tracing a live Xenomai system), our problem is back.

There are likely more cases like that, or there will be more. So we
should really resolve this limitation properly.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux