[Xenomai-help] CONFIG_PREEMPT & irqbench

[Xenomai-help] CONFIG_PREEMPT & irqbench

[Markus Franke]

Dear all,

I am running some latency tests with irqbench/irqloop. I am wondering
whether it would be possible to achieve better results when activating
CONFIG_PREEMPT and CONFIG_PREEMPT_VOLUNTARILY during the kernel
configuration and running irqloop in User Mode over this kernel? Does it
make any sense? I think in theory it should give better results because
this irqloop runs in secondary(linux) domain when started, right? An
increasing preemptibility of the linux kernel should be better for the
irqloop-task.
All tests were made under heavy I/O and CPU load by means of "dd",
"pingflood" and "cpuburn". Nevertheless, I can only achieve worse
results when activating CONFIG_PREEMPT.


Thanks in advance and regards,

Markus Franke

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

Markus Franke wrote:
> Dear all,
> 
> I am running some latency tests with irqbench/irqloop. I am wondering
> whether it would be possible to achieve better results when activating
> CONFIG_PREEMPT and CONFIG_PREEMPT_VOLUNTARILY during the kernel
> configuration and running irqloop in User Mode over this kernel?

Nope.

> Does it make any sense?

Nope. :)

> I think in theory it should give better results because
> this irqloop runs in secondary(linux) domain when started, right? An
> increasing preemptibility of the linux kernel should be better for the
> irqloop-task.

The Linux kernel is already fully preemptible by Xenomai once you
applied the I-pipe patch. Therefore, you are free to pick the Linux
preemption strategy according to your *Linux* load, independent of what
the real-time part needs.

Unless you have interactive Linux programs running, I can suggest to
pick PREEMPT_VOLUNTARILY or even PREEMPT_NONE, specifically on low-end
hardware.

> All tests were made under heavy I/O and CPU load by means of "dd",
> "pingflood" and "cpuburn". Nevertheless, I can only achieve worse
> results when activating CONFIG_PREEMPT.

Do you have CONFIG_DEBUG_PREEMPT set as well then? This option still as
a small but measurable impact on Xenomai due to micro-dependencies that
as scheduled to be removed in the near future.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

Jan Kiszka wrote:
> Markus Franke wrote:
>> Dear all,
>>
>> I am running some latency tests with irqbench/irqloop. I am wondering
>> whether it would be possible to achieve better results when activating
>> CONFIG_PREEMPT and CONFIG_PREEMPT_VOLUNTARILY during the kernel
>> configuration and running irqloop in User Mode over this kernel?
> 
> Nope.
> 
>> Does it make any sense?
> 
> Nope. :)
> 
>> I think in theory it should give better results because
>> this irqloop runs in secondary(linux) domain when started, right? An
>> increasing preemptibility of the linux kernel should be better for the
>> irqloop-task.
> 
> The Linux kernel is already fully preemptible by Xenomai once you
> applied the I-pipe patch. Therefore, you are free to pick the Linux
> preemption strategy according to your *Linux* load, independent of what
> the real-time part needs.
> 
> Unless you have interactive Linux programs running, I can suggest to
> pick PREEMPT_VOLUNTARILY or even PREEMPT_NONE, specifically on low-end
> hardware.
> 
>> All tests were made under heavy I/O and CPU load by means of "dd",
>> "pingflood" and "cpuburn". Nevertheless, I can only achieve worse
>> results when activating CONFIG_PREEMPT.
> 
> Do you have CONFIG_DEBUG_PREEMPT set as well then? This option still as
> a small but measurable impact on Xenomai due to micro-dependencies that
> as scheduled to be removed in the near future.

Hmm, I should have better said "tiny". This experience is based on
I-pipe tracer observations, and I guess you don't have that thing on,
have you?

How much is the difference? How long did you measure to exclude noise.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Markus Franke]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

> Hmm, I should have better said "tiny". This experience is based on
> I-pipe tracer observations, and I guess you don't have that thing on,
> have you?

I have set:

---snip---
# CONFIG_IPIPE_DEBUG is not set
CONFIG_IPIPE_TRACE_ENABLE_VALUE=0
---snap---

in both kernels (either with preempt or without).

> How much is the difference? How long did you measure to exclude noise.

The difference is about 1 us (in average and worst case) when running
irqloop in IRQ handler mode.

I have been running the measurements for 1 minute under heavy load with
an interrupt frequency of 1 ms. This gives 60000 samples. Is this too
less in order to exclude noise?


Regards,
Markus Franke

[-- Attachment #2: Markus.Franke.vcf --]
[-- Type: text/x-vcard, Size: 245 bytes --]

begin:vcard
fn:Markus Franke
n:Franke;Markus
adr;quoted-printable:;;Vettersstra=C3=9Fe 64/722;Chemnitz;Saxony;09126;Germany
email;internet:Markus.Franke@domain.hid
x-mozilla-html:FALSE
url:http://www.tu-chemnitz.de/~franm
version:2.1
end:vcard

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]

Markus Franke wrote:
>> Hmm, I should have better said "tiny". This experience is based on
>> I-pipe tracer observations, and I guess you don't have that thing on,
>> have you?
> 
> I have set:
> 
> ---snip---
> # CONFIG_IPIPE_DEBUG is not set
> CONFIG_IPIPE_TRACE_ENABLE_VALUE=0
> ---snap---
> 
> in both kernels (either with preempt or without).
> 
>> How much is the difference? How long did you measure to exclude noise.
> 
> The difference is about 1 us (in average and worst case) when running
> irqloop in IRQ handler mode.
> 
> I have been running the measurements for 1 minute under heavy load with
> an interrupt frequency of 1 ms. This gives 60000 samples. Is this too
> less in order to exclude noise?

1 us is definitely in the noise region even on today's high-end PCs,
specifically when running the test that short. 1/2 hour, or better
several hours are required to gain reliable worst-case numbers.

Regarding the average, your test duration is likely enough. It indicates
that CONFIG_PREEMPT worsens the cache hit-rate slightly.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Markus Franke]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

Jan Kiszka wrote:
> Markus Franke wrote:
>>I am running some latency tests with irqbench/irqloop. I am wondering
>>whether it would be possible to achieve better results when activating
>>CONFIG_PREEMPT and CONFIG_PREEMPT_VOLUNTARILY during the kernel
>>configuration and running irqloop in User Mode over this kernel?
> 
> 
> Nope.
> 
> 
>>Does it make any sense?
> 
> 
> Nope. :)
> The Linux kernel is already fully preemptible by Xenomai once you
> applied the I-pipe patch. Therefore, you are free to pick the Linux
> preemption strategy according to your *Linux* load, independent of what
> the real-time part needs.

Well, "Native-API-Tour.pdf" states that everytime a mode switch from
Primary Domain to Secondary Domain is made (e.g. Linux system call like
ioctl()), Xenomai can ease from the
"continuous trend of improvements of Linux 2.6 regarding
preemptability". So CONFIG_PREEMPT should have an impact on measurements
with Xenomai, maybe not with irqbench/irqloop. The problem here is that
we don't have a mode switch, right? When running irqloop as User-Task it
simply runs in Secondary Mode, right?

> Do you have CONFIG_DEBUG_PREEMPT set as well then? This option still as
> a small but measurable impact on Xenomai due to micro-dependencies that
> as scheduled to be removed in the near future.

I cannot see anything like

CONFIG_DEBUG_PREEMPT=y
or
#CONFIG_DEBUG_PREEMPT is not set

in my config-file. The parameter simply doesn't exist.

Markus

[-- Attachment #2: Markus.Franke.vcf --]
[-- Type: text/x-vcard, Size: 245 bytes --]

begin:vcard
fn:Markus Franke
n:Franke;Markus
adr;quoted-printable:;;Vettersstra=C3=9Fe 64/722;Chemnitz;Saxony;09126;Germany
email;internet:Markus.Franke@domain.hid
x-mozilla-html:FALSE
url:http://www.tu-chemnitz.de/~franm
version:2.1
end:vcard

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Jan Kiszka]

Markus Franke wrote:
> Jan Kiszka wrote:
>> Markus Franke wrote:
>>> I am running some latency tests with irqbench/irqloop. I am wondering
>>> whether it would be possible to achieve better results when activating
>>> CONFIG_PREEMPT and CONFIG_PREEMPT_VOLUNTARILY during the kernel
>>> configuration and running irqloop in User Mode over this kernel?
>>
>> Nope.
>>
>>
>>> Does it make any sense?
>>
>> Nope. :)
>> The Linux kernel is already fully preemptible by Xenomai once you
>> applied the I-pipe patch. Therefore, you are free to pick the Linux
>> preemption strategy according to your *Linux* load, independent of what
>> the real-time part needs.
> 
> Well, "Native-API-Tour.pdf" states that everytime a mode switch from
> Primary Domain to Secondary Domain is made (e.g. Linux system call like
> ioctl()), Xenomai can ease from the
> "continuous trend of improvements of Linux 2.6 regarding
> preemptability". So CONFIG_PREEMPT should have an impact on measurements
> with Xenomai, maybe not with irqbench/irqloop. The problem here is that
> we don't have a mode switch, right? When running irqloop as User-Task it
> simply runs in Secondary Mode, right?

Of course, irqloop runs in *primary* mode to be able to handle the
events deterministically. So it is not directly affected by CONFIG_PREEMPT.

Yes, if you have an RT thread that issues syscalls and wishes to have
them handled as fast as possible, CONFIG_PREEMPT should be enabled (and
CONFIG_XENO_OPT_RPIDISABLE should remain off, maybe you even want to
consider CONFIG_XENO_OPT_ISHIELD then). Such RT application designs are
tricky to get correct and deterministic, so it's often better to not
rely on these properties and rather seek a clear separation of pure RT
threads on the one side and Linux syscall issuing non-RT or RT
borderline threads (low-prio RT threads that are being switched back and
forth between primary and secondary mode) on the other.

> 
>> Do you have CONFIG_DEBUG_PREEMPT set as well then? This option still as
>> a small but measurable impact on Xenomai due to micro-dependencies that
>> as scheduled to be removed in the near future.
> 
> I cannot see anything like
> 
> CONFIG_DEBUG_PREEMPT=y
> or
> #CONFIG_DEBUG_PREEMPT is not set
> 
> in my config-file. The parameter simply doesn't exist.

Shadowed by CONFIG_DEBUG=n likely.

Jan

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Markus Franke]

Jan Kiszka wrote:
> Of course, irqloop runs in *primary* mode to be able to handle the
> events deterministically. So it is not directly affected by CONFIG_PREEMPT.

If I start irqloop in User-Mode, a thread is simply created via Linux
system call pthread_create() which interacts with the
xeno_irqbench-driver via ioctl(). But then I don't understand why
irqloop is running in Primary (Xenomai) Mode? Because of the interaction
with the RTDM-based driver?
I am just wondering because I can't see something like rt_task_create().

> Yes, if you have an RT thread that issues syscalls and wishes to have
> them handled as fast as possible, CONFIG_PREEMPT should be enabled (and
> CONFIG_XENO_OPT_RPIDISABLE should remain off, maybe you even want to
> consider CONFIG_XENO_OPT_ISHIELD then). Such RT application designs are
> tricky to get correct and deterministic, so it's often better to not
> rely on these properties and rather seek a clear separation of pure RT
> threads on the one side and Linux syscall issuing non-RT or RT
> borderline threads (low-prio RT threads that are being switched back and
> forth between primary and secondary mode) on the other.

Ok I understand. So native kernel preemption should only provide better
results if you have realtime-threads issuing linux systemcalls which is
not really convenient.

> Shadowed by CONFIG_DEBUG=n likely.

probably by CONFIG_DEBUG_KERNEL=n

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Dmitry Adamushko]

On 14/02/07, Markus Franke <Markus.Franke@domain.hid> wrote:
> Jan Kiszka wrote:
> > Of course, irqloop runs in *primary* mode to be able to handle the
> > events deterministically. So it is not directly affected by CONFIG_PREEMPT.
>
> If I start irqloop in User-Mode, a thread is simply created via Linux
> system call pthread_create() which interacts with the
> xeno_irqbench-driver via ioctl(). But then I don't understand why
> irqloop is running in Primary (Xenomai) Mode? Because of the interaction
> with the RTDM-based driver?
> I am just wondering because I can't see something like rt_task_create().

There is life beyond the native skin. e.g. the posix one.

After all, Xenomai is all about skins (layers implementing various RT
API interfaces).

So pthread_create() as well as ioctl() you may see in irqloop are
implemented ("shadowed") by the posix skin.


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Markus Franke]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

Dmitry Adamushko wrote:
> On 14/02/07, Markus Franke <Markus.Franke@domain.hid> wrote:
> 
>> Jan Kiszka wrote:
>> > Of course, irqloop runs in *primary* mode to be able to handle the
>> > events deterministically. So it is not directly affected by
>> CONFIG_PREEMPT.
>>
>> If I start irqloop in User-Mode, a thread is simply created via Linux
>> system call pthread_create() which interacts with the
>> xeno_irqbench-driver via ioctl(). But then I don't understand why
>> irqloop is running in Primary (Xenomai) Mode? Because of the interaction
>> with the RTDM-based driver?
>> I am just wondering because I can't see something like rt_task_create().
> 
> 
> There is life beyond the native skin. e.g. the posix one.

Thank you very much for this hint. :-)
Now I got it.

> After all, Xenomai is all about skins (layers implementing various RT
> API interfaces).
> 
> So pthread_create() as well as ioctl() you may see in irqloop are
> implemented ("shadowed") by the posix skin.

Actually I should know this.

Regards,
Markus


[-- Attachment #2: Markus.Franke.vcf --]
[-- Type: text/x-vcard, Size: 245 bytes --]

begin:vcard
fn:Markus Franke
n:Franke;Markus
adr;quoted-printable:;;Vettersstra=C3=9Fe 64/722;Chemnitz;Saxony;09126;Germany
email;internet:Markus.Franke@domain.hid
x-mozilla-html:FALSE
url:http://www.tu-chemnitz.de/~franm
version:2.1
end:vcard

Re: [Xenomai-help] CONFIG_PREEMPT & irqbench

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

Markus Franke wrote:
> Jan Kiszka wrote:
>> Of course, irqloop runs in *primary* mode to be able to handle the
>> events deterministically. So it is not directly affected by CONFIG_PREEMPT.
> 
> If I start irqloop in User-Mode, a thread is simply created via Linux
> system call pthread_create() which interacts with the
> xeno_irqbench-driver via ioctl(). But then I don't understand why
> irqloop is running in Primary (Xenomai) Mode? Because of the interaction
> with the RTDM-based driver?
> I am just wondering because I can't see something like rt_task_create().

<see Dmitry's reply>

> 
>> Yes, if you have an RT thread that issues syscalls and wishes to have
>> them handled as fast as possible, CONFIG_PREEMPT should be enabled (and
>> CONFIG_XENO_OPT_RPIDISABLE should remain off, maybe you even want to
>> consider CONFIG_XENO_OPT_ISHIELD then). Such RT application designs are
>> tricky to get correct and deterministic, so it's often better to not
>> rely on these properties and rather seek a clear separation of pure RT
>> threads on the one side and Linux syscall issuing non-RT or RT
>> borderline threads (low-prio RT threads that are being switched back and
>> forth between primary and secondary mode) on the other.
> 
> Ok I understand. So native kernel preemption should only provide better
> results if you have realtime-threads issuing linux systemcalls which is
> not really convenient.

It is for sure convenient. But it doesn't take the burden from you to
understand what is happening underneath. That's my point.

> 
>> Shadowed by CONFIG_DEBUG=n likely.
> 
> probably by CONFIG_DEBUG_KERNEL=n

Yep.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]