Detecting PREEMPT_RT

Detecting PREEMPT_RT

[John Morris]

Hi list,

I'm part of a project that recently added PREEMPT_RT support (as well as
Xenomai) to LinuxCNC/EMC2.

The next sub-project is a 'universal binary'.  The LCNC real-time module
will run some checks to determine what RT systems, if any, are available
on the running kernel, and then load the appropriate RT support module.

We wish to distinguish between PREEMPT_RT and non-RT kernels because
LCNC must confirm that the kernel really does have realtime
capabilities, if that's what the user expects, and print big warning
messages if not.  LCNC drives machines that weigh many tons and spins
spindles at 24k RPM, so this is important!

The RT PREEMPT HOWTO discusses checking the kernel here:

https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO#Checking_the_Kernel

One method is to use string matching against the kernel version, which
I'm a bit suspicious of.

An improvement, figure 7 suggests using string matching against the
process list in search of kernel thread names matching '[softirq-foo]'.

Figure 8 show how to check /proc/interrupts on older kernels, but 9
states this doesn't work for newer kernels, so this isn't an option.

We haven't found /proc/config.gz in any vendor-provided PREEMPT_RT
kernels, so that is not an option either.

So, is string matching against the process list the best detection
method, or is there something I've missed?

	John

Re: Detecting PREEMPT_RT

[Carsten Emde]

John,

> I'm part of a project that recently added PREEMPT_RT support (as well as
> Xenomai) to LinuxCNC/EMC2.
>
> The next sub-project is a 'universal binary'.  The LCNC real-time module
> will run some checks to determine what RT systems, if any, are available
> on the running kernel, and then load the appropriate RT support module.
>
> We wish to distinguish between PREEMPT_RT and non-RT kernels because
> LCNC must confirm that the kernel really does have realtime
> capabilities, if that's what the user expects, and print big warning
> messages if not.  LCNC drives machines that weigh many tons and spins
> spindles at 24k RPM, so this is important!
>
> The RT PREEMPT HOWTO discusses checking the kernel here:
>
> https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO#Checking_the_Kernel
>
> One method is to use string matching against the kernel version, which
> I'm a bit suspicious of.
Unfortunately, this wiki article is old and unmaintained. You're 
probably right to be a bit suspicious when checking for the "-rt" suffix 
of the kernel release (uname -r). But looking for the occurrence of 
"PREEMPT RT" in the kernel version (uname -v) should work. An 
application may use the system call uname() and check the version 
element of the utsname structure.

In addition you may want to make sure the system has high-resolution 
timers. If so, the timers in /proc/timer_list have ".resolution: 1 
nsecs". An application may use the function check_timer() from 
cyclictest for this purpose:
static int check_timer(void)
{
   struct timespec ts;

   if (clock_getres(CLOCK_MONOTONIC, &ts))
     return 1;

   return (ts.tv_sec != 0 || ts.tv_nsec != 1);
}

Hope this helps,
	-Carsten.

Re: Detecting PREEMPT_RT

[Michael Haberler]

Hi Carsten

Am 20.01.2013 um 02:43 schrieb Carsten Emde:

> Unfortunately, this wiki article is old and unmaintained. You're probably right to be a bit suspicious when checking for the "-rt" suffix of the kernel release (uname -r). But looking for the occurrence of "PREEMPT RT" in the kernel version (uname -v) should work. An application may use the system call uname() and check the version element of the utsname structure.

the utsname.version string match is fine, however:

> In addition you may want to make sure the system has high-resolution timers. If so, the timers in /proc/timer_list have ".resolution: 1 nsecs". An application may use the function check_timer() from cyclictest for this purpose:
> static int check_timer(void)
> {
>  struct timespec ts;
> 
>  if (clock_getres(CLOCK_MONOTONIC, &ts))
>    return 1;
> 
>  return (ts.tv_sec != 0 || ts.tv_nsec != 1);
> }

Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0 and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT from a vanilla kernel

- Michael

Re: Detecting PREEMPT_RT

[Carsten Emde]

Hi Michael,

>> Unfortunately, this wiki article is old and unmaintained. You're probably right to be a bit suspicious when checking for the "-rt" suffix of the kernel release (uname -r). But looking for the occurrence of "PREEMPT RT" in the kernel version (uname -v) should work. An application may use the system call uname() and check the version element of the utsname structure.
>
> the utsname.version string match is fine, however:
>
>> In addition you may want to make sure the system has high-resolution timers. If so, the timers in /proc/timer_list have ".resolution: 1 nsecs". An application may use the function check_timer() from cyclictest for this purpose:
>> static int check_timer(void)
>> {
>>   struct timespec ts;
>>
>>   if (clock_getres(CLOCK_MONOTONIC, &ts))
>>     return 1;
>>
>>   return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>> }
>
> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0 and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT from a vanilla kernel
I meant boolean AND when I said "In addition".

	-Carsten.

Re: Detecting PREEMPT_RT

[John Morris]

On 01/20/2013 11:32 AM, Carsten Emde wrote:
> Hi Michael,
> 
>>> Unfortunately, this wiki article is old and unmaintained. You're
probably right to be a bit suspicious when checking for the "-rt" suffix
of the kernel release (uname -r). But looking for the occurrence of
"PREEMPT RT" in the kernel version (uname -v) should work. An
application may use the system call uname() and check the version
element of the utsname structure.
>>
>> the utsname.version string match is fine, however:
>>
>>> In addition you may want to make sure the system has
>>> high-resolution
timers. If so, the timers in /proc/timer_list have ".resolution: 1
nsecs". An application may use the function check_timer() from
cyclictest for this purpose:
>>> static int check_timer(void)
>>> {
>>>   struct timespec ts;
>>>
>>>   if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>     return 1;
>>>
>>>   return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>> }
>>
>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
from a vanilla kernel
> I meant boolean AND when I said "In addition".

Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
separate check for timers, which we must confirm are available before
driving heavy machinery.

So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
module.  Once the module is loaded, from an init() function, run sanity
checks, including the clock_getres check above.  Is that about right?

Thanks for the tips, Carsten!

	John

Re: Detecting PREEMPT_RT

[Carsten Emde]

John,

>>>> Unfortunately, this wiki article is old and unmaintained. You're
>>>> probably right to be a bit suspicious when checking for the "-rt" suffix
>>>> of the kernel release (uname -r). But looking for the occurrence of
>>>> "PREEMPT RT" in the kernel version (uname -v) should work. An
>>>> application may use the system call uname() and check the version
>>>> element of the utsname structure.
>>> the utsname.version string match is fine, however:
>>>> In addition you may want to make sure the system has
>>>> high-resolution
>>> timers. If so, the timers in /proc/timer_list have ".resolution: 1
>>> nsecs". An application may use the function check_timer() from
>>> cyclictest for this purpose:
>>>> static int check_timer(void)
>>>> {
>>>>    struct timespec ts;
>>>>
>>>>    if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>>      return 1;
>>>>
>>>>    return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>>> }
>>>
>>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
>>> SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
>>> and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
>>> from a vanilla kernel
>> I meant boolean AND when I said "In addition".
> Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
> separate check for timers, which we must confirm are available before
> driving heavy machinery.
> So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
> module.  Once the module is loaded, from an init() function, run sanity
> checks, including the clock_getres check above.  Is that about right?
Yes. Sounds good to me.

	-Carsten.

Re: Detecting PREEMPT_RT

[John Kacur]

On Sun, Jan 20, 2013 at 9:18 PM, Carsten Emde <C.Emde@osadl.org> wrote:
> John,
>
>
>>>>> Unfortunately, this wiki article is old and unmaintained. You're
>>>>> probably right to be a bit suspicious when checking for the "-rt"
>>>>> suffix
>>>>> of the kernel release (uname -r). But looking for the occurrence of
>>>>> "PREEMPT RT" in the kernel version (uname -v) should work. An
>>>>> application may use the system call uname() and check the version
>>>>> element of the utsname structure.
>>>>
>>>> the utsname.version string match is fine, however:
>>>>>
>>>>> In addition you may want to make sure the system has
>>>>> high-resolution
>>>>
>>>> timers. If so, the timers in /proc/timer_list have ".resolution: 1
>>>> nsecs". An application may use the function check_timer() from
>>>> cyclictest for this purpose:
>>>>>
>>>>> static int check_timer(void)
>>>>> {
>>>>>    struct timespec ts;
>>>>>
>>>>>    if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>>>      return 1;
>>>>>
>>>>>    return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>>>> }
>>>>
>>>>
>>>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
>>>> SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
>>>> and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
>>>> from a vanilla kernel
>>>
>>> I meant boolean AND when I said "In addition".
>>
>> Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
>> separate check for timers, which we must confirm are available before
>> driving heavy machinery.
>> So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
>> module.  Once the module is loaded, from an init() function, run sanity
>> checks, including the clock_getres check above.  Is that about right?
>
> Yes. Sounds good to me.
>
Note we added a patch, called sysfs-realtime-entry.patch, but I'm not
sure how far back it goes, at least it's in 3.x-rt kernels.

This is the text from the patch

"Add a /sys/kernel entry to indicate that the kernel is a
realtime kernel.

Clark says that he needs this for udev rules, udev needs to evaluate
if its a PREEMPT_RT kernel a few thousand times and parsing uname
output is too slow or so.

Are there better solutions? Should it exist and return 0 on !-rt?"

To use it, you do something like
test -e /sys/kernel/realtime

If you need it in an older kernel that doesn't have it, it should port
trivially.

Thanks

Re: Detecting PREEMPT_RT

[Michael Haberler]

John,


Am 21.01.2013 um 12:36 schrieb John Kacur:

> On Sun, Jan 20, 2013 at 9:18 PM, Carsten Emde <C.Emde@osadl.org> wrote:
>> John,
>> 
>> 
>>>>>> Unfortunately, this wiki article is old and unmaintained. You're
>>>>>> probably right to be a bit suspicious when checking for the "-rt"
>>>>>> suffix
>>>>>> of the kernel release (uname -r). But looking for the occurrence of
>>>>>> "PREEMPT RT" in the kernel version (uname -v) should work. An
>>>>>> application may use the system call uname() and check the version
>>>>>> element of the utsname structure.
>>>>> 
>>>>> the utsname.version string match is fine, however:
>>>>>> 
>>>>>> In addition you may want to make sure the system has
>>>>>> high-resolution
>>>>> 
>>>>> timers. If so, the timers in /proc/timer_list have ".resolution: 1
>>>>> nsecs". An application may use the function check_timer() from
>>>>> cyclictest for this purpose:
>>>>>> 
>>>>>> static int check_timer(void)
>>>>>> {
>>>>>>   struct timespec ts;
>>>>>> 
>>>>>>   if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>>>>     return 1;
>>>>>> 
>>>>>>   return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>>>>> }
>>>>> 
>>>>> 
>>>>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
>>>>> SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
>>>>> and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
>>>>> from a vanilla kernel
>>>> 
>>>> I meant boolean AND when I said "In addition".
>>> 
>>> Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
>>> separate check for timers, which we must confirm are available before
>>> driving heavy machinery.
>>> So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
>>> module.  Once the module is loaded, from an init() function, run sanity
>>> checks, including the clock_getres check above.  Is that about right?
>> 
>> Yes. Sounds good to me.
>> 
> Note we added a patch, called sysfs-realtime-entry.patch, but I'm not
> sure how far back it goes, at least it's in 3.x-rt kernels.
> 
> This is the text from the patch
> 
> "Add a /sys/kernel entry to indicate that the kernel is a
> realtime kernel.

found this: https://bugzilla.redhat.com/show_bug.cgi?id=728551 and this:
https://github.com/clrkwllms/rt-linux/commit/274ca2cb72e013602cebb8dc142a7c69d42f92db

looks like it is being used elsewhere: http://softballinfo.net/ret/root/usr/lib/python2.6/site-packages/sos/plugins/kernel_realtime.py

> 
> Clark says that he needs this for udev rules, udev needs to evaluate
> if its a PREEMPT_RT kernel a few thousand times and parsing uname
> output is too slow or so.
> 
> Are there better solutions? Should it exist and return 0 on !-rt?"
> 
> To use it, you do something like
> test -e /sys/kernel/realtime

excellent - this works fine for me (I dont see us going back further than 3.4):

mah@atom:~$ uname -a
Linux atom 3.4.13-rt-preempt-rt22+ #2 SMP PREEMPT RT Fri Oct 19 08:56:20 CEST 2012 i686 GNU/Linux

mah@atom:~$ ls -l /sys/kernel/realtime 
-r--r--r-- 1 root root 4096 2013-01-21 12:53 /sys/kernel/realtime

mah@atom:~$ cat /sys/kernel/realtime
1

The way I read it the fingerprint is - '/sys/kernel/realtime' exists and contents is '1'

thanks a lot!

- Michael

> 
> If you need it in an older kernel that doesn't have it, it should port
> trivially.
> 
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Detecting PREEMPT_RT

[John Kacur]

On Mon, Jan 21, 2013 at 1:14 PM, Michael Haberler <mail17@mah.priv.at> wrote:
> John,
>
>
> Am 21.01.2013 um 12:36 schrieb John Kacur:
>
>> On Sun, Jan 20, 2013 at 9:18 PM, Carsten Emde <C.Emde@osadl.org> wrote:
>>> John,
>>>
>>>
>>>>>>> Unfortunately, this wiki article is old and unmaintained. You're
>>>>>>> probably right to be a bit suspicious when checking for the "-rt"
>>>>>>> suffix
>>>>>>> of the kernel release (uname -r). But looking for the occurrence of
>>>>>>> "PREEMPT RT" in the kernel version (uname -v) should work. An
>>>>>>> application may use the system call uname() and check the version
>>>>>>> element of the utsname structure.
>>>>>>
>>>>>> the utsname.version string match is fine, however:
>>>>>>>
>>>>>>> In addition you may want to make sure the system has
>>>>>>> high-resolution
>>>>>>
>>>>>> timers. If so, the timers in /proc/timer_list have ".resolution: 1
>>>>>> nsecs". An application may use the function check_timer() from
>>>>>> cyclictest for this purpose:
>>>>>>>
>>>>>>> static int check_timer(void)
>>>>>>> {
>>>>>>>   struct timespec ts;
>>>>>>>
>>>>>>>   if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>>>>>     return 1;
>>>>>>>
>>>>>>>   return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>>>>>> }
>>>>>>
>>>>>>
>>>>>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
>>>>>> SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
>>>>>> and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
>>>>>> from a vanilla kernel
>>>>>
>>>>> I meant boolean AND when I said "In addition".
>>>>
>>>> Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
>>>> separate check for timers, which we must confirm are available before
>>>> driving heavy machinery.
>>>> So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
>>>> module.  Once the module is loaded, from an init() function, run sanity
>>>> checks, including the clock_getres check above.  Is that about right?
>>>
>>> Yes. Sounds good to me.
>>>
>> Note we added a patch, called sysfs-realtime-entry.patch, but I'm not
>> sure how far back it goes, at least it's in 3.x-rt kernels.
>>
>> This is the text from the patch
>>
>> "Add a /sys/kernel entry to indicate that the kernel is a
>> realtime kernel.
>
> found this: https://bugzilla.redhat.com/show_bug.cgi?id=728551 and this:
> https://github.com/clrkwllms/rt-linux/commit/274ca2cb72e013602cebb8dc142a7c69d42f92db
>
> looks like it is being used elsewhere: http://softballinfo.net/ret/root/usr/lib/python2.6/site-packages/sos/plugins/kernel_realtime.py
>
>>
>> Clark says that he needs this for udev rules, udev needs to evaluate
>> if its a PREEMPT_RT kernel a few thousand times and parsing uname
>> output is too slow or so.
>>
>> Are there better solutions? Should it exist and return 0 on !-rt?"
>>
>> To use it, you do something like
>> test -e /sys/kernel/realtime
>
> excellent - this works fine for me (I dont see us going back further than 3.4):
>
> mah@atom:~$ uname -a
> Linux atom 3.4.13-rt-preempt-rt22+ #2 SMP PREEMPT RT Fri Oct 19 08:56:20 CEST 2012 i686 GNU/Linux
>
> mah@atom:~$ ls -l /sys/kernel/realtime
> -r--r--r-- 1 root root 4096 2013-01-21 12:53 /sys/kernel/realtime
>
> mah@atom:~$ cat /sys/kernel/realtime
> 1
>
> The way I read it the fingerprint is - '/sys/kernel/realtime' exists and contents is '1'
>
> thanks a lot!
>

No problem, now if you want to give back, please feel free to update
the wiki as well!

John

Re: Detecting PREEMPT_RT

[Michael Haberler]

Am 21.01.2013 um 13:17 schrieb John Kacur:

> On Mon, Jan 21, 2013 at 1:14 PM, Michael Haberler <mail17@mah.priv.at> wrote:
>> John,
>> 
>> 
>> Am 21.01.2013 um 12:36 schrieb John Kacur:
>> 
>>> On Sun, Jan 20, 2013 at 9:18 PM, Carsten Emde <C.Emde@osadl.org> wrote:
>>>> John,
>>>> 
>>>> 
>>>>>>>> Unfortunately, this wiki article is old and unmaintained. You're
>>>>>>>> probably right to be a bit suspicious when checking for the "-rt"
>>>>>>>> suffix
>>>>>>>> of the kernel release (uname -r). But looking for the occurrence of
>>>>>>>> "PREEMPT RT" in the kernel version (uname -v) should work. An
>>>>>>>> application may use the system call uname() and check the version
>>>>>>>> element of the utsname structure.
>>>>>>> 
>>>>>>> the utsname.version string match is fine, however:
>>>>>>>> 
>>>>>>>> In addition you may want to make sure the system has
>>>>>>>> high-resolution
>>>>>>> 
>>>>>>> timers. If so, the timers in /proc/timer_list have ".resolution: 1
>>>>>>> nsecs". An application may use the function check_timer() from
>>>>>>> cyclictest for this purpose:
>>>>>>>> 
>>>>>>>> static int check_timer(void)
>>>>>>>> {
>>>>>>>>  struct timespec ts;
>>>>>>>> 
>>>>>>>>  if (clock_getres(CLOCK_MONOTONIC, &ts))
>>>>>>>>    return 1;
>>>>>>>> 
>>>>>>>>  return (ts.tv_sec != 0 || ts.tv_nsec != 1);
>>>>>>>> }
>>>>>>> 
>>>>>>> 
>>>>>>> Just tried this on a generic kernel (2.6.32-45-generic #102-Ubuntu
>>>>>>> SMP Wed Jan 2 21:53:06 UTC 2013 i686 GNU/Linux) and I get ts.tv_sec == 0
>>>>>>> and ts.tv_nsec == 1, so it looks this cant be used to tell an RT_PREEMPT
>>>>>>> from a vanilla kernel
>>>>>> 
>>>>>> I meant boolean AND when I said "In addition".
>>>>> 
>>>>> Oh, I interpreted the timer check not as a PREEMPT_RT check, but a
>>>>> separate check for timers, which we must confirm are available before
>>>>> driving heavy machinery.
>>>>> So, if utsname.version contains "PREEMPT RT", then load the preempt-rt
>>>>> module.  Once the module is loaded, from an init() function, run sanity
>>>>> checks, including the clock_getres check above.  Is that about right?
>>>> 
>>>> Yes. Sounds good to me.
>>>> 
>>> Note we added a patch, called sysfs-realtime-entry.patch, but I'm not
>>> sure how far back it goes, at least it's in 3.x-rt kernels.
>>> 
>>> This is the text from the patch
>>> 
>>> "Add a /sys/kernel entry to indicate that the kernel is a
>>> realtime kernel.
>> 
>> found this: https://bugzilla.redhat.com/show_bug.cgi?id=728551 and this:
>> https://github.com/clrkwllms/rt-linux/commit/274ca2cb72e013602cebb8dc142a7c69d42f92db
>> 
>> looks like it is being used elsewhere: http://softballinfo.net/ret/root/usr/lib/python2.6/site-packages/sos/plugins/kernel_realtime.py
>> 
>>> 
>>> Clark says that he needs this for udev rules, udev needs to evaluate
>>> if its a PREEMPT_RT kernel a few thousand times and parsing uname
>>> output is too slow or so.
>>> 
>>> Are there better solutions? Should it exist and return 0 on !-rt?"
>>> 
>>> To use it, you do something like
>>> test -e /sys/kernel/realtime
>> 
>> excellent - this works fine for me (I dont see us going back further than 3.4):
>> 
>> mah@atom:~$ uname -a
>> Linux atom 3.4.13-rt-preempt-rt22+ #2 SMP PREEMPT RT Fri Oct 19 08:56:20 CEST 2012 i686 GNU/Linux
>> 
>> mah@atom:~$ ls -l /sys/kernel/realtime
>> -r--r--r-- 1 root root 4096 2013-01-21 12:53 /sys/kernel/realtime
>> 
>> mah@atom:~$ cat /sys/kernel/realtime
>> 1
>> 
>> The way I read it the fingerprint is - '/sys/kernel/realtime' exists and contents is '1'
>> 
>> thanks a lot!
>> 
> 
> No problem, now if you want to give back, please feel free to update
> the wiki as well!


https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO#Runtime_detection_of_an_RT-PREEMPT_Kernel

- Michael