beagleboard RT problem

beagleboard RT problem

[Maksym Parkachov]

Hi Everyone,

I'm trying to get PREEMPT RT kernel running on beagleboard (ARM).
After resolving many issues with the patch on the Angstrom stable distribution,
I've got the RT kernel running.

root@beagleboard:~/rt-tests# uname -a
Linux beagleboard 2.6.29.6-rt24-omap1 #1 PREEMPT RT Sun Jun 13
17:51:45 CEST 2010 armv7l unknown

So, it seems to be fine. But trying to run the Cyclictest tells me
that RT is not working.

0% - load
root@beagleboard:~/rt-tests# ./cyclictest -t1 -p 80 -n -i 10000 -l 10000 -q
T: 0 ( 2352) P:80 I:10000 C:  10000 Min:     15 Act:  203 Avg:  195 Max:     365

100% - load
root@beagleboard:~/rt-tests# ./cyclictest -t1 -p 80 -n -i 10000 -l 10000 -q
T: 0 ( 2348) P:80 I:10000 C:  10000 Min:     39 Act:48697 Avg: 1584 Max:   48727

I'm bit lost in how to debug the problem. I checked the /proc/config.gz

root@beagleboard:~/rt-tests# zcat /proc/config.gz | grep PREEMPT
CONFIG_PREEMPT_RCU=y
# CONFIG_PREEMPT_RCU_TRACE is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_DEBUG_PREEMPT=y
# CONFIG_PREEMPT_TRACER is not set

I would really appreciate some suggestions on how to identify the problem.

Regards,
Maksym Parkachov.

Re: beagleboard RT problem

[Sujit K M]

> CONFIG_PREEMPT_RCU=y
> # CONFIG_PREEMPT_RCU_TRACE is not set

Why is RCU being set to y?

> # CONFIG_PREEMPT_NONE is not set
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT_DESKTOP is not set

Fine.

> CONFIG_PREEMPT_RT=y

This is correct.

> CONFIG_PREEMPT=y
> CONFIG_PREEMPT_SOFTIRQS=y
> CONFIG_PREEMPT_HARDIRQS=y

I think this part is susceptible.

Thanks,
Sujit

Re: beagleboard RT problem

[Maksym Parkachov]

Sujit,

thanks for suggestions. I'll try to build the kernel with these
options switched off.

But, generally, is there way to trace what happens in kernel,
something in /proc or /sys ?
Do I need to switch tracing in configuration ?

Thanks,
Maksym.

On 14 June 2010 11:44, Sujit K M <sjt.kar@gmail.com> wrote:
>> CONFIG_PREEMPT_RCU=y
>> # CONFIG_PREEMPT_RCU_TRACE is not set
>
> Why is RCU being set to y?
>
>> # CONFIG_PREEMPT_NONE is not set
>> # CONFIG_PREEMPT_VOLUNTARY is not set
>> # CONFIG_PREEMPT_DESKTOP is not set
>
> Fine.
>
>> CONFIG_PREEMPT_RT=y
>
> This is correct.
>
>> CONFIG_PREEMPT=y
>> CONFIG_PREEMPT_SOFTIRQS=y
>> CONFIG_PREEMPT_HARDIRQS=y
>
> I think this part is susceptible.
>
> Thanks,
> Sujit
>

Re: beagleboard RT problem

[John Kacur]

On Mon, 14 Jun 2010, Sujit K M wrote:

> > CONFIG_PREEMPT_RCU=y
> > # CONFIG_PREEMPT_RCU_TRACE is not set
> 
> Why is RCU being set to y?

It should be fine to set it to y

> 
> > # CONFIG_PREEMPT_NONE is not set
> > # CONFIG_PREEMPT_VOLUNTARY is not set
> > # CONFIG_PREEMPT_DESKTOP is not set
> 
> Fine.
> 
> > CONFIG_PREEMPT_RT=y
> 
> This is correct.
> 
> > CONFIG_PREEMPT=y
> > CONFIG_PREEMPT_SOFTIRQS=y
> > CONFIG_PREEMPT_HARDIRQS=y
> 
> I think this part is susceptible.

Suceptible to what?
Those setting are all correct.
In fact, most are automatically set for you  after
choosing CONFIG_PREEMPT_RT=y

His problem is not with the config.

Re: beagleboard RT problem

[Sujit K M]

On Mon, Jun 14, 2010 at 4:35 PM, John Kacur <jkacur@redhat.com> wrote:
>
>
> On Mon, 14 Jun 2010, Sujit K M wrote:
>
>> > CONFIG_PREEMPT_RCU=y
>> > # CONFIG_PREEMPT_RCU_TRACE is not set
>>
>> Why is RCU being set to y?
>
> It should be fine to set it to y

Does Angstrom stable distribution need RCU?

>
>>
>> > # CONFIG_PREEMPT_NONE is not set
>> > # CONFIG_PREEMPT_VOLUNTARY is not set
>> > # CONFIG_PREEMPT_DESKTOP is not set
>>
>> Fine.
>>
>> > CONFIG_PREEMPT_RT=y
>>
>> This is correct.
>>
>> > CONFIG_PREEMPT=y
>> > CONFIG_PREEMPT_SOFTIRQS=y
>> > CONFIG_PREEMPT_HARDIRQS=y
>>
>> I think this part is susceptible.
>
> Suceptible to what?
> Those setting are all correct.
> In fact, most are automatically set for you  after
> choosing CONFIG_PREEMPT_RT=y

I thought in any Real Time system there can be only SoftIRQ's or
HardIRQ's. Also CONFIG_PREEMPT is an Non RT design.

>
> His problem is not with the config.
>

Above I think It is certainly an Config problem.

Re: beagleboard RT problem

[Sujit K M]

> But, generally, is there way to trace what happens in kernel,
> something in /proc or /sys ?
> Do I need to switch tracing in configuration ?

I think you should look at kernel module level of Tracing. You could
have tracing with something like cyclic-test.
Also some tracing with options like # CONFIG_PREEMPT_RCU_TRACE is not set.
>
> Thanks,
> Maksym.
>
> On 14 June 2010 11:44, Sujit K M <sjt.kar@gmail.com> wrote:
>>> CONFIG_PREEMPT_RCU=y
>>> # CONFIG_PREEMPT_RCU_TRACE is not set
>>
>> Why is RCU being set to y?
>>
>>> # CONFIG_PREEMPT_NONE is not set
>>> # CONFIG_PREEMPT_VOLUNTARY is not set
>>> # CONFIG_PREEMPT_DESKTOP is not set
>>
>> Fine.
>>
>>> CONFIG_PREEMPT_RT=y
>>
>> This is correct.
>>
>>> CONFIG_PREEMPT=y
>>> CONFIG_PREEMPT_SOFTIRQS=y
>>> CONFIG_PREEMPT_HARDIRQS=y
>>
>> I think this part is susceptible.
>>
>> Thanks,
>> Sujit
>>
>



-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)

Re: beagleboard RT problem

[John Kacur]

On Mon, 14 Jun 2010, Sujit K M wrote:

> On Mon, Jun 14, 2010 at 4:35 PM, John Kacur <jkacur@redhat.com> wrote:
> >
> >
> > On Mon, 14 Jun 2010, Sujit K M wrote:
> >
> >> > CONFIG_PREEMPT_RCU=y
> >> > # CONFIG_PREEMPT_RCU_TRACE is not set
> >>
> >> Why is RCU being set to y?
> >
> > It should be fine to set it to y
> 
> Does Angstrom stable distribution need RCU?
> 
> >
> >>
> >> > # CONFIG_PREEMPT_NONE is not set
> >> > # CONFIG_PREEMPT_VOLUNTARY is not set
> >> > # CONFIG_PREEMPT_DESKTOP is not set
> >>
> >> Fine.
> >>
> >> > CONFIG_PREEMPT_RT=y
> >>
> >> This is correct.
> >>
> >> > CONFIG_PREEMPT=y
> >> > CONFIG_PREEMPT_SOFTIRQS=y
> >> > CONFIG_PREEMPT_HARDIRQS=y
> >>
> >> I think this part is susceptible.
> >
> > Suceptible to what?
> > Those setting are all correct.
> > In fact, most are automatically set for you  after
> > choosing CONFIG_PREEMPT_RT=y
> 
> I thought in any Real Time system there can be only SoftIRQ's or
> HardIRQ's. Also CONFIG_PREEMPT is an Non RT design.
> 
> >
> > His problem is not with the config.
> >
> 
> Above I think It is certainly an Config problem.

I'm sorry, but that is simply incorrect.
Please have a look at kernel/Kconfig.preempt

config PREEMPT_RT
	bool "Complete Preemption (Real-Time)"
	select PREEMPT_SOFTIRQS
	select PREEMPT_HARDIRQS
	select PREEMPT_RCU
	select RT_MUTEXES

This means that if you choose PREEMPT_RT
then all those other options are selected automatically for you.

config PREEMPT
	bool
	default y
	depends on PREEMPT_DESKTOP || PREEMPT_RT

This means that the symbol PREEMPT can only be enabled
if PREEMPT_RT (or PREEMPT_DESKTOP instead) are set to y

There is nothing wrong with his config.

Re: beagleboard RT problem

[Sven-Thorsten Dietrich]

On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
> 
> On Mon, 14 Jun 2010, Sujit K M wrote:
> 
> > > CONFIG_PREEMPT_RCU=y
%<
> His problem is not with the config.
> --

Nope - the config is just fine.

Turn on the lock validator and
recompile the Kernel.

Then repeat the test that produced the latency spike.

If that doesn't produce any locking issues,
turn on latency tracing and repeat.

> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Maksym Parkachov]

Hi folks,

thanks for all suggestions.
I'll try with lock validation and see if I could come with more details.

Thanks,
Maksym.

On 14 June 2010 13:27, Sven-Thorsten Dietrich
<thebigcorporation@gmail.com> wrote:
> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
>>
>> On Mon, 14 Jun 2010, Sujit K M wrote:
>>
>> > > CONFIG_PREEMPT_RCU=y
> %<
>> His problem is not with the config.
>> --
>
> Nope - the config is just fine.
>
> Turn on the lock validator and
> recompile the Kernel.
>
> Then repeat the test that produced the latency spike.
>
> If that doesn't produce any locking issues,
> turn on latency tracing and repeat.
>
>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Maksym Parkachov]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

Folks,

it's me again with beagleboard RT problem.

I recompiled the kernel and got some statistics on locks, but I'm not
sure how to interpret it.  Searching on google didn't help, probably,
not asking right question.

Here are stats from /proc after running cyclictest.

If you could take a look at it, it would be great.

Thanks,
Maksym.

On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> Hi folks,
>
> thanks for all suggestions.
> I'll try with lock validation and see if I could come with more details.
>
> Thanks,
> Maksym.
>
> On 14 June 2010 13:27, Sven-Thorsten Dietrich
> <thebigcorporation@gmail.com> wrote:
>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
>>>
>>> On Mon, 14 Jun 2010, Sujit K M wrote:
>>>
>>> > > CONFIG_PREEMPT_RCU=y
>> %<
>>> His problem is not with the config.
>>> --
>>
>> Nope - the config is just fine.
>>
>> Turn on the lock validator and
>> recompile the Kernel.
>>
>> Then repeat the test that produced the latency spike.
>>
>> If that doesn't produce any locking issues,
>> turn on latency tracing and repeat.
>>
>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>

[-- Attachment #2: lock_stat.gz --]
[-- Type: application/x-gzip, Size: 3279 bytes --]

[-- Attachment #3: lockdep_stats.gz --]
[-- Type: application/x-gzip, Size: 341 bytes --]

[-- Attachment #4: lockdep.gz --]
[-- Type: application/x-gzip, Size: 4147 bytes --]

Re: beagleboard RT problem

[Sujit K M]

Searching for "kernel tuning in linux" I found some relevant but not
upto date articles.

http://www.linux.com/archive/feature/146599
http://www.linuxforums.org/articles/linux-performance-tuning_107.html

By the way I found something funny with your "lock_stat" gzip file. It
had the following warning.

"lock_stat version 0.3
*WARNING* lock debugging disabled!! - possibly due to a lockdep warning"

Also I think you could run dmesg to see the actvity for some clarity.

On Sun, Jun 20, 2010 at 9:13 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> Folks,
>
> it's me again with beagleboard RT problem.
>
> I recompiled the kernel and got some statistics on locks, but I'm not
> sure how to interpret it.  Searching on google didn't help, probably,
> not asking right question.
>
> Here are stats from /proc after running cyclictest.
>
> If you could take a look at it, it would be great.
>
> Thanks,
> Maksym.
>
> On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
>> Hi folks,
>>
>> thanks for all suggestions.
>> I'll try with lock validation and see if I could come with more details.
>>
>> Thanks,
>> Maksym.
>>
>> On 14 June 2010 13:27, Sven-Thorsten Dietrich
>> <thebigcorporation@gmail.com> wrote:
>>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
>>>>
>>>> On Mon, 14 Jun 2010, Sujit K M wrote:
>>>>
>>>> > > CONFIG_PREEMPT_RCU=y
>>> %<
>>>> His problem is not with the config.
>>>> --
>>>
>>> Nope - the config is just fine.
>>>
>>> Turn on the lock validator and
>>> recompile the Kernel.
>>>
>>> Then repeat the test that produced the latency spike.
>>>
>>> If that doesn't produce any locking issues,
>>> turn on latency tracing and repeat.
>>>
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>
>



-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Maksym Parkachov]

Sujit,

thanks for the links, though they mostly have to do with application
performance. I don't have any problem with app performance as there is
no apps running yet :)

This is embedded system. There is no modules loaded, there is no
network, there is no processes running beside busybox. Clean, fast,
just real-time is not working :)

I did check the dmesg, and unfortunately there is nothing unusual. No
new messages after running cyclictest.

I'll try to enable all kernel debugging options, but I'm not sure if
it helps, I don't really know what to look for in the messages or in
statistics.

Cheers,
Maksym.

On 21 June 2010 12:34, Sujit K M <sjt.kar@gmail.com> wrote:
> Searching for "kernel tuning in linux" I found some relevant but not
> upto date articles.
>
> http://www.linux.com/archive/feature/146599
> http://www.linuxforums.org/articles/linux-performance-tuning_107.html
>
> By the way I found something funny with your "lock_stat" gzip file. It
> had the following warning.
>
> "lock_stat version 0.3
> *WARNING* lock debugging disabled!! - possibly due to a lockdep warning"
>
> Also I think you could run dmesg to see the actvity for some clarity.
>
> On Sun, Jun 20, 2010 at 9:13 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
>> Folks,
>>
>> it's me again with beagleboard RT problem.
>>
>> I recompiled the kernel and got some statistics on locks, but I'm not
>> sure how to interpret it.  Searching on google didn't help, probably,
>> not asking right question.
>>
>> Here are stats from /proc after running cyclictest.
>>
>> If you could take a look at it, it would be great.
>>
>> Thanks,
>> Maksym.
>>
>> On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
>>> Hi folks,
>>>
>>> thanks for all suggestions.
>>> I'll try with lock validation and see if I could come with more details.
>>>
>>> Thanks,
>>> Maksym.
>>>
>>> On 14 June 2010 13:27, Sven-Thorsten Dietrich
>>> <thebigcorporation@gmail.com> wrote:
>>>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
>>>>>
>>>>> On Mon, 14 Jun 2010, Sujit K M wrote:
>>>>>
>>>>> > > CONFIG_PREEMPT_RCU=y
>>>> %<
>>>>> His problem is not with the config.
>>>>> --
>>>>
>>>> Nope - the config is just fine.
>>>>
>>>> Turn on the lock validator and
>>>> recompile the Kernel.
>>>>
>>>> Then repeat the test that produced the latency spike.
>>>>
>>>> If that doesn't produce any locking issues,
>>>> turn on latency tracing and repeat.
>>>>
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>>
>>>
>>
>
>
>
> --
> -- Sujit K M
>
> blog(http://kmsujit.blogspot.com/)
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Sujit K M]

I think there have been effort before on BeagleBoard for the Linux
kernel below is an link.
http://elinux.org/BeagleBoard#Linux_kernel

Try and do the entire build as per this or If you are looking for angstrom
http://beagleboard.org/project/angstrom/

I think you would have hit these in your efforts, But any way checking.

On Tue, Jun 22, 2010 at 1:20 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> Sujit,
>
> thanks for the links, though they mostly have to do with application
> performance. I don't have any problem with app performance as there is
> no apps running yet :)
>
> This is embedded system. There is no modules loaded, there is no
> network, there is no processes running beside busybox. Clean, fast,
> just real-time is not working :)
>
> I did check the dmesg, and unfortunately there is nothing unusual. No
> new messages after running cyclictest.
>
> I'll try to enable all kernel debugging options, but I'm not sure if
> it helps, I don't really know what to look for in the messages or in
> statistics.
>
> Cheers,
> Maksym.
>
> On 21 June 2010 12:34, Sujit K M <sjt.kar@gmail.com> wrote:
>> Searching for "kernel tuning in linux" I found some relevant but not
>> upto date articles.
>>
>> http://www.linux.com/archive/feature/146599
>> http://www.linuxforums.org/articles/linux-performance-tuning_107.html
>>
>> By the way I found something funny with your "lock_stat" gzip file. It
>> had the following warning.
>>
>> "lock_stat version 0.3
>> *WARNING* lock debugging disabled!! - possibly due to a lockdep warning"
>>
>> Also I think you could run dmesg to see the actvity for some clarity.
>>
>> On Sun, Jun 20, 2010 at 9:13 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
>>> Folks,
>>>
>>> it's me again with beagleboard RT problem.
>>>
>>> I recompiled the kernel and got some statistics on locks, but I'm not
>>> sure how to interpret it.  Searching on google didn't help, probably,
>>> not asking right question.
>>>
>>> Here are stats from /proc after running cyclictest.
>>>
>>> If you could take a look at it, it would be great.
>>>
>>> Thanks,
>>> Maksym.
>>>
>>> On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
>>>> Hi folks,
>>>>
>>>> thanks for all suggestions.
>>>> I'll try with lock validation and see if I could come with more details.
>>>>
>>>> Thanks,
>>>> Maksym.
>>>>
>>>> On 14 June 2010 13:27, Sven-Thorsten Dietrich
>>>> <thebigcorporation@gmail.com> wrote:
>>>>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
>>>>>>
>>>>>> On Mon, 14 Jun 2010, Sujit K M wrote:
>>>>>>
>>>>>> > > CONFIG_PREEMPT_RCU=y
>>>>> %<
>>>>>> His problem is not with the config.
>>>>>> --
>>>>>
>>>>> Nope - the config is just fine.
>>>>>
>>>>> Turn on the lock validator and
>>>>> recompile the Kernel.
>>>>>
>>>>> Then repeat the test that produced the latency spike.
>>>>>
>>>>> If that doesn't produce any locking issues,
>>>>> turn on latency tracing and repeat.
>>>>>
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> -- Sujit K M
>>
>> blog(http://kmsujit.blogspot.com/)
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Sven-Thorsten Dietrich]

On Tue, 2010-06-22 at 09:50 +0200, Maksym Parkachov wrote:
> Sujit,
> 
> thanks for the links, though they mostly have to do with application
> performance. I don't have any problem with app performance as there is
> no apps running yet :)

cyclictest is an app, and that is what you started this thread with.

you reported high latencies when running cyclictest, and you were asked
to turn on lockdep (and its assumed you went back and re-ran cyclic
test)


> 
> This is embedded system. There is no modules loaded, there is no
> network, there is no processes running beside busybox. Clean, fast,
> just real-time is not working :)
> 

> I did check the dmesg, and unfortunately there is nothing unusual. No
> new messages after running cyclictest.
> 
it does appear from an earlier post, that lock dep triggered on
something but you never reported any traces.


> I'll try to enable all kernel debugging options, but I'm not sure if
> it helps, I don't really know what to look for in the messages or in
> statistics.
> 


So assuming, that the lockdependency checker and lock validator aren't
in fact reporting locking problems, then you should be able to capture
your latency using the latency tracer.

This should report a nice stack dump for the path that is causing
cyclictest delays.  Take a look over the cyclictest parameters, or
forget about cyclictest altogether and run your app with latency
tracing.

Regards,

Sven


> Cheers,
> Maksym.
> 
> On 21 June 2010 12:34, Sujit K M <sjt.kar@gmail.com> wrote:
> > Searching for "kernel tuning in linux" I found some relevant but not
> > upto date articles.
> >
> > http://www.linux.com/archive/feature/146599
> > http://www.linuxforums.org/articles/linux-performance-tuning_107.html
> >
> > By the way I found something funny with your "lock_stat" gzip file. It
> > had the following warning.
> >
> > "lock_stat version 0.3
> > *WARNING* lock debugging disabled!! - possibly due to a lockdep warning"
> >
> > Also I think you could run dmesg to see the actvity for some clarity.
> >
> > On Sun, Jun 20, 2010 at 9:13 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> >> Folks,
> >>
> >> it's me again with beagleboard RT problem.
> >>
> >> I recompiled the kernel and got some statistics on locks, but I'm not
> >> sure how to interpret it.  Searching on google didn't help, probably,
> >> not asking right question.
> >>
> >> Here are stats from /proc after running cyclictest.
> >>
> >> If you could take a look at it, it would be great.
> >>
> >> Thanks,
> >> Maksym.
> >>
> >> On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> >>> Hi folks,
> >>>
> >>> thanks for all suggestions.
> >>> I'll try with lock validation and see if I could come with more details.
> >>>
> >>> Thanks,
> >>> Maksym.
> >>>
> >>> On 14 June 2010 13:27, Sven-Thorsten Dietrich
> >>> <thebigcorporation@gmail.com> wrote:
> >>>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
> >>>>>
> >>>>> On Mon, 14 Jun 2010, Sujit K M wrote:
> >>>>>
> >>>>> > > CONFIG_PREEMPT_RCU=y
> >>>> %<
> >>>>> His problem is not with the config.
> >>>>> --
> >>>>
> >>>> Nope - the config is just fine.
> >>>>
> >>>> Turn on the lock validator and
> >>>> recompile the Kernel.
> >>>>
> >>>> Then repeat the test that produced the latency spike.
> >>>>
> >>>> If that doesn't produce any locking issues,
> >>>> turn on latency tracing and repeat.
> >>>>
> >>>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > -- Sujit K M
> >
> > blog(http://kmsujit.blogspot.com/)
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: beagleboard RT problem

[Chatterjee, Amit]

Please use the following config settings for RT tests. The following option enables the timer to work in high resolution mode.

CONFIG_OMAP_MPU_TIMER=y
# CONFIG_OMAP_32K_TIMER is not set

Also increase the default timer resolution from 10 msec to 1 msec
--- arch/arm/Kconfig.orig	2010-06-23 14:28:30.000000000 +0530
+++ arch/arm/Kconfig	2010-03-12 14:58:31.000000000 +0530
@@ -1078,7 +1078,7 @@ config HZ
 	default 200 if ARCH_EBSA110 || ARCH_S3C2410
 	default OMAP_32K_TIMER_HZ if ARCH_OMAP && OMAP_32K_TIMER
 	default AT91_TIMER_HZ if ARCH_AT91
-	default 100
+	default 1000
 
 config THUMB2_KERNEL
 	bool "Compile the kernel in Thumb-2 mode"

Regards,
Amit

-----Original Message-----
From: linux-rt-users-owner@vger.kernel.org [mailto:linux-rt-users-owner@vger.kernel.org] On Behalf Of Sven-Thorsten Dietrich
Sent: Tuesday, June 22, 2010 8:17 PM
To: Maksym Parkachov
Cc: linux-rt-users@vger.kernel.org
Subject: Re: beagleboard RT problem

On Tue, 2010-06-22 at 09:50 +0200, Maksym Parkachov wrote:
> Sujit,
> 
> thanks for the links, though they mostly have to do with application
> performance. I don't have any problem with app performance as there is
> no apps running yet :)

cyclictest is an app, and that is what you started this thread with.

you reported high latencies when running cyclictest, and you were asked
to turn on lockdep (and its assumed you went back and re-ran cyclic
test)


> 
> This is embedded system. There is no modules loaded, there is no
> network, there is no processes running beside busybox. Clean, fast,
> just real-time is not working :)
> 

> I did check the dmesg, and unfortunately there is nothing unusual. No
> new messages after running cyclictest.
> 
it does appear from an earlier post, that lock dep triggered on
something but you never reported any traces.


> I'll try to enable all kernel debugging options, but I'm not sure if
> it helps, I don't really know what to look for in the messages or in
> statistics.
> 


So assuming, that the lockdependency checker and lock validator aren't
in fact reporting locking problems, then you should be able to capture
your latency using the latency tracer.

This should report a nice stack dump for the path that is causing
cyclictest delays.  Take a look over the cyclictest parameters, or
forget about cyclictest altogether and run your app with latency
tracing.

Regards,

Sven


> Cheers,
> Maksym.
> 
> On 21 June 2010 12:34, Sujit K M <sjt.kar@gmail.com> wrote:
> > Searching for "kernel tuning in linux" I found some relevant but not
> > upto date articles.
> >
> > http://www.linux.com/archive/feature/146599
> > http://www.linuxforums.org/articles/linux-performance-tuning_107.html
> >
> > By the way I found something funny with your "lock_stat" gzip file. It
> > had the following warning.
> >
> > "lock_stat version 0.3
> > *WARNING* lock debugging disabled!! - possibly due to a lockdep warning"
> >
> > Also I think you could run dmesg to see the actvity for some clarity.
> >
> > On Sun, Jun 20, 2010 at 9:13 PM, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> >> Folks,
> >>
> >> it's me again with beagleboard RT problem.
> >>
> >> I recompiled the kernel and got some statistics on locks, but I'm not
> >> sure how to interpret it.  Searching on google didn't help, probably,
> >> not asking right question.
> >>
> >> Here are stats from /proc after running cyclictest.
> >>
> >> If you could take a look at it, it would be great.
> >>
> >> Thanks,
> >> Maksym.
> >>
> >> On 15 June 2010 09:34, Maksym Parkachov <lazy.gopher@gmail.com> wrote:
> >>> Hi folks,
> >>>
> >>> thanks for all suggestions.
> >>> I'll try with lock validation and see if I could come with more details.
> >>>
> >>> Thanks,
> >>> Maksym.
> >>>
> >>> On 14 June 2010 13:27, Sven-Thorsten Dietrich
> >>> <thebigcorporation@gmail.com> wrote:
> >>>> On Mon, 2010-06-14 at 13:05 +0200, John Kacur wrote:
> >>>>>
> >>>>> On Mon, 14 Jun 2010, Sujit K M wrote:
> >>>>>
> >>>>> > > CONFIG_PREEMPT_RCU=y
> >>>> %<
> >>>>> His problem is not with the config.
> >>>>> --
> >>>>
> >>>> Nope - the config is just fine.
> >>>>
> >>>> Turn on the lock validator and
> >>>> recompile the Kernel.
> >>>>
> >>>> Then repeat the test that produced the latency spike.
> >>>>
> >>>> If that doesn't produce any locking issues,
> >>>> turn on latency tracing and repeat.
> >>>>
> >>>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > -- Sujit K M
> >
> > blog(http://kmsujit.blogspot.com/)
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Maksym Parkachov]

Hi Amit,

thanks for the tip. I'll try the setting and will get back with the
results of my testing.

Cheers,
Maksym.

On 23 June 2010 10:51, Chatterjee, Amit <amit.chatterjee@ti.com> wrote:
> Please use the following config settings for RT tests. The following option enables the timer to work in high resolution mode.
>
> CONFIG_OMAP_MPU_TIMER=y
> # CONFIG_OMAP_32K_TIMER is not set
>
> Also increase the default timer resolution from 10 msec to 1 msec
> --- arch/arm/Kconfig.orig       2010-06-23 14:28:30.000000000 +0530
> +++ arch/arm/Kconfig    2010-03-12 14:58:31.000000000 +0530
> @@ -1078,7 +1078,7 @@ config HZ
>        default 200 if ARCH_EBSA110 || ARCH_S3C2410
>        default OMAP_32K_TIMER_HZ if ARCH_OMAP && OMAP_32K_TIMER
>        default AT91_TIMER_HZ if ARCH_AT91
> -       default 100
> +       default 1000
>
>  config THUMB2_KERNEL
>        bool "Compile the kernel in Thumb-2 mode"
>
> Regards,
> Amit
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: beagleboard RT problem

[Maksym Parkachov]

Hi folks,

it's me, back, with strange behavior of kernel on beagleboard (arm).

The kernel is the latest one with options recommended by Amit.
Linux version 2.6.33.7-rt29 (oe@server) (gcc version 4.3.3 (GCC) ) #1
PREEMPT RT

last time I made mistake of not enabling the debug correctly.
This time following was reported:

1. During boot the self-test were activated, and now kernel says
something useful:

[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat,
Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 3951 kB
[    0.000000]  per task-struct memory footprint: 2304 bytes
[    0.000000] ------------------------
[    0.000000] | Locking API testsuite:
[    0.000000] ----------------------------------------------------------------------------
[    0.000000]                                  | spin |wlock |rlock
|mutex | wsem | rsem |
[    0.000000]
--------------------------------------------------------------------------
[    0.000000]                      A-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]                  A-B-B-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]              A-B-B-C-C-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]              A-B-C-A-B-C deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]          A-B-B-C-C-D-D-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-D-D-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-C-D-A deadlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]                     double unlock:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]                   initialize held:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]                  bad unlock order:  ok  |  ok  |  ok  |
 ok  |  ok  |  ok  |
[    0.000000]
--------------------------------------------------------------------------
[    0.000000]               recursive read-lock:             |  ok  |
            |  ok  |
[    0.000000]            recursive read-lock #2:             |  ok  |
            |  ok  |
[    0.000000]             mixed read-write-lock:             |  ok  |
            |  ok  |
[    0.000000]             mixed write-read-lock:             |  ok  |
            |  ok  |
[    0.000000]
--------------------------------------------------------------------------
[    0.000000] -------------------------------------------------------
[    0.000000] Good, all  68 testcases passed! |
[    0.000000] ---------------------------------
[    0.000000] BUG: swapper/0: lock count overflow!
[    0.000000] [<c0040da8>] (unwind_backtrace+0x0/0xe8) from
[<c05778c8>] (dump_stack+0x1c/0x20)
[    0.000000] [<c05778c8>] (dump_stack+0x1c/0x20) from [<c00a8d2c>]
(rt_mutex_deadlock_account_lock+0x70/0x78)
[    0.000000] [<c00a8d2c>] (rt_mutex_deadlock_account_lock+0x70/0x78)
from [<c00a7a40>] (do_try_to_take_rt_mutex+0x138/0x17c)
[    0.000000] [<c00a7a40>] (do_try_to_take_rt_mutex+0x138/0x17c) from
[<c0579918>] (rt_spin_lock_slowlock+0xe4/0x274)
[    0.000000] [<c0579918>] (rt_spin_lock_slowlock+0xe4/0x274) from
[<c057a4c4>] (rt_spin_lock+0x58/0x98)
[    0.000000] [<c057a4c4>] (rt_spin_lock+0x58/0x98) from [<c011230c>]
(kmem_cache_alloc+0x1d0/0x224)
[    0.000000] [<c011230c>] (kmem_cache_alloc+0x1d0/0x224) from
[<c0112638>] (kmem_cache_create+0x1d4/0x4d0)
[    0.000000] [<c0112638>] (kmem_cache_create+0x1d4/0x4d0) from
[<c001d8d4>] (idr_init_cache+0x2c/0x44)
[    0.000000] [<c001d8d4>] (idr_init_cache+0x2c/0x44) from
[<c0008c18>] (start_kernel+0x210/0x340)
[    0.000000] [<c0008c18>] (start_kernel+0x210/0x340) from
[<80008034>] (0x80008034)
[    0.000000] Calibrating delay loop... 490.49 BogoMIPS (lpj=245248)

After that  kernel with swap switched off was rebuilt, and still
generating the same error. System don't really have or need swap.

2. Latency timer was activated, this is the report of latency under 100% load.

Latency Top version : v0.1
10 3751 864 n_tty_read tty_read vfs_read sys_read ret_fast_syscall
21 2048 298 do_wait sys_wait4 ret_fast_syscall
2 13589 8553 do_get_write_access journal_get_write_access
__ext3_journal_get_write_access ext3_reserve_inode_write
ext3_mark_inode_dirty ext3_dirty_inode
1 5751 5751 do_get_write_access journal_get_undo_access
__ext3_journal_get_undo_access ext3_free_blocks_sb ext3_free_blocks
ext3_clear_blocks ext3_free_d
1 6347 6347 sync_page sync_page_killable __lock_page_killable
generic_file_aio_read do_sync_read vfs_read kernel_read prepare_binprm
do_execve sys_execve
1 2264 2264 sync_page __lock_page find_lock_page filemap_fault
__do_fault handle_mm_fault do_page_fault do_translation_fault
do_DataAbort __dabt_svc __cl
1 5857110 5857110 sync_buffer __lock_buffer do_get_write_access
journal_get_write_access __ext3_journal_get_write_access
ext3_orphan_add ext3_setattr not
1 11715918 11715918 do_get_write_access journal_get_write_access
__ext3_journal_get_write_access ext3_reserve_inode_write
ext3_mark_inode_dirty ext3_dirt
1 417 417 __make_request generic_make_request submit_bio submit_bh
__block_write_full_page block_write_full_page_endio
block_write_full_page ext3_writeb

seems like something wrong on my system with ext3 layer. I will try to
setup system without journal.

If you guys have some more ideas I would really appreciate.

Cheers,
Maksym.

Re: beagleboard RT problem

[Maksym Parkachov]

Folks,

one more thing, when I run pi_stess, I got on console:

sched: RT throttling activated

After I set root to be ext2, the result is very similar:

root@beagleboard:~/rt-tests# cat /proc/latency_stats
Latency Top version : v0.1
18 2317 387 do_wait sys_wait4 ret_fast_syscall
2 864 447 n_tty_read tty_read vfs_read sys_read ret_fast_syscall
1 1043 1043 sync_page __lock_page do_read_cache_page
read_cache_page_async read_cache_page ext2_get_page ext2_readdir
vfs_readdir sys_getdents64 ret_fast
3 2115 715 sync_buffer __wait_on_buffer __bread ext2_get_inode
ext2_iget ext2_lookup do_lookup link_path_walk path_walk
do_path_lookup user_path_at vfs_f
1 3457 3457 sync_buffer __wait_on_buffer __bread ext2_get_branch
ext2_get_block do_mpage_readpage mpage_readpages ext2_readpages
__do_page_cache_readahea
1 3695 3695 sync_page __lock_page find_lock_page filemap_fault
__do_fault handle_mm_fault do_page_fault do_translation_fault
do_DataAbort __dabt_svc __cl
1 1951545 1951545 sync_page wait_on_page_bit
truncate_inode_pages_range truncate_inode_pages truncate_pagecache
vmtruncate inode_setattr ext2_setattr not
1 119 119 get_signal_to_deliver do_notify_resume work_pending

Thanks,
Maksym.