ubiformat & linux 3.0.14-rt

ubiformat & linux 3.0.14-rt

[Tim Sander]

Hi

We are currently using the UBIFS with a preemt rt patched kernel + some local 
platform adaptions. After a firmware update of the root filesystem with 
ubiformat we see the following message on the next reboot:
sched: RT throttling activated

This message is only seen after firmware update so i suspect that the reason 
for this message lies within the ubifs/ubi layer. With the RT enabled kernel 
all interrupt run as realtime tasks. This means that if an interrupt takes too 
much cpu for some time this is noticed by the scheduler and it throttles the 
realtime tasks to 95% of available cpu time. 

This happens with or without of ubifs master for 3.0 applied from
git://git.infradead.org/~dedekind/ubifs-v3.0.git

So is there some condition which causes UBI(FS) to take more time in an 
interrupt routine if the filesystem has been updated? If yes wouldn't it be 
possible to move this work into:
ubifs_bgt0_0

Best regards
Tim

Re: ubiformat & linux 3.0.14-rt

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On Fri, 2012-01-13 at 10:06 +0100, Tim Sander wrote:
> This happens with or without of ubifs master for 3.0 applied from
> git://git.infradead.org/~dedekind/ubifs-v3.0.git
> 
> So is there some condition which causes UBI(FS) to take more time in an 
> interrupt routine if the filesystem has been updated? If yes wouldn't it be 
> possible to move this work into:
> ubifs_bgt0_0

Please the very latest patches:

http://git.infradead.org/ubifs-2.6.git

Namely, these should fix the issue:

http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aaa2d0f17624f8c2a6c52
http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146140ea4b277193ee5e5f

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubiformat & linux 3.0.14-rt

[Tim Sander]

Hi Artem

Thanks for your reply.
Am Freitag, 13. Januar 2012 schrieb Artem Bityutskiy:
> On Fri, 2012-01-13 at 10:06 +0100, Tim Sander wrote:
> > This happens with or without of ubifs master for 3.0 applied from
> > git://git.infradead.org/~dedekind/ubifs-v3.0.git
> > 
> > So is there some condition which causes UBI(FS) to take more time in an
> > interrupt routine if the filesystem has been updated? If yes wouldn't it
> > be possible to move this work into:
> > ubifs_bgt0_0
> 
> Please the very latest patches:
Mh, i'd like to. But this is with the realtime patches and i have only got 3.0 
up and running. To my knowledge this error is only visible with the preempt rt
patch applied. So there are way to many merge conflicts for me to test :-( 
sorry. I probably could give it a spin with 3.2-rt but i havent got this one 
booting.

> http://git.infradead.org/ubifs-2.6.git
> 
> Namely, these should fix the issue:
> 
> http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aaa2d0f176
> 24f8c2a6c52
> http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146140ea4b
> 277193ee5e5f
Do you think it is worthwhile checking the ubifs 3.0 backport with the above 
two cherry-picked?

Best regards
Tim

Re: ubiformat & linux 3.0.14-rt

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Mon, 2012-01-16 at 11:43 +0100, Tim Sander wrote:
> > http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aaa2d0f176
> > 24f8c2a6c52
> > http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146140ea4b
> > 277193ee5e5f
> Do you think it is worthwhile checking the ubifs 3.0 backport with the above 
> two cherry-picked?

I've pushed them to the ubifs-v3.0 back-port tree as well. But I think
they are useless - sorry, I was confused.

Also, if you say this happens when you run ubiformat - this is something
about your flash driver. Do you use NOR or NAND? Can you detect which
interrupt takes too much time? If the theory is that this is your flash
driver interrupt - can you verify it by injecting some instrumentation
to the interrupt handler?

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubiformat & linux 3.0.14-rt

[Tim Sander]

Hi Artem

Am Montag, 16. Januar 2012 schrieb Artem Bityutskiy:
> On Mon, 2012-01-16 at 11:43 +0100, Tim Sander wrote:
> > > http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aaa2d0
> > > f176 24f8c2a6c52
> > > http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146140e
> > > a4b 277193ee5e5f
> > 
> > Do you think it is worthwhile checking the ubifs 3.0 backport with the
> > above two cherry-picked?
> 
> I've pushed them to the ubifs-v3.0 back-port tree as well. But I think
> they are useless - sorry, I was confused.
Ok i was wondering that logging was causing this cind of behavoir...

> Also, if you say this happens when you run ubiformat - this is something
> about your flash driver. Do you use NOR or NAND? Can you detect which
> interrupt takes too much time? If the theory is that this is your flash
> driver interrupt - can you verify it by injecting some instrumentation
> to the interrupt handler?
This is a arm i.mx35 (pcm43) platform using the nor and nand driver. The 
mxc_nand driver is loaded later which seems to be around the same time this 
message:
"sched: RT throttling activated" comes up and the ubi on the nand gets 
attached. I will see what if i can dig s.t. up with the instrumentation and 
will report back.

Best regards
Tim

Re: ubiformat & linux 3.0.14-rt

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

On Mon, 2012-01-16 at 13:59 +0100, Tim Sander wrote:
> Hi Artem
> 
> Am Montag, 16. Januar 2012 schrieb Artem Bityutskiy:
> > On Mon, 2012-01-16 at 11:43 +0100, Tim Sander wrote:
> > > > http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aaa2d0
> > > > f176 24f8c2a6c52
> > > > http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146140e
> > > > a4b 277193ee5e5f
> > > 
> > > Do you think it is worthwhile checking the ubifs 3.0 backport with the
> > > above two cherry-picked?
> > 
> > I've pushed them to the ubifs-v3.0 back-port tree as well. But I think
> > they are useless - sorry, I was confused.
> Ok i was wondering that logging was causing this cind of behavoir...
> 
> > Also, if you say this happens when you run ubiformat - this is something
> > about your flash driver. Do you use NOR or NAND? Can you detect which
> > interrupt takes too much time? If the theory is that this is your flash
> > driver interrupt - can you verify it by injecting some instrumentation
> > to the interrupt handler?
> This is a arm i.mx35 (pcm43) platform using the nor and nand driver. The 
> mxc_nand driver is loaded later which seems to be around the same time this 
> message:
> "sched: RT throttling activated" comes up and the ubi on the nand gets 
> attached. I will see what if i can dig s.t. up with the instrumentation and 
> will report back.

My theory is that this is related to the NOR driver then. UBI does not
use interrups. UBIFS uses hrtimer so it does have some code which may
run in interrupt context, but all it does - it wakes up a thread and
finishes.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubiformat & linux 3.0.14-rt

[Tim Sander]

Hi Artem
Am Montag, 16. Januar 2012 schrieb Artem Bityutskiy:
> On Mon, 2012-01-16 at 13:59 +0100, Tim Sander wrote:
> > Am Montag, 16. Januar 2012 schrieb Artem Bityutskiy:
> > > On Mon, 2012-01-16 at 11:43 +0100, Tim Sander wrote:
> > > > > http://git.infradead.org/ubifs-2.6.git/commit/1f5d78dc4823a85f112aa
> > > > > a2d0 f176 24f8c2a6c52
> > > > > http://git.infradead.org/ubifs-2.6.git/commit/d34315da9146253351146
> > > > > 140e a4b 277193ee5e5f
> > > > 
> > > > Do you think it is worthwhile checking the ubifs 3.0 backport with
> > > > the above two cherry-picked?
> > > 
> > > I've pushed them to the ubifs-v3.0 back-port tree as well. But I think
> > > they are useless - sorry, I was confused.
> > 
> > Ok i was wondering that logging was causing this cind of behavoir...
> > 
> > > Also, if you say this happens when you run ubiformat - this is
> > > something about your flash driver. Do you use NOR or NAND? Can you
> > > detect which interrupt takes too much time? If the theory is that this
> > > is your flash driver interrupt - can you verify it by injecting some
> > > instrumentation to the interrupt handler?
> > 
> > This is a arm i.mx35 (pcm43) platform using the nor and nand driver. The
> > mxc_nand driver is loaded later which seems to be around the same time
> > this message:
> > "sched: RT throttling activated" comes up and the ubi on the nand gets
> > attached. I will see what if i can dig s.t. up with the instrumentation
> > and will report back.
> 
> My theory is that this is related to the NOR driver then. UBI does not
> use interrups. UBIFS uses hrtimer so it does have some code which may
> run in interrupt context, but all it does - it wakes up a thread and
> finishes.
The pcm-043 platform uses the "physmap-flash" nor driver. I have not found 
any related interrupt handler? So probably the problem is more sublte. For 
example that the ubi background daemon is working and the timings are different 
after a firmware update which triggers this interrupt. Initially i thought this 
is closely related to the loading of the mxc_nand driver, attaching the ubi 
and mounting the ubifs filesystem since this message is output at roughly the 
same time. But it is only the nor flash which has been updated with ubiformat 
and not the nand flash when this error occurs.

So is there some work to be done, after mounting a ubiformat updated image?

Best regards
Tim

Re: ubiformat & linux 3.0.14-rt

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]

On Mon, 2012-01-16 at 15:59 +0100, Tim Sander wrote:
> So is there some work to be done, after mounting a ubiformat updated image?

Hmm, should not be. If there is, this is most probably a bug or I forgot
something.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubiformat & linux 3.0.14-rt

[Tim Sander]

Hi Artem

I just found out that the error i attributed to ubifs is most likely within 
the network code. Its just that after the ubiimage has been attached
the network was reconfigured. I first attributed it to the ubi code since this 
always happened so close to the ubiattach and was related with firmware update. 
Sorry for the noise and thanks for your support.

Tim