Player Thread is not woken after period elapsed

Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

Hi,

When I tried to root cause a glitch in the audio playback, I found a weird behavior. 

User thread which invokes the writei function which in turn  invokes a kernel function which waits for the free buffer to write the audio data.  This kernel function adds this thread to a wake(sleep) queue  and calls a schedule_timeout (msecs_to_jiffies(10000)).

When the audio playback of a period is completed, the hardware generates an interrupt. The handler in the path wake_up the thread added to the sleep queue.  Most of the time, the playback thread is woken up.

During this error case, the thread is not getting woken up. However the sleep queue is not empty.  This continues until the playback enter into underrun due to all the periods in the buffer are played back. Audio playback continues after recovery from xrun. 

Will you provide some hint on how to go about identifying the root cause?

Thanks
-Sing

Re: Player Thread is not woken after period elapsed

[Arjan van de Ven]

On Mon, 5 Sep 2011 23:08:53 +0530
"Nallasellan, Singaravelan" <singaravelan.nallasellan@intel.com> wrote:

> Hi,
> 
> When I tried to root cause a glitch in the audio playback, I found a
> weird behavior. 
> 
> User thread which invokes the writei function which in turn  invokes
> a kernel function which waits for the free buffer to write the audio
> data.  This kernel function adds this thread to a wake(sleep) queue
> and calls a schedule_timeout (msecs_to_jiffies(10000)).

actually it doesn't... you must be on some really really old kernel or
something.


> 
> Will you provide some hint on how to go about identifying the root
> cause?

https://lkml.org/lkml/2011/9/5/201



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

RE: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

Sorry, I have not provided the kernel version. Kernel version is 2.6.35. 
However, wait_for_avail() does not have the code similar to the one provided in the patch below. I will give a try. 

-----Original Message-----
From: Arjan van de Ven [mailto:arjan@infradead.org] 
Sent: Monday, September 05, 2011 11:23 PM
To: Nallasellan, Singaravelan
Cc: alsa-devel@alsa-project.org; linux-kernel@vger.kernel.org
Subject: Re: Player Thread is not woken after period elapsed

On Mon, 5 Sep 2011 23:08:53 +0530
"Nallasellan, Singaravelan" <singaravelan.nallasellan@intel.com> wrote:

> Hi,
> 
> When I tried to root cause a glitch in the audio playback, I found a 
> weird behavior.
> 
> User thread which invokes the writei function which in turn  invokes a 
> kernel function which waits for the free buffer to write the audio 
> data.  This kernel function adds this thread to a wake(sleep) queue 
> and calls a schedule_timeout (msecs_to_jiffies(10000)).

actually it doesn't... you must be on some really really old kernel or something.


> 
> Will you provide some hint on how to go about identifying the root 
> cause?

https://lkml.org/lkml/2011/9/5/201



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, visit http://www.lesswatts.org

Re: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

Sorry, I have not provided the kernel version. Kernel version is 2.6.35. 
However, wait_for_avail() does not have the code similar to the one provided in the patch below. I will give a try. 

-----Original Message-----
From: Arjan van de Ven [mailto:arjan@infradead.org] 
Sent: Monday, September 05, 2011 11:23 PM
To: Nallasellan, Singaravelan
Cc: alsa-devel@alsa-project.org; linux-kernel@vger.kernel.org
Subject: Re: Player Thread is not woken after period elapsed

On Mon, 5 Sep 2011 23:08:53 +0530
"Nallasellan, Singaravelan" <singaravelan.nallasellan@intel.com> wrote:

> Hi,
> 
> When I tried to root cause a glitch in the audio playback, I found a 
> weird behavior.
> 
> User thread which invokes the writei function which in turn  invokes a 
> kernel function which waits for the free buffer to write the audio 
> data.  This kernel function adds this thread to a wake(sleep) queue 
> and calls a schedule_timeout (msecs_to_jiffies(10000)).

actually it doesn't... you must be on some really really old kernel or something.


> 
> Will you provide some hint on how to go about identifying the root 
> cause?

https://lkml.org/lkml/2011/9/5/201



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, visit http://www.lesswatts.org

Re: Player Thread is not woken after period elapsed

[Arjan van de Ven]

On Mon, 5 Sep 2011 23:38:49 +0530
"Nallasellan, Singaravelan" <singaravelan.nallasellan@intel.com> wrote:

please do not top post!

correcting that mistake for you for this time


>> > User thread which invokes the writei function which in turn
>> > invokes a kernel function which waits for the free buffer to write
>> > the audio data.  This kernel function adds this thread to a
>> > wake(sleep) queue and calls a schedule_timeout
>> > (msecs_to_jiffies(10000)).
> 
>> actually it doesn't... you must be on some really really old kernel
>> or something.
>> 

> Sorry, I have not provided the kernel version. Kernel version is
> 2.6.35. 

Please always test bugs like this with a recent kernel. 2.6.35 is
pretty much 6 versions (and 18 months) behind and at that point you
really have to test with something newer first before asking others to
help you and spend their time on your problem.


>> > Will you provide some hint on how to go about identifying the root 
>> > cause?

>> https://lkml.org/lkml/2011/9/5/201

> However, wait_for_avail() does not have the code similar to
> the one provided in the patch below. 

the patch is against a recent kernel; no surprise that the code has
changed in the last 18 months....


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

RE: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> please do not top post!
> 
Ok. 
> correcting that mistake for you for this time
> 
Thanks so much.
> 
> >> > User thread which invokes the writei function which in turn invokes
> >> > a kernel function which waits for the free buffer to write the
> >> > audio data.  This kernel function adds this thread to a
> >> > wake(sleep) queue and calls a schedule_timeout
> >> > (msecs_to_jiffies(10000)).
> >
> >> actually it doesn't... you must be on some really really old kernel
> >> or something.
> >>
> 
> > Sorry, I have not provided the kernel version. Kernel version is
> > 2.6.35.
> 
> Please always test bugs like this with a recent kernel. 2.6.35 is pretty much 6
> versions (and 18 months) behind and at that point you really have to test with
> something newer first before asking others to help you and spend their time on
> your problem.
> 
Unfortunately, I am stuck with that version as the product that I work on is on that version. 
> 
> >> > Will you provide some hint on how to go about identifying the root
> >> > cause?
> 
> >> https://lkml.org/lkml/2011/9/5/201
> 
I manually ported the changes and verified. I still see the issue. Found more information which might throw more light into the problem. 
p->state in try_to_wake_up() is 0 when it is expected to be 2 or 3.

> > However, wait_for_avail() does not have the code similar to the one
> > provided in the patch below.
> 
> the patch is against a recent kernel; no surprise that the code has changed in the
> last 18 months....
> 
> 
> --
> Arjan van de Ven 	Intel Open Source Technology Centre
> For development, discussion and tips for power savings, visit
> http://www.lesswatts.org

Re: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> please do not top post!
> 
Ok. 
> correcting that mistake for you for this time
> 
Thanks so much.
> 
> >> > User thread which invokes the writei function which in turn invokes
> >> > a kernel function which waits for the free buffer to write the
> >> > audio data.  This kernel function adds this thread to a
> >> > wake(sleep) queue and calls a schedule_timeout
> >> > (msecs_to_jiffies(10000)).
> >
> >> actually it doesn't... you must be on some really really old kernel
> >> or something.
> >>
> 
> > Sorry, I have not provided the kernel version. Kernel version is
> > 2.6.35.
> 
> Please always test bugs like this with a recent kernel. 2.6.35 is pretty much 6
> versions (and 18 months) behind and at that point you really have to test with
> something newer first before asking others to help you and spend their time on
> your problem.
> 
Unfortunately, I am stuck with that version as the product that I work on is on that version. 
> 
> >> > Will you provide some hint on how to go about identifying the root
> >> > cause?
> 
> >> https://lkml.org/lkml/2011/9/5/201
> 
I manually ported the changes and verified. I still see the issue. Found more information which might throw more light into the problem. 
p->state in try_to_wake_up() is 0 when it is expected to be 2 or 3.

> > However, wait_for_avail() does not have the code similar to the one
> > provided in the patch below.
> 
> the patch is against a recent kernel; no surprise that the code has changed in the
> last 18 months....
> 
> 
> --
> Arjan van de Ven 	Intel Open Source Technology Centre
> For development, discussion and tips for power savings, visit
> http://www.lesswatts.org

Re: Player Thread is not woken after period elapsed

[Arjan van de Ven]

> > 
> > Please always test bugs like this with a recent kernel. 2.6.35 is
> > pretty much 6 versions (and 18 months) behind and at that point you
> > really have to test with something newer first before asking others
> > to help you and spend their time on your problem.
> > 
> Unfortunately, I am stuck with that version as the product that I
> work on is on that version. 

and that is everyone elses problem ... how?

if you can't be bothered to try at least a recent kernel, why would
anyone else be bothered to help you on their own time?????


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

RE: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> 
> > >
> > > Please always test bugs like this with a recent kernel. 2.6.35 is
> > > pretty much 6 versions (and 18 months) behind and at that point you
> > > really have to test with something newer first before asking others
> > > to help you and spend their time on your problem.
> > >
> > Unfortunately, I am stuck with that version as the product that I work
> > on is on that version.
> 
> and that is everyone elses problem ... how?
> 
> if you can't be bothered to try at least a recent kernel, why would anyone else
> be bothered to help you on their own time?????
> 
I completely agree with you. Sorry for the inconvenience. But I have no choice. 
Further investigation of this issue shows that the thread is in running state and it is not scheduled. I am not sure how I will go about debugging this issue. Any pointer would help. 
> 
> --

Re: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> 
> > >
> > > Please always test bugs like this with a recent kernel. 2.6.35 is
> > > pretty much 6 versions (and 18 months) behind and at that point you
> > > really have to test with something newer first before asking others
> > > to help you and spend their time on your problem.
> > >
> > Unfortunately, I am stuck with that version as the product that I work
> > on is on that version.
> 
> and that is everyone elses problem ... how?
> 
> if you can't be bothered to try at least a recent kernel, why would anyone else
> be bothered to help you on their own time?????
> 
I completely agree with you. Sorry for the inconvenience. But I have no choice. 
Further investigation of this issue shows that the thread is in running state and it is not scheduled. I am not sure how I will go about debugging this issue. Any pointer would help. 
> 
> --

Re: Player Thread is not woken after period elapsed

[Yong Zhang]

On Wed, Sep 07, 2011 at 05:32:10PM +0530, Nallasellan, Singaravelan wrote:
> 
> > 
> > > >
> > > > Please always test bugs like this with a recent kernel. 2.6.35 is
> > > > pretty much 6 versions (and 18 months) behind and at that point you
> > > > really have to test with something newer first before asking others
> > > > to help you and spend their time on your problem.
> > > >
> > > Unfortunately, I am stuck with that version as the product that I work
> > > on is on that version.
> > 
> > and that is everyone elses problem ... how?
> > 
> > if you can't be bothered to try at least a recent kernel, why would anyone else
> > be bothered to help you on their own time?????
> > 
> I completely agree with you. Sorry for the inconvenience. But I have no choice. 
> Further investigation of this issue shows that the thread is in running state and it is not scheduled. I am not sure how I will go about debugging this issue. Any pointer would help. 

Could you confirm if [commit f26f9af Sched: fix skip_clock_update optimization]
residents in your kernel?

BTW, I'm refering mainline commit here, I think it also has been backported
to -statbe.

Thanks,
Yong

> > 
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Only stand for myself

Re: Player Thread is not woken after period elapsed

[Yong Zhang]

On Thu, Sep 08, 2011 at 10:13:27AM +0800, Yong Zhang wrote:
> On Wed, Sep 07, 2011 at 05:32:10PM +0530, Nallasellan, Singaravelan wrote:
> > 
> > > 
> > > > >
> > > > > Please always test bugs like this with a recent kernel. 2.6.35 is
> > > > > pretty much 6 versions (and 18 months) behind and at that point you
> > > > > really have to test with something newer first before asking others
> > > > > to help you and spend their time on your problem.
> > > > >
> > > > Unfortunately, I am stuck with that version as the product that I work
> > > > on is on that version.
> > > 
> > > and that is everyone elses problem ... how?
> > > 
> > > if you can't be bothered to try at least a recent kernel, why would anyone else
> > > be bothered to help you on their own time?????
> > > 
> > I completely agree with you. Sorry for the inconvenience. But I have no choice. 
> > Further investigation of this issue shows that the thread is in running state and it is not scheduled. I am not sure how I will go about debugging this issue. Any pointer would help. 
> 
> Could you confirm if [commit f26f9af Sched: fix skip_clock_update optimization]
> residents in your kernel?

And commit da7a735e51f9622eb3e1672594d4a41da01d7e4f
http://marc.info/?l=linux-kernel&m=129527509622696&w=2

Thanks,
Yong

RE: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 664 bytes --]

> > > Further investigation of this issue shows that the thread is in running state
> and it is not scheduled. I am not sure how I will go about debugging this issue.
> Any pointer would help.
> >
> > Could you confirm if [commit f26f9af Sched: fix skip_clock_update
> > optimization] residents in your kernel?
I applied this patch manually. I still see the issue.
> 
> And commit da7a735e51f9622eb3e1672594d4a41da01d7e4f
> http://marc.info/?l=linux-kernel&m=129527509622696&w=2
This is already applied.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> > > Further investigation of this issue shows that the thread is in running state
> and it is not scheduled. I am not sure how I will go about debugging this issue.
> Any pointer would help.
> >
> > Could you confirm if [commit f26f9af Sched: fix skip_clock_update
> > optimization] residents in your kernel?
I applied this patch manually. I still see the issue.
> 
> And commit da7a735e51f9622eb3e1672594d4a41da01d7e4f
> http://marc.info/?l=linux-kernel&m=129527509622696&w=2
This is already applied.

Re: Player Thread is not woken after period elapsed

[Yong Zhang]

On Thu, Sep 08, 2011 at 03:15:38PM +0530, Nallasellan, Singaravelan wrote:
> > > > Further investigation of this issue shows that the thread is in running state

Hmm, how do you detect it?

> > and it is not scheduled. I am not sure how I will go about debugging this issue.
> > Any pointer would help.

Could 'cat /proc/sched_debug' give some useful info?

And BTW, have you tried CONFIG_DETECT_SOFTLOCKUP = y?

Thanks,
Yong


> > >
> > > Could you confirm if [commit f26f9af Sched: fix skip_clock_update
> > > optimization] residents in your kernel?
> I applied this patch manually. I still see the issue.
> > 
> > And commit da7a735e51f9622eb3e1672594d4a41da01d7e4f
> > http://marc.info/?l=linux-kernel&m=129527509622696&w=2
> This is already applied.
> \x13��칻\x1c�&�~�&�\x18��+-��ݶ\x17��w��˛���m�b��dz�ޖ)���w*\x1fjg���\x1e�����ݢj/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥>W����i�axP\x10j�m����\f-�\x1a+��d�_

-- 
Only stand for myself

RE: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 576 bytes --]

> > > > > Further investigation of this issue shows that the thread is in
> > > > > running state
> 
> Hmm, how do you detect it?
> 
> > > and it is not scheduled. I am not sure how I will go about debugging this
> issue.
> > > Any pointer would help.
> 
> Could 'cat /proc/sched_debug' give some useful info?
Tried. It did not help.
> 
> And BTW, have you tried CONFIG_DETECT_SOFTLOCKUP = y?
> 
It is already set. 
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: Player Thread is not woken after period elapsed

[Nallasellan, Singaravelan]

> > > > > Further investigation of this issue shows that the thread is in
> > > > > running state
> 
> Hmm, how do you detect it?
> 
> > > and it is not scheduled. I am not sure how I will go about debugging this
> issue.
> > > Any pointer would help.
> 
> Could 'cat /proc/sched_debug' give some useful info?
Tried. It did not help.
> 
> And BTW, have you tried CONFIG_DETECT_SOFTLOCKUP = y?
> 
It is already set. 

Re: Player Thread is not woken after period elapsed

[Yong Zhang]

On Fri, Sep 09, 2011 at 06:12:03PM +0530, Nallasellan, Singaravelan wrote:
> > > > > > Further investigation of this issue shows that the thread is in
> > > > > > running state
> > 
> > Hmm, how do you detect it?
> > 
> > > > and it is not scheduled. I am not sure how I will go about debugging this
> > issue.
> > > > Any pointer would help.
> > 
> > Could 'cat /proc/sched_debug' give some useful info?
> Tried. It did not help.
> > 
> > And BTW, have you tried CONFIG_DETECT_SOFTLOCKUP = y?
> > 
> It is already set. 

That sound at least kwatchdogd could be scheduled suscessfully.

And if CONFIG_DETECT_HUNG_TASK could not bite your "Player Thread",
I have no idea what's going on.

Maybe you could try the latest kernel?

Thanks,
Yong

Re: Player Thread is not woken after period elapsed

[Shan Hai]

On 09/06/2011 01:38 AM, Nallasellan, Singaravelan wrote:
> Hi,
>
> When I tried to root cause a glitch in the audio playback, I found a weird behavior.
>
> User thread which invokes the writei function which in turn  invokes a kernel function which waits for the free buffer to write the audio data.  This kernel function adds this thread to a wake(sleep) queue  and calls a schedule_timeout (msecs_to_jiffies(10000)).
>
> When the audio playback of a period is completed, the hardware generates an interrupt. The handler in the path wake_up the thread added to the sleep queue.  Most of the time, the playback thread is woken up.
>
> During this error case, the thread is not getting woken up. However the sleep queue is not empty.  This continues until the playback enter into underrun due to all the periods in the buffer are played back. Audio playback continues after recovery from xrun.
>

Its possible that your user thread was woken up by the IRQ handler but
it found out the 'free buffer' is not available and go back to sleep.

- check the /proc/<pid of user thread>/schedstat for whether the thread 
was scheduled
- your shared buffer design might has problems in it, it seemed there is 
a race between
     your IRQ handler and 'free buffer' checking logic.
- ftrace might be helpful

Cheers
Shan Hai

> Will you provide some hint on how to go about identifying the root cause?
>
> Thanks
> -Sing
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/