mx6: Audio does not work on linux-next 20140107

mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

Hi,

Playing audio on a mx6qsabresd board running linux-next 20140107 leads
to the following crash:

$ aplay clarinet.wav
random: nonblocking pool is initialized
Playing WAVE 'clarinet.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Mono
underrun!!! (at least -477981.433 ms long)
underrun!!! (at least -477981.435 ms long)
underrun!!! (at least -477981.436 ms long)
underrun!!! (at least -477981.435 ms long)
underrun!!! (at least -477981.436 ms long)
underrun!!! (at least -477981.435 ms long)
underrun!!! (at least -477981.436 ms long)
underrun!!! (at least -477981.436 ms long)
underrun!!! (at least -477981.436 ms long)
underrun!!! (at least -477981.436 ms long)
Unable to handle kernel NULL pointer dereference at virtual address 00000044
pgd = 80004000
[00000044] *pgd=00000000
Internal error: Oops: 17 [#1] SMP ARM
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.13.0-rc7-next-20140107+ #607
task: 8089fef0 ti: 80894000 task.ti: 80894000
PC is at dmaengine_pcm_dma_complete+0x10/0x50
LR is at sdma_tasklet+0x128/0x130
pc : [<804aee6c>]    lr : [<802e6428>]    psr: a0000113
sp : 80895df8  ip : 80895e08  fp : 80895e04
r10: 808f64c0  r9 : bfaa41b4  r8 : 00000000
r7 : 8089cd24  r6 : 00000001  r5 : 00000003  r4 : bfaa40e8
r3 : 00000000  r2 : 00000020  r1 : 00000000  r0 : bf15c000
Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c53c7d  Table: 4eedc04a  DAC: 00000017
Process swapper/0 (pid: 0, stack limit = 0x80894240)
Stack: (0x80895df8 to 0x80896000)
5de0:                                                       80895e2c 80895e08
5e00: 802e6428 804aee68 bfaa41b0 00000000 80894000 8089cd24 00000000 808f64c0
5e20: 80895e64 80895e30 8002c4b8 802e630c 80062a6c 80890390 00000040 00000000
5e40: 80896098 80894000 80896080 00000006 00000100 00000006 80895eb4 80895e68
5e60: 8002ca38 8002c434 00000022 00000000 bf807a14 00000001 00200000 ffff915f
5e80: 0000000a 00000000 f4000100 80894000 80890ff0 80894000 00000022 00000000
5ea0: 8063350c 808f4abd 80895ecc 80895eb8 8002cf4c 8002c914 00000184 8089cd24
5ec0: 80895ef4 80895ed0 8000f35c 8002cea8 000000a0 f400010c 8089ce80 80895f18
5ee0: f4000100 80894000 80895f14 80895ef8 80008640 8000f310 8000f734 20000113
5f00: ffffffff 80895f4c 80895f6c 80895f18 800130a4 8000861c 00000001 00000001
5f20: 00000000 8089fef0 00000000 8089c99c 8089c938 808f4abd 80894000 8063350c
5f40: 808f4abd 80895f6c 80895f30 80895f60 80062ab0 8000f734 20000113 ffffffff
5f60: 80895f84 80895f70 8006ad14 8000f718 00000000 8181d140 80895fb4 80895f88
5f80: 80624260 8006acb8 00000001 00000000 806241ac 8089ca38 808f4cc0 8089ca38
5fa0: 808f4cc0 ffffffff 80895ff4 80895fb8 80843b40 806241b8 ffffffff ffffffff
5fc0: 808435d0 00000000 00000000 80883970 10c53c7d 8089c928 8088396c 808a1524
5fe0: 1000406a 00000000 00000000 80895ff8 10008074 80843840 00000000 00000000
Backtrace:
[<804aee5c>] (dmaengine_pcm_dma_complete) from [<802e6428>] (sdma_tasklet+0x128)
[<802e6300>] (sdma_tasklet) from [<8002c4b8>] (tasklet_action+0x90/0x148)
 r10:808f64c0 r8:00000000 r7:8089cd24 r6:80894000 r5:00000000 r4:bfaa41b0
[<8002c428>] (tasklet_action) from [<8002ca38>] (__do_softirq+0x130/0x278)
 r10:00000006 r9:00000100 r8:00000006 r7:80896080 r6:80894000 r5:80896098
 r4:00000000
[<8002c908>] (__do_softirq) from [<8002cf4c>] (irq_exit+0xb0/0x100)
 r10:808f4abd r9:8063350c r8:00000000 r7:00000022 r6:80894000 r5:80890ff0
 r4:80894000
[<8002ce9c>] (irq_exit) from [<8000f35c>] (handle_IRQ+0x58/0xb8)
 r4:8089cd24 r3:00000184
[<8000f304>] (handle_IRQ) from [<80008640>] (gic_handle_irq+0x30/0x64)
 r8:80894000 r7:f4000100 r6:80895f18 r5:8089ce80 r4:f400010c r3:000000a0
[<80008610>] (gic_handle_irq) from [<800130a4>] (__irq_svc+0x44/0x5c)
Exception stack(0x80895f18 to 0x80895f60)
5f00:                                                       00000001 00000001
5f20: 00000000 8089fef0 00000000 8089c99c 8089c938 808f4abd 80894000 8063350c
5f40: 808f4abd 80895f6c 80895f30 80895f60 80062ab0 8000f734 20000113 ffffffff
 r7:80895f4c r6:ffffffff r5:20000113 r4:8000f734
[<8000f70c>] (arch_cpu_idle) from [<8006ad14>] (cpu_startup_entry+0x68/0x150)
[<8006acac>] (cpu_startup_entry) from [<80624260>] (rest_init+0xb4/0xdc)
 r7:8181d140 r3:00000000
[<806241ac>] (rest_init) from [<80843b40>] (start_kernel+0x30c/0x364)
 r6:ffffffff r5:808f4cc0 r4:8089ca38
[<80843834>] (start_kernel) from [<10008074>] (0x10008074)
Code: e1a0c00d e92dd800 e24cb004 e59030c8 (e5931044)
---[ end trace 076f597fb036d850 ]---
Kernel panic - not syncing: Fatal exception in interrupt
CPU1: stopping
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D      3.13.0-rc7-next-20140107+ 7
Backtrace:
[<8001233c>] (dump_backtrace) from [<800124d8>] (show_stack+0x18/0x1c)
 r6:80890ff0 r5:8089cd24 r4:00000000 r3:00000000
[<800124c0>] (show_stack) from [<80629fac>] (dump_stack+0x80/0x9c)
[<80629f2c>] (dump_stack) from [<80015494>] (handle_IPI+0x148/0x174)
 r4:00000001 r3:bf87d100
[<8001534c>] (handle_IPI) from [<8000866c>] (gic_handle_irq+0x5c/0x64)
 r8:bf89c000 r7:f4000100 r6:bf89df70 r5:8089ce80 r4:f400010c r3:bf87d100
[<80008610>] (gic_handle_irq) from [<800130a4>] (__irq_svc+0x44/0x5c)
Exception stack(0xbf89df70 to 0xbf89dfb8)
df60:                                     8000f730 bf89df80 00000000 00000000
df80: 00000000 8089c99c 8089c938 808f4abd bf89c000 8063350c 808f4abd bf89dfc4
dfa0: bf89dfa8 bf89dfb8 80062b9c 8000f734 60000113 ffffffff
 r7:bf89dfa4 r6:ffffffff r5:60000113 r4:8000f734
[<8000f70c>] (arch_cpu_idle) from [<8006ad14>] (cpu_startup_entry+0x68/0x150)
[<8006acac>] (cpu_startup_entry) from [<800150e0>] (secondary_start_kernel+0x12)
 r7:808f4fd0 r3:bf87d100
[<80014fb4>] (secondary_start_kernel) from [<10008704>] (0x10008704)
 r5:0000001f r4:4f88406a
CPU3: stopping
CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D      3.13.0-rc7-next-20140107+ 7
Backtrace:
[<8001233c>] (dump_backtrace) from [<800124d8>] (show_stack+0x18/0x1c)
 r6:80890ff0 r5:8089cd24 r4:00000000 r3:00000000
[<800124c0>] (show_stack) from [<80629fac>] (dump_stack+0x80/0x9c)
[<80629f2c>] (dump_stack) from [<80015494>] (handle_IPI+0x148/0x174)
 r4:00000003 r3:bf87e300
[<8001534c>] (handle_IPI) from [<8000866c>] (gic_handle_irq+0x5c/0x64)
 r8:bf8a0000 r7:f4000100 r6:bf8a1f70 r5:8089ce80 r4:f400010c r3:bf87e300
[<80008610>] (gic_handle_irq) from [<800130a4>] (__irq_svc+0x44/0x5c)
Exception stack(0xbf8a1f70 to 0xbf8a1fb8)
1f60:                                     8000f730 bf8a1f80 00000000 00000000
1f80: 00000000 8089c99c 8089c938 808f4abd bf8a0000 8063350c 808f4abd bf8a1fc4
1fa0: bf8a1fa8 bf8a1fb8 80062b9c 8000f734 60000013 ffffffff
 r7:bf8a1fa4 r6:ffffffff r5:60000013 r4:8000f734
[<8000f70c>] (arch_cpu_idle) from [<8006ad14>] (cpu_startup_entry+0x68/0x150)
[<8006acac>] (cpu_startup_entry) from [<800150e0>] (secondary_start_kernel+0x12)
 r7:808f4fd0 r3:bf87e300
[<80014fb4>] (secondary_start_kernel) from [<10008704>] (0x10008704)
 r5:0000001f r4:4f88406a
CPU2: stopping
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D      3.13.0-rc7-next-20140107+ 7
Backtrace:
[<8001233c>] (dump_backtrace) from [<800124d8>] (show_stack+0x18/0x1c)
 r6:80890ff0 r5:8089cd24 r4:00000000 r3:00000000
[<800124c0>] (show_stack) from [<80629fac>] (dump_stack+0x80/0x9c)
[<80629f2c>] (dump_stack) from [<80015494>] (handle_IPI+0x148/0x174)
 r4:00000002 r3:bf87da00
[<8001534c>] (handle_IPI) from [<8000866c>] (gic_handle_irq+0x5c/0x64)
 r8:bf89e000 r7:f4000100 r6:bf89ff70 r5:8089ce80 r4:f400010c r3:bf87da00
[<80008610>] (gic_handle_irq) from [<800130a4>] (__irq_svc+0x44/0x5c)
Exception stack(0xbf89ff70 to 0xbf89ffb8)
ff60:                                     8000f730 bf89ff80 00000000 00000000
ff80: 00000000 8089c99c 8089c938 808f4abd bf89e000 8063350c 808f4abd bf89ffc4
ffa0: bf89ffa8 bf89ffb8 80062b9c 8000f734 60000013 ffffffff
 r7:bf89ffa4 r6:ffffffff r5:60000013 r4:8000f734
[<8000f70c>] (arch_cpu_idle) from [<8006ad14>] (cpu_startup_entry+0x68/0x150)
[<8006acac>] (cpu_startup_entry) from [<800150e0>] (secondary_start_kernel+0x12)
 r7:808f4fd0 r3:bf87da00
[<80014fb4>] (secondary_start_kernel) from [<10008704>] (0x10008704)
 r5:0000001f r4:4f88406a
drm_kms_helper: panic occurred, switching back to text console

I haven't started debugging it, but if this looks familiar to someone,
please let me know.

Regards,

Fabio Estevam

Re: mx6: Audio does not work on linux-next 20140107

[Lars-Peter Clausen]

On 01/07/2014 12:56 PM, Fabio Estevam wrote:
[...]
> Backtrace:
> [<804aee5c>] (dmaengine_pcm_dma_complete) from [<802e6428>] (sdma_tasklet+0x128)
> [<802e6300>] (sdma_tasklet) from [<8002c4b8>] (tasklet_action+0x90/0x148)
>  r10:808f64c0 r8:00000000 r7:8089cd24 r6:80894000 r5:00000000 r4:bfaa41b0
[...]
> 
> I haven't started debugging it, but if this looks familiar to someone,
> please let me know.
> 

We've had similar problems like this before. Typically this is caused by the
dmaengine driver calling the complete callback when it shouldn't. E.g. after
dmaegine_terminate_all() has already been called for the channel. There is
also unfortunately a small chance condition (which needs some extensions to
the dmaengine API before we can fix it). But the chance for this race
condition to occur is really small and has only been observed on a 4 (or 8,
not sure) core system so far. Is the problem reproducible in your case? Also
do you have a know good kernel version?

> Regards,
> 
> Fabio Estevam
> 

Re: mx6: Audio does not work on linux-next 20140107

[Nicolin Chen]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On Tue, Jan 07, 2014 at 10:29:48AM -0200, Fabio Estevam wrote:
> Nicolin,
> 
> On linux-next I am no longer able to play audio without the sdma
> firmware (ok, maybe we should handle this topic on another thread).

I haven't tested the linux-next tree with my SabreSD recently. But..would that
be caused by the SDMA firmware version 2? Please try this patch to trick the
original sdma firmware and see if it's gonna work.

Surely, I'll test on my side tomorrow.

Thank you.
Nicolin Chen

[-- Attachment #2: 0002-firmware-imx-sdma-v2.patch --]
[-- Type: text/x-diff, Size: 806 bytes --]

>From d033e19003ee4e0e51b617a808cd068899f7fc01 Mon Sep 17 00:00:00 2001
From: Nicolin Chen <b42378@freescale.com>
Date: Wed, 6 Nov 2013 15:06:50 +0800
Subject: [PATCH 2/2] firmware: imx: sdma v2

Signed-off-by: Nicolin Chen <b42378@freescale.com>
---
 firmware/imx/sdma/sdma-imx6q.bin.ihex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/firmware/imx/sdma/sdma-imx6q.bin.ihex b/firmware/imx/sdma/sdma-imx6q.bin.ihex
index 2e561f0..ceb114b 100644
--- a/firmware/imx/sdma/sdma-imx6q.bin.ihex
+++ b/firmware/imx/sdma/sdma-imx6q.bin.ihex
@@ -1,4 +1,4 @@
-:1000000053444D4101000000010000001C000000AD
+:1000000053444D4102000000010000001C000000AC
 :1000100026000000B40000007A0600008202000002
 :10002000FFFFFFFF00000000FFFFFFFFFFFFFFFFDC
 :10003000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD0
-- 
1.8.4


[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

Hi Lars-Peter,

On Tue, Jan 7, 2014 at 10:07 AM, Lars-Peter Clausen <lars@metafoo.de> wrote:

> We've had similar problems like this before. Typically this is caused by the
> dmaengine driver calling the complete callback when it shouldn't. E.g. after
> dmaegine_terminate_all() has already been called for the channel. There is
> also unfortunately a small chance condition (which needs some extensions to
> the dmaengine API before we can fix it). But the chance for this race
> condition to occur is really small and has only been observed on a 4 (or 8,
> not sure) core system so far. Is the problem reproducible in your case? Also

Thanks for your comments. Yes, I get this issue on a mx6 quad all the
time running linux-next, but not on a mx6solo.

> do you have a know good kernel version?

Tried it on 3.13-rc7 and it works fine.

Nicolin,

On linux-next I am no longer able to play audio without the sdma
firmware (ok, maybe we should handle this topic on another thread).

Regards,

Fabio Estevam

Re: mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

On Tue, Jan 7, 2014 at 10:28 AM, Nicolin Chen
<Guangyu.Chen@freescale.com> wrote:
> On Tue, Jan 07, 2014 at 10:29:48AM -0200, Fabio Estevam wrote:
>> Nicolin,
>>
>> On linux-next I am no longer able to play audio without the sdma
>> firmware (ok, maybe we should handle this topic on another thread).
>
> I haven't tested the linux-next tree with my SabreSD recently. But..would that
> be caused by the SDMA firmware version 2? Please try this patch to trick the
> original sdma firmware and see if it's gonna work.

firmware/imx/sdma/sdma-imx6q.bin.ihex does not exist in mainline, so I
can't apply your patch.

Please send me your modified firmware offline and I can give it a try.

It would be really better if we could play audio without using any
firmware as in 3.13-rc7.

Regards,

Fabio Estevam

Re: mx6: Audio does not work on linux-next 20140107

[Lars-Peter Clausen]

On 01/07/2014 01:29 PM, Fabio Estevam wrote:
> Hi Lars-Peter,
> 
> On Tue, Jan 7, 2014 at 10:07 AM, Lars-Peter Clausen <lars@metafoo.de> wrote:
> 
>> We've had similar problems like this before. Typically this is caused by the
>> dmaengine driver calling the complete callback when it shouldn't. E.g. after
>> dmaegine_terminate_all() has already been called for the channel. There is
>> also unfortunately a small chance condition (which needs some extensions to
>> the dmaengine API before we can fix it). But the chance for this race
>> condition to occur is really small and has only been observed on a 4 (or 8,
>> not sure) core system so far. Is the problem reproducible in your case? Also
> 
> Thanks for your comments. Yes, I get this issue on a mx6 quad all the
> time running linux-next, but not on a mx6solo.

Looking at the imx-sdma dmaengine driver there is a whole bunch of issues in
regard to concurrency. The problem might have existed before and only
surfaced now due to some random other changes.

For starters there is absolutely no protection against concurrent access to
the drivers state struct. Access to buf_tail, bd, num_bd and possible other
fields should be protected by a lock. The second problem is that the active
descriptor is not invalidated when dmaengine_terminate_all() is called, so
if the tasklet runs after dmaengine_terminate_all() has been called the DMA
descriptor callback will be called even though it shouldn't.

And another problem is that the imx-sdma driver doesn't really conform to
the semantics of the dmaengine API with regrads to the prepare, submit and
issue_pending. But I don't think this is the source of the problem in this case.

- Lars

Re: mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

Hi Nicolin,

On Tue, Jan 7, 2014 at 10:49 AM, Fabio Estevam <festevam@gmail.com> wrote:
> On Tue, Jan 7, 2014 at 10:28 AM, Nicolin Chen
> <Guangyu.Chen@freescale.com> wrote:
>> On Tue, Jan 07, 2014 at 10:29:48AM -0200, Fabio Estevam wrote:
>>> Nicolin,
>>>
>>> On linux-next I am no longer able to play audio without the sdma
>>> firmware (ok, maybe we should handle this topic on another thread).
>>
>> I haven't tested the linux-next tree with my SabreSD recently. But..would that
>> be caused by the SDMA firmware version 2? Please try this patch to trick the
>> original sdma firmware and see if it's gonna work.
>
> firmware/imx/sdma/sdma-imx6q.bin.ihex does not exist in mainline, so I
> can't apply your patch.
>
> Please send me your modified firmware offline and I can give it a try.
>
> It would be really better if we could play audio without using any
> firmware as in 3.13-rc7.

Just retried with your updated SDMA firmware (version 2.1) and now I
can get audio playback to work again.

Also confirmed that running with no SDMA firmware the crash still happens.

In 3.12 and 3.13-rc we are able to play audio without using any SDMA
firmware, but it we are not longer able to do this now. Other than
that, people will have to use the exact 2.1 firmware version, so there
will be a regression in 3.14, right?

Couldn't we keep the 3.12/3.13 behaviour, ie, be able to play audio
without loading any SDMA firmware?

Regards,

Fabio Estevam

Re: mx6: Audio does not work on linux-next 20140107

[Nicolin Chen]

Hi Fabio,

On Wed, Jan 08, 2014 at 01:50:45AM -0200, Fabio Estevam wrote:
> In 3.12 and 3.13-rc we are able to play audio without using any SDMA
> firmware, but it we are not longer able to do this now. Other than
> that, people will have to use the exact 2.1 firmware version, so there
> will be a regression in 3.14, right?

I think it is impossible to run SDMA without firmware, which stores all
the scripts we need, including Audio's one. Although the upstream kernel
doesn't have firmware, the reason why you were able to run audio playback
is because you have the firmware in your rootfs. So since the driver's
upgraded, it's plausible for us to upgrade the firmware as well. But I
do agree the way we handle the old version while trying to use the new
script isn't professional. Ideally we should switch the script to the
old one if we find the firmware version is 1.1. But technically it's
hard to achieve that since we assign the script in DT.

So currently if people are gonna use 3.14, they might need to upgrade
their firmware as you just tried to your rootfs.
 
> Couldn't we keep the 3.12/3.13 behaviour, ie, be able to play audio
> without loading any SDMA firmware?

There are two ways to keep it as the old one. First is patching the SDMA
firmware patch, including v2, to upstream kernel. And the second is to
revert the patch (ARM: dts: imx: use dual-fifo sdma script for ssi) so
that the SSI driver and SDMA driver would continue to use the old single
FIFO mode.

I'll align this issue with Shawn first to see if there's a better solution.

Thank you,
Nicolin Chen

Re: mx6: Audio does not work on linux-next 20140107

[Nicolin Chen]

On Wed, Jan 08, 2014 at 08:01:56AM -0200, Fabio Estevam wrote:
> Hi Nicolin,
> 
> On Wed, Jan 8, 2014 at 3:04 AM, Nicolin Chen <Guangyu.Chen@freescale.com> wrote:
> 
> > I think it is impossible to run SDMA without firmware, which stores all
> > the scripts we need, including Audio's one. Although the upstream kernel
> > doesn't have firmware, the reason why you were able to run audio playback
> > is because you have the firmware in your rootfs. So since the driver's
> > upgraded, it's plausible for us to upgrade the firmware as well. But I
> 
> No, this is not correct.
> 

Yea, Shawn just told me there's an inner firmware in the ROM code while
this firmware only supports some basic scripts. And I've already sent
my fixing patches (you're in the to list). Please take a look and test
if you can.

Thank you,
Nicolin Chen

Re: mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

Hi Nicolin,

On Wed, Jan 8, 2014 at 3:04 AM, Nicolin Chen <Guangyu.Chen@freescale.com> wrote:

> I think it is impossible to run SDMA without firmware, which stores all
> the scripts we need, including Audio's one. Although the upstream kernel
> doesn't have firmware, the reason why you were able to run audio playback
> is because you have the firmware in your rootfs. So since the driver's
> upgraded, it's plausible for us to upgrade the firmware as well. But I

No, this is not correct.

Please see this commit from Sascha:

commit dcfec3c09890120d86d4e86887074c
76763075ca
Author: Sascha Hauer <s.hauer@pengutronix.de>
Date:   Tue Aug 20 10:04:32 2013 +0200

    dma: imx-sdma: Add ROM script addresses to driver

    This adds the ROM script addresses for i.MX25, i.MX5x and i.MX6 to the
    SDMA driver needed for the driver to work without additional firmware.

    The ROM script addresses are SoC specific and in some cases even tapeout
    specific. This patch adds the ROM script addresses only for SoCs which
    do not have a tapeout specific SDMA ROM, because currently it's unclear
    how this case should be handled.

    Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
    Acked-by: Shawn Guo <shawn.guo@linaro.org>
    Signed-off-by: Vinod Koul <vinod.koul@intel.com>

It allows audio to be played without any additional SDMA firmware.

Please try running 3.13-rc7 without the SDMA firmware and linux-next
without firmware and you will get the difference: 3.13-rc7 will play
the audio just fine, but linux-next will crash the system.

Regards,

Fabio Estevam

Re: mx6: Audio does not work on linux-next 20140107

[Fabio Estevam]

On Wed, Jan 8, 2014 at 7:53 AM, Nicolin Chen <Guangyu.Chen@freescale.com> wrote:

> Yea, Shawn just told me there's an inner firmware in the ROM code while
> this firmware only supports some basic scripts. And I've already sent
> my fixing patches (you're in the to list). Please take a look and test
> if you can.

Your patches fix the issue. Thanks, Nicolin.