[3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Tetsuo Handa]

Hello.

I'm experiencing two regressions in 3.8-rcX.

Regression:

  (1) Console shows nothing, which makes it impossible to determine whether
      the system is working or not.

  (2) SCSI_SYM53C8XX_2 cannot register IRQ, showing below messages upon load.

      PCI: enabling device 0000:00:0c.0 (0100 -> 0103)
      sym0: <895a> rev 0x0 at pci 0000:00:0c.0 irq 27
      sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
      sym0: request irq 27 failure
      sym0: giving up ...

While (1) was fixed by now, I can't bisect (2) since (1) is preventing (2) from
printing messages.

Bisection result:

  No problem until commit b1112249 "OMAPDSS: DISPC: cleanup lcd/digit enable/disable".

  (1) was introduced by commit 07c9249f "ARM: 7554/1: VIC: use irq_domain_add_simple()".

  (2) was introduced between commit 07c9249f "ARM: 7554/1: VIC: use irq_domain_add_simple()"
  and commit 5c49985c "Merge branch 'fixes' of git://git.linaro.org/people/rmk/linux-arm".

  (1) was fixed by commit 5c49985c "Merge branch 'fixes' of git://git.linaro.org/people/rmk/linux-arm".

  (2) is not yet fixed as of commit 5da1f88b "Merge tag 'usb-3.8-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb".

Environment:

  Host is Ubuntu 12.04 on x86_32.
  Cross compiling with ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- using
  gcc-arm-linux-gnueabi 4:4.6.2-7 and booting with arch/arm/boot/zImage using
  qemu-system 1.0.50-2012.03-0ubuntu2.1 .

Command line:

  $ qemu-system-arm -M versatilepb -hda hda.img -kernel arch/arm/boot/zImage -append "root=/dev/sda1"

Kernel config:

  http://I-love.SAKURA.ne.jp/tmp/config-3.8-rc1-arm

Regards.

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Sun, Jan 20, 2013 at 1:24 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:

> While (1) was fixed by now, I can't bisect (2) since (1) is preventing (2) from
> printing messages.

I usually bisect like this:

1.  Get hold of the patch fixing the boot regression:
    git format-patch
f5565295892eb93c3191aa241405fe8b685542d6^..f5565295892eb93c3191aa241405fe8b685542d6

2. At each iterative step using git bisect:
   patch -p1 < 0001-ARM-7610-1-versatile-bump-IRQ-numbers.patch

3. Boot/trial-and-horror

4. git reset --hard HEAD
   To nullify the effects of the patch.

5. git bisect [good|bad]

6. Iterate from (2) until you find the bug.

Yours,
Linus Walleij

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Tetsuo Handa]

Linus Walleij wrote:
> > While (1) was fixed by now, I can't bisect (2) since (1) is preventing (2) from
> > printing messages.
> 
> I usually bisect like this:
> 
> 1.  Get hold of the patch fixing the boot regression:
>     git format-patch
> f5565295892eb93c3191aa241405fe8b685542d6^..f5565295892eb93c3191aa241405fe8b685542d6

Thanks, but sorry, I couldn't understand how this command line works.

Which one ("blank console bug" or "IRQ registration failure bug") does
"the boot regression" refer?

Your command line generated only one patch but you say "iterate".

  $ git format-patch f5565295^..f5565295
  0001-ARM-7610-1-versatile-bump-IRQ-numbers.patch

If I use 5c49985c^..5c49985c (the patch which the blank console bug was fixed),
it generated 8 patches.

  $ git format-patch 5c49985c^..5c49985c
  0001-ARM-7608-1-l2x0-Only-set-.set_debug-on-PL310-r3p0-an.patch
  0002-ARM-7609-1-disable-errata-work-arounds-which-access-.patch
  0003-ARM-7610-1-versatile-bump-IRQ-numbers.patch
  0004-ARM-7611-1-VIC-fix-bug-in-VIC-irqdomain-code.patch
  0005-ARM-7612-1-imx-Do-not-select-some-errata-that-depend.patch
  0006-ARM-7614-1-mm-fix-wrong-branch-from-Cortex-A9-to-PJ4.patch
  0007-ARM-7615-1-cache-l2x0-aurora-Invalidate-during-clean.patch
  0008-ARM-7616-1-cache-l2x0-aurora-Use-writel_relaxed-inst.patch
  $ grep -il irq 000*
  0003-ARM-7610-1-versatile-bump-IRQ-numbers.patch
  0004-ARM-7611-1-VIC-fix-bug-in-VIC-irqdomain-code.patch
  0008-ARM-7616-1-cache-l2x0-aurora-Use-writel_relaxed-inst.patch

Since the IRQ registration failure bug message contains IRQ number "27",
I suspect f5565295 "ARM: 7610/1: versatile: bump IRQ numbers"

  The Versatile starts to register Linux IRQ numbers from offset 0
  which is illegal, since this is NO_IRQ. Bump all hard-coded IRQs
  by 32 to get rid of the problem.

and 5ced33bc06 "ARM: 7611/1: VIC: fix bug in VIC irqdomain code"

  The VIC irqdomain code added in commit
  07c9249f1fa90cc8189bed44c0bcece664596a72
  "ARM: 7554/1: VIC: use irq_domain_add_simple()"

  Had two bugs:

  1) It didn't call irq_create_mapping() once on each
     valid irq source in the slowpath when registering
     the controller.

  2) It passed a -1 as IRQ offset for the DT case, whereas
     0 should be passed as invalid IRQ instead.

but how can I test them?

> 
> 2. At each iterative step using git bisect:
>    patch -p1 < 0001-ARM-7610-1-versatile-bump-IRQ-numbers.patch
> 
> 3. Boot/trial-and-horror
> 
> 4. git reset --hard HEAD
>    To nullify the effects of the patch.
> 
> 5. git bisect [good|bad]
> 
Excuse me, what range do I need to pass to "git bisect start" command line?

> 6. Iterate from (2) until you find the bug.

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Tue, Jan 22, 2013 at 10:17 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:

>> 1.  Get hold of the patch fixing the boot regression:
>>     git format-patch
>> f5565295892eb93c3191aa241405fe8b685542d6^..f5565295892eb93c3191aa241405fe8b685542d6
>
> Thanks, but sorry, I couldn't understand how this command line works.
>
> Which one ("blank console bug" or "IRQ registration failure bug") does
> "the boot regression" refer?

They are likely one and the same bug? The only regression on
the versatile I have caused came from commit
f5565295892eb93c3191aa241405fe8b685542d6.

> Your command line generated only one patch but you say "iterate".

I mean you should iterate the process, not the patch generation.

>   $ git format-patch f5565295^..f5565295
>   0001-ARM-7610-1-versatile-bump-IRQ-numbers.patch
>
> If I use 5c49985c^..5c49985c (the patch which the blank console bug was fixed),
> it generated 8 patches.

This is because you have made git format-patch in a merge commit
which contains several patches.

You can probably actually apply all of them at each step like
that if  you prefer: patch -p1 < *.patch

> Since the IRQ registration failure bug message contains IRQ number "27",
> I suspect f5565295 "ARM: 7610/1: versatile: bump IRQ numbers"

Me too :-) Atleast Grant told me that that one patch fixed the
boot regression.

And that is why I mentioned that patch specifically above,
I think it's the only one you need.

> and 5ced33bc06 "ARM: 7611/1: VIC: fix bug in VIC irqdomain code"

I don't think you need this patch.

> but how can I test them?

Apply/compile/boot?

Sorry I don't quite get the question?

>> 2. At each iterative step using git bisect:
>>    patch -p1 < 0001-ARM-7610-1-versatile-bump-IRQ-numbers.patch
>>
>> 3. Boot/trial-and-horror
>>
>> 4. git reset --hard HEAD
>>    To nullify the effects of the patch.
>>
>> 5. git bisect [good|bad]
>>
> Excuse me, what range do I need to pass to "git bisect start" command line?

Same as you mentioned in your first mail I suppose?

>> 6. Iterate from (2) until you find the bug.

When you cannot proceed because you need a patch unrelated to the
problem you want to solve, you apply that patch on top of the
bisection HEAD with patch -p1 < foo.patch, test, reset HEAD,
git bisect <result> etc....

I'm sorry if this is a bit complicated but bisection down several
regressions is one of the more advanced forms of regression
bug hunt... You're quite cool if you can handle it :-)

Yours,
Linus Walleij

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Tetsuo Handa]

Linus Walleij wrote:
> > Which one ("blank console bug" or "IRQ registration failure bug") does
> > "the boot regression" refer?
> 
> They are likely one and the same bug? The only regression on
> the versatile I have caused came from commit
> f5565295892eb93c3191aa241405fe8b685542d6.
> 
It seems that my case

  PCI: enabling device 0000:00:0c.0 (0100 -> 0103)
  sym0: <895a> rev 0x0 at pci 0000:00:0c.0 irq 27
  sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
  sym0: request irq 27 failure
  sym0: giving up ...

may not come from that commit.

> > Since the IRQ registration failure bug message contains IRQ number "27",
> > I suspect f5565295 "ARM: 7610/1: versatile: bump IRQ numbers"
> 
> Me too :-) Atleast Grant told me that that one patch fixed the
> boot regression.
> 
> And that is why I mentioned that patch specifically above,
> I think it's the only one you need.
> 
> > and 5ced33bc06 "ARM: 7611/1: VIC: fix bug in VIC irqdomain code"
> 
> I don't think you need this patch.
> 
> > but how can I test them?
> 
> Apply/compile/boot?
> 
> Sorry I don't quite get the question?

Well, my case was introduced between 07c9249f and 5c49985c but I cannot see
console messages from 07c9249f till 5c49985c.

I tried reverting f5565295 and/or 5ced33bc06 (as shown below) but it did not
help.

$ git pull origin master # Move to latest commit.
$ make -s
$ qemu-system-arm

The message in question ( "sym0: request irq 27 failure" ) was printed.

$ git show f5565295 | patch -p1 -R # Revert only "ARM: 7610/1: versatile: bump IRQ numbers"
$ make -s
$ qemu-system-arm

Cannot determine whether the message in question was printed or not, for
console shows no messages.

$ git show f5565295 | patch -p1 # Move to latest commit
$ git show 5ced33bc | patch -p1 -R # Revert only "ARM: 7611/1: VIC: fix bug in VIC irqdomain code"
$ make -s
$ qemu-system-arm

The message in question was printed.

$ git show f5565295 | patch -p1 -R # Also revert "ARM: 7610/1: versatile: bump IRQ numbers"
$ make -s
$ qemu-system-arm

Cannot determine whether the message in question was printed or not, for
console shows no messages.

> When you cannot proceed because you need a patch unrelated to the
> problem you want to solve, you apply that patch on top of the
> bisection HEAD with patch -p1 < foo.patch, test, reset HEAD,
> git bisect <result> etc....
> 
> I'm sorry if this is a bit complicated but bisection down several
> regressions is one of the more advanced forms of regression
> bug hunt... You're quite cool if you can handle it :-)

Sorry, I'm a beginner about git. I already know that 07c9249f is the cause of
"no console messages" and 5c49985c is the fix for "no console messages".
How can I setup a snapshot with 07c9249f and 5c49985c applied (I tried

  $ git reset --hard 07c9249f
  $ git format-patch --stdout 5c49985c^..5c49985c | patch -p1

but patch does not apply cleanly) so that I can find the commit causing
"sym0: request irq 27 failure" which is between 07c9249f and 5c49985c?

Regards.

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Geert Uytterhoeven]

On Tue, Jan 22, 2013 at 9:03 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
> 1.  Get hold of the patch fixing the boot regression:
>     git format-patch
> f5565295892eb93c3191aa241405fe8b685542d6^..f5565295892eb93c3191aa241405fe8b685542d6

git format-patch -1 f5565295892eb93c3191aa241405fe8b685542d6

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Tue, Jan 22, 2013 at 1:25 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> On Tue, Jan 22, 2013 at 9:03 AM, Linus Walleij <linus.walleij@linaro.org> wrote:
>> 1.  Get hold of the patch fixing the boot regression:
>>     git format-patch
>> f5565295892eb93c3191aa241405fe8b685542d6^..f5565295892eb93c3191aa241405fe8b685542d6
>
> git format-patch -1 f5565295892eb93c3191aa241405fe8b685542d6

Ah, I always wondered if there was a better way to do that,
thanks Geert!

Yours,
Linus Walleij

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Tetsuo Handa]

Tetsuo Handa wrote:
> > I'm sorry if this is a bit complicated but bisection down several
> > regressions is one of the more advanced forms of regression
> > bug hunt... You're quite cool if you can handle it :-)
> 
> Sorry, I'm a beginner about git. I already know that 07c9249f is the cause of
> "no console messages" and 5c49985c is the fix for "no console messages".
> How can I setup a snapshot with 07c9249f and 5c49985c applied (I tried
> 
>   $ git reset --hard 07c9249f
>   $ git format-patch --stdout 5c49985c^..5c49985c | patch -p1
> 
> but patch does not apply cleanly) so that I can find the commit causing
> "sym0: request irq 27 failure" which is between 07c9249f and 5c49985c?

I did a blind git bisection (i.e. starting

  $ qemu-system-arm -M versatilepb -hda hda.img -kernel arch/arm/boot/zImage -append "root=/dev/sda1 init=/bin/sh" -nographic

and watching "top" for %CPU usage of qemu-system-arm , assuming that it goes to
100% only if detection of block device for / partition failed and kernel called
panic(), goes to 0% otherwise) in two patterns.

  $ git bisect start HEAD b1112249 v3.7 v3.6 v3.5 v3.4 v3.3 v3.2 v3.1 v3.0 -- arch/arm drivers/scsi/sym53c8xx_2/ drivers/scsi/*.[ch]

  $ git bisect start v3.8-rc1 95e629b7 b8db6b8 810883f0 b10bca0b 14318efb 414a6750e b1112249 v3.7 v3.6 v3.5 v3.4 v3.3 v3.2 v3.1 v3.0

Both patterns resulted in that commit 07c9249f
"ARM: 7554/1: VIC: use irq_domain_add_simple()" is the cause of

  PCI: enabling device 0000:00:0c.0 (0100 -> 0103)
  sym0: <895a> rev 0x0 at pci 0000:00:0c.0 irq 27
  sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
  sym0: request irq 27 failure
  sym0: giving up ...

message.

Would you have a look?

Regards.

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Sat, Jan 26, 2013 at 9:17 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:

> I did a blind git bisection (i.e. starting
>
>   $ qemu-system-arm -M versatilepb -hda hda.img -kernel arch/arm/boot/zImage -append "root=/dev/sda1 init=/bin/sh" -nographic
>
> and watching "top" for %CPU usage of qemu-system-arm , assuming that it goes to
> 100% only if detection of block device for / partition failed and kernel called
> panic(), goes to 0% otherwise) in two patterns.
>
>   $ git bisect start HEAD b1112249 v3.7 v3.6 v3.5 v3.4 v3.3 v3.2 v3.1 v3.0 -- arch/arm drivers/scsi/sym53c8xx_2/ drivers/scsi/*.[ch]
>
>   $ git bisect start v3.8-rc1 95e629b7 b8db6b8 810883f0 b10bca0b 14318efb 414a6750e b1112249 v3.7 v3.6 v3.5 v3.4 v3.3 v3.2 v3.1 v3.0
>
> Both patterns resulted in that commit 07c9249f
> "ARM: 7554/1: VIC: use irq_domain_add_simple()" is the cause of
>
>   PCI: enabling device 0000:00:0c.0 (0100 -> 0103)
>   sym0: <895a> rev 0x0 at pci 0000:00:0c.0 irq 27
>   sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
>   sym0: request irq 27 failure
>   sym0: giving up ...
>
> message.
>
> Would you have a look?

I'm trying to reproduce this, but how do you reconfigure the kernel to
get PCI, SCSI and such stuff enabled?

The stock versatile_defconfig does not even have SCSI enabled...

Yours,
Linus Walleij

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Tetsuo Handa]

Linus Walleij wrote:
> I'm trying to reproduce this, but how do you reconfigure the kernel to
> get PCI, SCSI and such stuff enabled?
> 
> The stock versatile_defconfig does not even have SCSI enabled...

I'm using a customized config for qemu. I've just updated the config to
http://I-love.SAKURA.ne.jp/tmp/config-3.8-rc1-arm .

Regards.

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Sun, Jan 27, 2013 at 1:16 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> Linus Walleij wrote:
>> I'm trying to reproduce this, but how do you reconfigure the kernel to
>> get PCI, SCSI and such stuff enabled?
>>
>> The stock versatile_defconfig does not even have SCSI enabled...
>
> I'm using a customized config for qemu. I've just updated the config to
> http://I-love.SAKURA.ne.jp/tmp/config-3.8-rc1-arm .

Hm when I try this on the v3.7 I get this:

PCI: enabling device 0000:00:0d.0 (0100 -> 0102)
sym0: <895a> rev 0x0 at pci 0000:00:0d.0 irq 28
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym0: SCSI BUS has been reset.

So IRQ 28 instead of 27???

Then the device does not work (no clue why it's like that)
but it's trying IRQ 28 atleast.

On v3.8-rc5 I get this:

PCI: enabling device 0000:00:0d.0 (0100 -> 0103)
sym0: <895a> rev 0x0 at pci 0000:00:0d.0 irq 28
sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
sym0: request irq 28 failure
sym0: giving up ...

OK, so half-guessing I've reproduced part of the problem
atleast...

Yours,
Linus Walleij

Re: [3.8-rc4 arm] SCSI_SYM53C8XX_2 module cannot register IRQ

[Linus Walleij]

On Sun, Jan 20, 2013 at 1:24 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:

> I'm experiencing two regressions in 3.8-rcX.
>
> Regression:
>
>   (1) Console shows nothing, which makes it impossible to determine whether
>       the system is working or not.
>
>   (2) SCSI_SYM53C8XX_2 cannot register IRQ, showing below messages upon load.
>
>       PCI: enabling device 0000:00:0c.0 (0100 -> 0103)
>       sym0: <895a> rev 0x0 at pci 0000:00:0c.0 irq 27
>       sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking
>       sym0: request irq 27 failure
>       sym0: giving up ...

I think I've fixed it now, mailed out a patch. Please test this and confirm
if it fixes your problem!

Yours,
Linus Walleij