EdgeRouter Pro supported? Strange FP problems

EdgeRouter Pro supported? Strange FP problems

[Rob Kendrick]

Hi,

I'm trying to build a kernel for an Ubiquiti EdgeRouter Pro (not a
Lite).  I'm using the current master from linux-mti, and this produces a
kernel that boots and has network support (but bizarrely not activity
LEDs) and USB support, which is good.  However, what I am seeing is
bizarre floating point behavior.

Is there a known issue with master on these Octeon2-based boards?
Should I be pointing my finger of blame at the compiler I've built
(using crosstool-ng) or my configuration of the kernel?

Is there a better choice of compiler and kernel to be using for these
boards?

Thanks for any input,
-- 
Rob Kendrick, Senior Consulting Developer                Codethink Ltd.
Telephone: +44 7880 657 193              302 Ducie House, Ducie Street,
http://www.codethink.co.uk/         Manchester, M1 2JW, United Kingdom.

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/22/2014 02:06 PM, Rob Kendrick wrote:
> Hi,
> 
> I'm trying to build a kernel for an Ubiquiti EdgeRouter Pro (not a
> Lite).  I'm using the current master from linux-mti, and this produces a
> kernel that boots and has network support (but bizarrely not activity
> LEDs) and USB support, which is good.  However, what I am seeing is
> bizarre floating point behavior.
> 
> Is there a known issue with master on these Octeon2-based boards?
> Should I be pointing my finger of blame at the compiler I've built
> (using crosstool-ng) or my configuration of the kernel?
> 
> Is there a better choice of compiler and kernel to be using for these
> boards?
> 
> Thanks for any input,
> 
Hi Rob,

When you say "master from linux-mti" I presume you mean this tree:

http://git.linux-mips.org/?p=linux-mti.git;a=shortlog;h=refs/heads/master

right?

What FPU problem are you seeing exactly? Could you show us a log for the
failure? Can you post a simple test case which will allow someone to
reproduce it?

(I am also surprised you have ethernet support since the ethernet
support has not reached the mainline tree yet as far as I know
https://git.kernel.org/cgit/linux/kernel/git/gregkh/staging.git/commit/?id=ec3a2207c322e518f7f42c80e54b8ecaf8a6f03e).

-- 
markos

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/22/2014 02:06 PM, Rob Kendrick wrote:
> Hi,
> 
> I'm trying to build a kernel for an Ubiquiti EdgeRouter Pro (not a
> Lite).  I'm using the current master from linux-mti, and this produces a
> kernel that boots and has network support (but bizarrely not activity
> LEDs) and USB support, which is good.  However, what I am seeing is
> bizarre floating point behavior.
> 
> Is there a known issue with master on these Octeon2-based boards?
> Should I be pointing my finger of blame at the compiler I've built
> (using crosstool-ng) or my configuration of the kernel?
> 
> Is there a better choice of compiler and kernel to be using for these
> boards?
> 
> Thanks for any input,
> 
Hi Rob,

When you say "master from linux-mti" I presume you mean this tree:

http://git.linux-mips.org/?p=linux-mti.git;a=shortlog;h=refs/heads/master

right?

What FPU problem are you seeing exactly? Could you show us a log for the
failure? Can you post a simple test case which will allow someone to
reproduce it?

(I am also surprised you have ethernet support since the ethernet
support has not reached the mainline tree yet as far as I know
https://git.kernel.org/cgit/linux/kernel/git/gregkh/staging.git/commit/?id=ec3a2207c322e518f7f42c80e54b8ecaf8a6f03e).

-- 
markos

Re: EdgeRouter Pro supported? Strange FP problems

[Rob Kendrick]

On Tue, Jul 22, 2014 at 03:21:34PM +0100, Markos Chandras wrote:
> On 07/22/2014 02:06 PM, Rob Kendrick wrote:
> 
> When you say "master from linux-mti" I presume you mean this tree:
> 
> http://git.linux-mips.org/?p=linux-mti.git;a=shortlog;h=refs/heads/master
> 
> right?

Correct.

> What FPU problem are you seeing exactly? Could you show us a log for the
> failure? Can you post a simple test case which will allow someone to
> reproduce it?

A strange problem with awk (substr() doesn't work in either gawk or
mawk).  I've tried binaries of awk that I've built myself, and binaries
from Debian.  They all fail with my kernel, work fine with the 3.4
kernel (with a load of Cavium patches) that ship with the EdgeRouter
Pro.

I can see it with this awk script:

{
  line = $ 0
  prefix = substr(line, 1, 3)
  suffix = substr(line, 9)
  print "prefix is '" prefix "'"
  print "suffix is '" suffix "'"
}

execute with `echo "foo bar baz" | awk -f test.awk`.  With gawk, I get
both outputs being "foo bar baz" and with mawk "".  Correct answers
occur when using the shipped 3.4.

When you run gawk in linting mode, you get amusing errors like:
	warning: substr: length 3 too big for string indexing,
	truncating to 1.84467e+19

(Both are using doubles as internal representations, AFAICT.)

> (I am also surprised you have ethernet support since the ethernet
> support has not reached the mainline tree yet as far as I know
> https://git.kernel.org/cgit/linux/kernel/git/gregkh/staging.git/commit/?id=ec3a2207c322e518f7f42c80e54b8ecaf8a6f03e).

I was happily NFS-booting.  The activity LEDs don't work, the two
banks of four ports are identified in a different order, and it
complains about the board type being unknown, but otherwise it works.

-- 
Rob Kendrick, Senior Consulting Developer                Codethink Ltd.
Telephone: +44 7880 657 193              302 Ducie House, Ducie Street,
http://www.codethink.co.uk/         Manchester, M1 2JW, United Kingdom.

Re: EdgeRouter Pro supported? Strange FP problems

[John Crispin]

On 22/07/2014 15:06, Rob Kendrick wrote:
> Hi,
> 
> I'm trying to build a kernel for an Ubiquiti EdgeRouter Pro (not a 
> Lite).  I'm using the current master from linux-mti, and this
> produces a kernel that boots and has network support (but bizarrely
> not activity LEDs) and USB support, which is good.  However, what I
> am seeing is bizarre floating point behavior.
> 
> Is there a known issue with master on these Octeon2-based boards? 
> Should I be pointing my finger of blame at the compiler I've built 
> (using crosstool-ng) or my configuration of the kernel?
> 
> Is there a better choice of compiler and kernel to be using for
> these boards?
> 
> Thanks for any input,
> 


Hi Rob,

we had a quite some trouble adding support to openwrt. in the end we
needed a few uclibc patches and gxx4.8 seems utterly foo'ed on this.
gcc 4.6 and 4.9 seem to be running fine though.

what compiler, libc, ... version are you using ?

	John

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/22/2014 03:33 PM, Rob Kendrick wrote:
> 
>> What FPU problem are you seeing exactly? Could you show us a log for the
>> failure? Can you post a simple test case which will allow someone to
>> reproduce it?
> 
> A strange problem with awk (substr() doesn't work in either gawk or
> mawk).  I've tried binaries of awk that I've built myself, and binaries
> from Debian.  They all fail with my kernel, work fine with the 3.4
> kernel (with a load of Cavium patches) that ship with the EdgeRouter
> Pro.
> 
> I can see it with this awk script:
> 
> {
>   line = $ 0
>   prefix = substr(line, 1, 3)
>   suffix = substr(line, 9)
>   print "prefix is '" prefix "'"
>   print "suffix is '" suffix "'"
> }
> 
> execute with `echo "foo bar baz" | awk -f test.awk`.  With gawk, I get
> both outputs being "foo bar baz" and with mawk "".  Correct answers
> occur when using the shipped 3.4.
> 
> When you run gawk in linting mode, you get amusing errors like:
> 	warning: substr: length 3 too big for string indexing,
> 	truncating to 1.84467e+19

there have been quite a few FPU changes in 3.16. I am able to reproduce
the problem on EdgeRouter (non pro) but the result is not the same as
yours. The output using the 'gawk' is:

prefix is ''
suffix is 'baz'

a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
all the new FPU changes, leads to the following bad commit:

08a07904e182895e1205f399465a3d622c0115b8
MIPS: math-emu: Remove most ifdefery.

Reverting the commit is not trivial unfortunately. Perhaps you could use
a v3.15 kernel for now (assuming it boots on that board) or perhaps use
an earlier revision before this bad commit (eg
9e8bad1f9c0370b2635175b34d6151b90a53da5c, which boots and your test
works for me)

[root@buildroot ~]# uname -a
Linux buildroot 3.15.0-rc5-00171-g9e8bad1f9c03 #1280 SMP PREEMPT Tue Jul
22 16:36:19 BST 2014 mips64 GNU/Linux
[root@buildroot ~]# echo "foo bar baz" | awk -f awk-test
prefix is 'foo'
suffix is 'baz'

I have CC'd Ralf who is the author of the commit. Ralf any idea?

-- 
markos

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/22/2014 03:33 PM, Rob Kendrick wrote:
> 
>> What FPU problem are you seeing exactly? Could you show us a log for the
>> failure? Can you post a simple test case which will allow someone to
>> reproduce it?
> 
> A strange problem with awk (substr() doesn't work in either gawk or
> mawk).  I've tried binaries of awk that I've built myself, and binaries
> from Debian.  They all fail with my kernel, work fine with the 3.4
> kernel (with a load of Cavium patches) that ship with the EdgeRouter
> Pro.
> 
> I can see it with this awk script:
> 
> {
>   line = $ 0
>   prefix = substr(line, 1, 3)
>   suffix = substr(line, 9)
>   print "prefix is '" prefix "'"
>   print "suffix is '" suffix "'"
> }
> 
> execute with `echo "foo bar baz" | awk -f test.awk`.  With gawk, I get
> both outputs being "foo bar baz" and with mawk "".  Correct answers
> occur when using the shipped 3.4.
> 
> When you run gawk in linting mode, you get amusing errors like:
> 	warning: substr: length 3 too big for string indexing,
> 	truncating to 1.84467e+19

there have been quite a few FPU changes in 3.16. I am able to reproduce
the problem on EdgeRouter (non pro) but the result is not the same as
yours. The output using the 'gawk' is:

prefix is ''
suffix is 'baz'

a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
all the new FPU changes, leads to the following bad commit:

08a07904e182895e1205f399465a3d622c0115b8
MIPS: math-emu: Remove most ifdefery.

Reverting the commit is not trivial unfortunately. Perhaps you could use
a v3.15 kernel for now (assuming it boots on that board) or perhaps use
an earlier revision before this bad commit (eg
9e8bad1f9c0370b2635175b34d6151b90a53da5c, which boots and your test
works for me)

[root@buildroot ~]# uname -a
Linux buildroot 3.15.0-rc5-00171-g9e8bad1f9c03 #1280 SMP PREEMPT Tue Jul
22 16:36:19 BST 2014 mips64 GNU/Linux
[root@buildroot ~]# echo "foo bar baz" | awk -f awk-test
prefix is 'foo'
suffix is 'baz'

I have CC'd Ralf who is the author of the commit. Ralf any idea?

-- 
markos

Re: EdgeRouter Pro supported? Strange FP problems

[Rob Kendrick]

On Tue, Jul 22, 2014 at 04:38:24PM +0100, Markos Chandras wrote:
> there have been quite a few FPU changes in 3.16. I am able to reproduce
> the problem on EdgeRouter (non pro) but the result is not the same as
> yours. The output using the 'gawk' is:
> 
> prefix is ''
> suffix is 'baz'

This is still, of course, wrong ;)  I'm glad somebody's managed to
reproduce it and it's not just madness on my part.

> a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
> all the new FPU changes, leads to the following bad commit:
> 
> 08a07904e182895e1205f399465a3d622c0115b8
> MIPS: math-emu: Remove most ifdefery.
> 
> Reverting the commit is not trivial unfortunately. Perhaps you could use
> a v3.15 kernel for now (assuming it boots on that board) or perhaps use
> an earlier revision before this bad commit (eg
> 9e8bad1f9c0370b2635175b34d6151b90a53da5c, which boots and your test
> works for me)

I'll give it a whirl; I don't think 3.15 will boot at all on this
hardware, though.

Thanks for your help!
-- 
Rob Kendrick, Senior Consulting Developer                Codethink Ltd.
Telephone: +44 7880 657 193              302 Ducie House, Ducie Street,
http://www.codethink.co.uk/         Manchester, M1 2JW, United Kingdom.

Re: EdgeRouter Pro supported? Strange FP problems

[Aaro Koskinen]

Hi,

On Tue, Jul 22, 2014 at 04:49:14PM +0100, Rob Kendrick wrote:
> On Tue, Jul 22, 2014 at 04:38:24PM +0100, Markos Chandras wrote:
> > there have been quite a few FPU changes in 3.16. I am able to reproduce
> > the problem on EdgeRouter (non pro) but the result is not the same as
> > yours. The output using the 'gawk' is:
> > 
> > prefix is ''
> > suffix is 'baz'
> 
> This is still, of course, wrong ;)  I'm glad somebody's managed to
> reproduce it and it's not just madness on my part.

If your userspace is small, maybe you could try rebuilding it
with softfloat. Your script works on octeon with 3.16-rc6 + gcc 4.9 +
glibc 2.19 + gawk 4.0.3 when compiled with softfloat.

A.

Re: EdgeRouter Pro supported? Strange FP problems

[Rob Kendrick]

On Tue, Jul 22, 2014 at 05:26:46PM +0200, John Crispin wrote:
> we had a quite some trouble adding support to openwrt. in the end we
> needed a few uclibc patches and gxx4.8 seems utterly foo'ed on this.
> gcc 4.6 and 4.9 seem to be running fine though.
> 
> what compiler, libc, ... version are you using ?

mips64-unknown-linux-gnu-gcc (crosstool-NG git+7f1c646) 4.9.0
eglibc 2.15

But this does indeed appear problem with the kernel and not the
toolchain.  Sadly I can't opt for softfloat with this.  I'm going to be
trying to look at the patch that Markos identified; it looks like it may
have mixed up MIPSInst_RT and MIPSInst_FD in a copy-and-paste.
(Guessing, new to MIPS.)

Thanks everyone for your help so far!

-- 
Rob Kendrick, Senior Consulting Developer                Codethink Ltd.
Telephone: +44 7880 657 193              302 Ducie House, Ducie Street,
http://www.codethink.co.uk/         Manchester, M1 2JW, United Kingdom.

Re: EdgeRouter Pro supported? Strange FP problems

[Rob Kendrick]

On Tue, Jul 22, 2014 at 04:38:24PM +0100, Markos Chandras wrote:
> a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
> all the new FPU changes, leads to the following bad commit:
> 
> 08a07904e182895e1205f399465a3d622c0115b8
> MIPS: math-emu: Remove most ifdefery.

This appears to fix the problem:

diff --git a/arch/mips/math-emu/cp1emu.c b/arch/mips/math-emu/cp1emu.c
index 736c17a..bf0fc6b 100644
--- a/arch/mips/math-emu/cp1emu.c
+++ b/arch/mips/math-emu/cp1emu.c
@@ -1827,7 +1827,7 @@ dcopuop:
        case -1:
 
                if (cpu_has_mips_4_5_r)
-                       cbit = fpucondbit[MIPSInst_RT(ir) >> 2];
+                       cbit = fpucondbit[MIPSInst_FD(ir) >> 2];
                else
                        cbit = FPU_CSR_COND;
                if (rv.w)

Sadly I am totally ignorant of kernel patch processes.  What would be
approach to having this properly submitted?

-- 
Rob Kendrick, Senior Consulting Developer                Codethink Ltd.
Telephone: +44 7880 657 193              302 Ducie House, Ducie Street,
http://www.codethink.co.uk/         Manchester, M1 2JW, United Kingdom.

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/23/2014 10:03 AM, Rob Kendrick wrote:
> On Tue, Jul 22, 2014 at 04:38:24PM +0100, Markos Chandras wrote:
>> a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
>> all the new FPU changes, leads to the following bad commit:
>>
>> 08a07904e182895e1205f399465a3d622c0115b8
>> MIPS: math-emu: Remove most ifdefery.
> 
> This appears to fix the problem:
> 
> diff --git a/arch/mips/math-emu/cp1emu.c b/arch/mips/math-emu/cp1emu.c
> index 736c17a..bf0fc6b 100644
> --- a/arch/mips/math-emu/cp1emu.c
> +++ b/arch/mips/math-emu/cp1emu.c
> @@ -1827,7 +1827,7 @@ dcopuop:
>         case -1:
>  
>                 if (cpu_has_mips_4_5_r)
> -                       cbit = fpucondbit[MIPSInst_RT(ir) >> 2];
> +                       cbit = fpucondbit[MIPSInst_FD(ir) >> 2];
>                 else
>                         cbit = FPU_CSR_COND;
>                 if (rv.w)
> 
> Sadly I am totally ignorant of kernel patch processes.  What would be
> approach to having this properly submitted?
> 
Hi Rob,

The patch looks correct. See this document for the process

https://www.kernel.org/doc/Documentation/SubmittingPatches

So in general, you need to create a patch using git-format-patch
and use git-send-email to send it to this mailing list.

-- 
markos

Re: EdgeRouter Pro supported? Strange FP problems

[Markos Chandras]

On 07/23/2014 10:03 AM, Rob Kendrick wrote:
> On Tue, Jul 22, 2014 at 04:38:24PM +0100, Markos Chandras wrote:
>> a quick bisect between v3.15 and v3.16-rc1, which is the first tag with
>> all the new FPU changes, leads to the following bad commit:
>>
>> 08a07904e182895e1205f399465a3d622c0115b8
>> MIPS: math-emu: Remove most ifdefery.
> 
> This appears to fix the problem:
> 
> diff --git a/arch/mips/math-emu/cp1emu.c b/arch/mips/math-emu/cp1emu.c
> index 736c17a..bf0fc6b 100644
> --- a/arch/mips/math-emu/cp1emu.c
> +++ b/arch/mips/math-emu/cp1emu.c
> @@ -1827,7 +1827,7 @@ dcopuop:
>         case -1:
>  
>                 if (cpu_has_mips_4_5_r)
> -                       cbit = fpucondbit[MIPSInst_RT(ir) >> 2];
> +                       cbit = fpucondbit[MIPSInst_FD(ir) >> 2];
>                 else
>                         cbit = FPU_CSR_COND;
>                 if (rv.w)
> 
> Sadly I am totally ignorant of kernel patch processes.  What would be
> approach to having this properly submitted?
> 
Hi Rob,

The patch looks correct. See this document for the process

https://www.kernel.org/doc/Documentation/SubmittingPatches

So in general, you need to create a patch using git-format-patch
and use git-send-email to send it to this mailing list.

-- 
markos