Linux Sparc FPU register corruption

Linux Sparc FPU register corruption

[James Y Knight]

[-- Attachment #1: Type: text/plain, Size: 7497 bytes --]

Those for whom this is a followup: the ASI_BLK_P stores used by memcpy, memset, and bzero in glibc seem to definitely be a problem. And, I'm going to going to have to take back some of my previous words -- this really is looking like a kernel bug (or two) now, rather than a glibc bug as I had thought likely. Oops. :)


For those just joining my tale: I have a Sparc T3-1, with Debian Unstable's Sparc port on it as a guest in a "Oracle VM Server for Sparc" ("ldm") VM. I'm running a recent debian kernel: linux-image-4.0.0-2-sparc64-smp 4.0.4-1. In case you're not familiar, note that Debian has a 64bit kernel, with a 32bit userland, which has been compiled to assume at least sparcv9 CPUs (the so called "v8plus" mode in gcc).

I unfortunately found that sometimes, programs like "gcc" are getting incorrect behavior or segfaults. Aurelian reported the same issue a couple years ago, reporting that it only occurs on UltraSparc T1 and later processors, not earlier UltraSparcs (I have no other Sparc to test on, so I cannot confirm that myself). I wanted to fix the problem, so I could use the machine. And I did -- by getting rid of the sparcv9 optimized asm routines in glibc (going back to the base sparc32 routines), everything seems to be working reliably. The machine became properly usable with Linux for the first time.

I thus guessed that the issue was just a bug in glibc's routines. But, despite having seemingly fixed my problem, not knowing *why* it had been broken bugged me...So, further down this rabbit hole I went...

...And found that the glibc routines were broken because the floating point registers they were blatting to memory sometimes got replaced with garbage, not because of anything they were doing wrong themselves. D'oh!

Attached is a simple reproduction case, not involving the glibc functions, which exhibits the same FP register corruption problem I see when using the memset routine in glibc's sysdeps/sparc/sparc32/sparcv9/memset.S.

For comparison, this test program appears to runs perfectly reliably under Solaris on the same hardware.

Problem 1 - Lower 16 fp regs (%f0 - %f31) getting overwritten with garbage.
=========
Basically, it appears that if you do:
  stda %f0, [%0] ASI_BLK_P
...and the address you're storing into causes a fault to the kernel (e.g. in the example, it's a freshly mmapped zero-page), then, SOMETIMES, the first 16 fp registers seem to get trashed.

I'm guessing the kernel uses the FPU internally, and then forgets to restore the registers upon return from the trap handler sometimes, for some reason? But why only for this instruction? (Does the kernel care which instruction caused the page fault trap? Or, could it be a hardware bug that's trashing the fpu registers, not a kernel bug? But since it works in Solaris, probably not?)

If you then do a syscall like usleep, these registers are *usually* restored to their previous, valid, contents. This appears to be since I didn't modify any fp registers, so they're not marked as dirty, and thus the corrupted versions never get saved to the kernel thread_info struct on the way into the usleep. Then, on resume of the task after the usleep, the correct register contents get properly restored from the saved location. (I've verified that touching a low fp reg before the usleep *does* cause the bogus values to be preserved across the usleep call).

Here's some typical looking output from running:
  gcc ~/test-err.c
  seq 64 | xargs -n1 -P64 /bin/sh -c 'while ./a.out; do : ; done'

(I have 64 vCPUs assigned to linux, that's why 64 parallelism). This typically reproduces the issue for me in 10 seconds or less.

FP regs xx: 0: 0xffff8005784d5600 0xffff800554ff3500 0xffff800088091900 0xffff800dd7dda600 0xffff800de669ee00 0xffff800dd865d800 0xffff800dd5c18500 0xffff800de6717500
FP regs xx: 1: 0xffff800088091900 0xffff800dd7dda600 0xffff800de669ee00 0xffff800dd865d800 0xffff800dd5c18500 0xffff800de6717500 0xffff8005784d5600 0xffff800554ff3500
FP regs xx: 2: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs xx: 3: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708

FP regs yy: 0: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs yy: 1: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs yy: 2: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs yy: 3: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708

Note that all regs were supposed to be 0x102030405060708, but that the first 16 are trashed in "xx", and then corrected in "yy" ("yy" being the second time I read them, after the usleep). BTW, all those garbage register values sure do look suspiciously like they might be kernel memory addresses, don't you think? ("MM: PAGE_OFFSET is 0xffff800000000000 (max_phys_bits == 43)") The values that appear are not always the same. They also don't always look like kernel memory addresses, but most often, they do.

Anyhow -- #define USE_BLKSTORE 0 in the test program "fixes" this problem. If I use a normal store instruction instead of the block store, the lower 16 fp regs seem always reliably preserved.


Problem 2 - *Upper* 16 fp regs (%f32 - %f63) getting replaced by zeros.
=========

Additionally, while creating the test case for Problem 1, I've found what at first glance appears to be a separate bug -- potentially a worse one, since it can affect code that doesn't use ASI_BLK_P, so it's not as easy to fix. (Although...this issue seems to occur *much* less often, sometimes taking a few minutes to reproduce). In the same example program as before, once I've "#define USE_BLKSTORE 0", sometimes I see the upper 16 regs have all turned into zeros! (This can also be observed with USE_BLOCKSTORE=1, too, it's just usually overshadowed by the more frequent other issue). In either case, the usleep call doesn't restore the lost upper registers.

E.g., I'll get this output:

FP regs xx: 0: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs xx: 1: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs xx: 2: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0
FP regs xx: 3: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

FP regs yy: 0: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs yy: 1: 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708 0x102030405060708
FP regs yy: 2: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0
FP regs yy: 3: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0


In summary:
Always: upper 16 fp regs sometimes zeroed.
Using ASI_BLK_P: lower 16 fp regs often corrupted.
Without ASI_BLK_P: lower 16 fp regs (seemingly) always preserved.

Any ideas?


[-- Attachment #2: test-err.c --]
[-- Type: application/octet-stream, Size: 3537 bytes --]

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>

#define USE_BLKSTORE 1

static void set_fp_regs(long long val) {
  long long *pval = &val;
  asm volatile (
"ldd [%0], %%f0; ldd [%0], %%f2; ldd [%0], %%f4; ldd [%0], %%f6; ldd [%0], %%f8;"
"ldd [%0], %%f10; ldd [%0], %%f12; ldd [%0], %%f14; ldd [%0], %%f16; ldd [%0], %%f18;"
"ldd [%0], %%f20; ldd [%0], %%f22; ldd [%0], %%f24; ldd [%0], %%f26; ldd [%0], %%f28;"
"ldd [%0], %%f30; ldd [%0], %%f32; ldd [%0], %%f34; ldd [%0], %%f36; ldd [%0], %%f38; "
"ldd [%0], %%f40; ldd [%0], %%f42; ldd [%0], %%f44; ldd [%0], %%f46; ldd [%0], %%f48; "
"ldd [%0], %%f50; ldd [%0], %%f52; ldd [%0], %%f54; ldd [%0], %%f56; ldd [%0], %%f58; "
"ldd [%0], %%f60; ldd [%0], %%f62" : : "r"(pval));
}

static void get_fp_regs(long long *xx) {
  asm volatile (
"std %%f0,  [%0 + 0x00]; std %%f2, [%0 + 0x08]; "
"std %%f4, [%0 + 0x10]; std %%f6, [%0 + 0x18]; "
"std %%f8, [%0 + 0x20]; std %%f10, [%0 + 0x28]; "
"std %%f12, [%0 + 0x30]; std %%f14, [%0 + 0x38]; "
"std %%f16, [%0 + 0x40]; std %%f18, [%0 + 0x48]; "
"std %%f20, [%0 + 0x50]; std %%f22, [%0 + 0x58]; "
"std %%f24, [%0 + 0x60]; std %%f26, [%0 + 0x68]; "
"std %%f28, [%0 + 0x70]; std %%f30, [%0 + 0x78]; "
"std %%f32, [%0 + 0x80]; std %%f34, [%0 + 0x88]; "
"std %%f36, [%0 + 0x90]; std %%f38, [%0 + 0x98]; "
"std %%f40, [%0 + 0xa0]; std %%f42, [%0 + 0xa8]; "
"std %%f44, [%0 + 0xb0]; std %%f46, [%0 + 0xb8]; "
"std %%f48, [%0 + 0xc0]; std %%f50, [%0 + 0xc8]; "
"std %%f52, [%0 + 0xd0]; std %%f54, [%0 + 0xd8]; "
"std %%f56, [%0 + 0xe0]; std %%f58, [%0 + 0xe8]; "
"std %%f60, [%0 + 0xf0]; std %%f62, [%0 + 0xf8]" : : "r"(xx) : "memory");
}

static void print_fp_regs(char *prefix, long long *xx) {
  printf("%s0: 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx\n",
	 prefix, xx[0], xx[1], xx[2], xx[3], xx[4], xx[5], xx[6], xx[7]);
  printf("%s1: 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx\n",
	 prefix, xx[8], xx[9], xx[10], xx[11], xx[12], xx[13], xx[14], xx[15]);
  printf("%s2: 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx\n",
	 prefix, xx[16], xx[17], xx[18], xx[19], xx[20], xx[21], xx[22], xx[23]);
  printf("%s3: 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx 0x%llx\n",
	 prefix, xx[24], xx[25], xx[26], xx[27], xx[28], xx[29], xx[30], xx[31]);
  printf("\n");
}

int main() {
  void *res = mmap(NULL, 0x14a000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
  if (res == MAP_FAILED || ((long)res & 0xfff) != 0) {
    printf("Unexpected address %p", res);
    return 1;
  }

  long long __attribute__((aligned (256))) xx[32];
  long long __attribute__((aligned (256))) yy[32];
      
  set_fp_regs(0x0102030405060708);
#if USE_BLKSTORE
  int *mem;
  for (mem = (int*)res; mem < (int*)((char*)res + 0x140000); mem += 16) {
    // 0xf0 == ASI_BLK_P
    asm volatile("stda %%f0, [%0] 0xf0" : : "r"(mem) : "memory");
  }
#else
  int *mem;
  for (mem = (int*)res; mem < (int*)((char*)res + 0x140000); mem += 16) {
    *mem = 0;
  }
  asm volatile("" : : "r"(mem) : "memory");
#endif
  get_fp_regs(xx);

  int i;
  for(i = 0; i < 32; ++i) {
    if (xx[i] != 0x0102030405060708) {
      // Modifying a low-fp register causes the usleep to not restore the values.
      // asm volatile("fsrc2 %f32, %f0");
      // Calling usleep causes the fp registers to regain their proper values.
      usleep(10);
      get_fp_regs(yy);
      print_fp_regs("FP regs xx: ", xx);
      print_fp_regs("FP regs yy: ", yy);
      abort();
    }
  }
    
  return 0;
}


[-- Attachment #3: Type: text/plain, Size: 5284 bytes --]



James

On Jun 5, 2015, at 5:39 PM, Patrick Baggett <baggett.patrick@gmail.com> wrote:

> I would ask David S. Miller about the sparc ASM stuff - he seems to be the resident sparc genius and linux kernel maintainer.
> 
> On Fri, Jun 5, 2015 at 4:18 PM, James Y Knight <jyknight@google.com> wrote:
> 
>> On Jun 4, 2015, at 11:07 AM, James Y Knight <jyknight@google.com> wrote:
>>> GLibc
>>> =====
>>> After that, everything seemed to be going fine, except that programs like GCC would randomly segfault and give parse errors. This has been reported before, e.g. http://thread.gmane.org/gmane.linux.ports.sparc/16835, from 2 years ago. Things were stable enough to use interactively, if you're willing to keep retrying a build until it works, but not stable enough to use for any autobuild system.
>>> 
>>> After a getting a hint from Aurelien that disabling optimized memcpy routines in glibc (eglibc 2.19-1, on Wed, 04 Jun 2014 20:32:06 +0200) had improved, but did not fix, the problem, I started looking into that....
>>> 
>>> ...And found that recompiling glibc, disabling the sparcv9 optimizations (that is: eliminating debian/patches/sparc/local-sparcv9-target.diff), *appears* to have completely fixed the stability issue!
>>> 
>>> To try to verify that, I ran a loop building and rebuilding 'clang' (with full "ninja" parallelism) overnight, and it's had zero crashes in all 14 builds of clang that it got through. Prior to fixing glibc, at least one of the ~2300 build steps (gcc/as/ld) was sure to crash unreproducibly.
>>> 
>>> It'd be great if someone wants to try to figure out exactly /which/ of the asm routines in the various sysdeps/**/sparc32/sparcv9 are broken, to narrow down the problem better, too. I highly suspect there's just something wrong in one or more of the hand-written asm files, but it's certainly possible there's some wider problem that the sparcv9 optimizations of glibc (but nothing else I've seen so far), just happens to expose.
>> 
>> So, bad news and good news: 
>> 
>> Bad News: the above solution of simply disabling sparcv9 breaks some things (other than gcc). It breaks something about atomics or semaphores, likely due to a mismatch of expectations between libc and other things (the sparc32 routines, when *NOT* compiled in a shared library, dynamically choose between the v8 and v9 ways of doing things, so it's entirely reasonable to assume that doing it the v8 way cannot work right).
>> 
>> Good News:
>> 
>> My next attempt at a fix, is to just disable the optimized string ops:
>>  rm sysdeps/sparc/sparc32/sparcv9/*mem* sysdeps/sparc/sparc32/sparcv9/*st*
>> That seems to still have fixed the random gcc crashes, AND doesn't break other things. :)
>> 
>> 
>> Looking into what the deleted routines are doing that's "interesting":
>> 
>> * memcpy and memset:
>> 
>> They're using LDBLOCKF STBLOCKF "block copy" instructions, which are:
>> 1) Not actually part of the Sparcv9 standard instruction set, but rather are processor-specific (Although, these processor-specific instructions have been implemented since the UltraSPARC I).
>> "The LDBLOCKF instruction is intended to be a processor-specific instruction, which may or may not be implemented in future Oracle SPARC Architecture implementations. Therefore, it should only be used in platform-specific dynamically-linked libraries or in software created by a runtime code generator that is aware of the specific virtual processor implementation on which it is executing."
>> 
>> 2) Marked deprecated.
>> "The LDBLOCKF instructions are deprecated and should not be used in new software. A sequence of LDDF instructions should be used instead."
>> 
>> 3) Don't follow the normal TSO memory model ordering that everything else does; they require explicit MEMBARs in the right places to ensure even *single-thread/cpu* memory ordering correctness.
>> "Block operations do not generally conform to dependence order on the issuing virtual processor; that is, no read-after-write or write-after-read checking occurs between block loads and stores. Explicit MEMBARs are required to enforce dependence ordering between block operations that reference the same address."
>> 
>> It certainly looks like the author of those routines *tried* to do the right thing w.r.t. inserting membar instructions in the right place, but I can easily imagine it's wrong somehow. And it is entirely plausible that the behavior would be hardware-generation specific, since it has, by design, weird hardware-specific memory semantics. I'm placing my bets on this one being the problem.
>> 
>> * memchr, memcmp, strcmp, strcpy, etc.
>> 
>> These are using a nonfaulting load instruction. The nonfaulting load doesn't actually mean the hardware doesn't fault on loading from an unmapped page. Actually, unmapped pages still cause a fault, but the fault is supposed to be handled by the OS. It's also possible to map pages as "for use by nonfaulting loads only" (linux doesn't appear to do this).
>> 
>> That's a rare instruction -- not generated by GCC I think, so I could imagine there being a bug in the fault handler for it. I think that's less likely though, since it doesn't seem like it'd be CPU-architecture specific.
>> 
>> James
>> 
>> 

Re: Linux Sparc FPU register corruption

[David Miller]

Your test is faulty.

You cannot use ASI_BLK_P loads or stores without appropriate memory
barriers around them.

The rules for when and where you need the memory barriers are
complicated, especially if you want to incur the cost of the memory
barrier as infrequently as possible.

Re: Linux Sparc FPU register corruption

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 09 Jun 2015 00:34:14 -0700 (PDT)

> Your test is faulty.
> 
> You cannot use ASI_BLK_P loads or stores without appropriate memory
> barriers around them.
> 
> The rules for when and where you need the memory barriers are
> complicated, especially if you want to incur the cost of the memory
> barrier as infrequently as possible.

FWIW, you're probably hitting the bug fixed by the following commit in
glibc:

commit 834caf06f33d79be54cff63c274fba2845513593
Author: Jose E. Marchesi <jose.marchesi@oracle.com>
Date:   Sat May 17 11:20:27 2014 -0700

    Fix sparc memcpy data corruption when using niagara2 optimized routines.
    
        * sysdeps/sparc/sparc64/multiarch/memcpy-niagara2.S: Add missing
        membar to avoid block loads/stores to overlap previous stores.

Re: Linux Sparc FPU register corruption

[James Y Knight]

From: David Miller <davem@davemloft.net>
Date: Tue, 09 Jun 2015 00:34:14 -0700 (PDT)

> Your test is faulty.
> 
> You cannot use ASI_BLK_P loads or stores without appropriate memory
> barriers around them.

Um, but my test isn't testing what is being stored to memory at all. It is storing to memory and **never loading from the memory after**. Why would writing FROM fp registers TO memory corrupt the *registers* due to a missing memory barrier?

If problem 1 is due to a missing membar somewhere, I think it's surely gotta be the kernel that's missing it, not my test code. No?

> FWIW, you're probably hitting the bug fixed by the following commit in
> glibc:
> 
> commit 834caf06f33d79be54cff63c274fba2845513593
> Author: Jose E. Marchesi <jose.marchesi@oracle.com>
> Date:   Sat May 17 11:20:27 2014 -0700
> 
>    Fix sparc memcpy data corruption when using niagara2 optimized routines.
> 
>        * sysdeps/sparc/sparc64/multiarch/memcpy-niagara2.S: Add missing
>        membar to avoid block loads/stores to overlap previous stores.
> 

Debian glibc has multiarch support disabled (done a couple years ago to try to workaround the unreliability, not entirely successfully..), so it's not using that routine you mention. It's using sysdeps/sparc/sparc32/sparcv9/memcpy.S which points to sysdeps/sparc/sparc64/memcpy.S

[sorry for the duplicate post -- didn't realize vger.kernel.org doesn't accept emails sent by the android gmail client]

Re: Linux Sparc FPU register corruption

[David Miller]

From: James Y Knight <jyknight@google.com>
Date: Tue, 9 Jun 2015 08:07:11 -0400

> Debian glibc has multiarch support disabled (done a couple years ago to try
> to workaround the unreliability, not entirely successfully..), so it's not
> using that routine you mention. It's using
> sysdeps/sparc/sparc32/sparcv9/memcpy.S which points to
> sysdeps/sparc/sparc64/memcpy.S

I bet you stop getting corruptions if you use the appropriate optimized
routine.

Re: Linux Sparc FPU register corruption

[David Miller]

From: James Y Knight <jyknight@google.com>
Date: Tue, 9 Jun 2015 08:13:58 -0400

> Um, but my test isn't testing what is being stored to memory at
> all. It is storing to memory and **never loading from the memory
> after**. Why would writing FROM fp registers TO memory corrupt the
> *registers* due to a missing memory barrier?

The memory barrier is necessary for two reasons, only one of them is
to handle the asynchronousness of the memory operations.

The other reason is that there are strict rules for accessing the FPU
register file around block loads and stores.

Therefore if you don't do the proper memory barriers you can get
corrupted FPU register as well as memory contents.

And complicating things even more, what you can get away with is
different on every single cpu variant.  That's why I really wish
debian didn't disable multiarch as that makes the code use the
UltraSPARC-I/II/III memcpy, which might not be %100 kosher on
Niagara-T1 and later.

Anyways, please put proper memory barriers into your testcase and
let's see if the problem still triggers.

Thanks.

Re: Linux Sparc FPU register corruption

[David Miller]

Just FYI, I commented out the usleep() in your test program and have
been running your:

seq 64 | xargs -n1 -P64 /bin/sh -c 'while ./a.out; do : ; done'

test, and it's been running flawlessly for 2 hours on my T4.

This is with Linus's current tree.

I'll fire up my T3 later and try to reproduce it there.

Re: Linux Sparc FPU register corruption

[James Y Knight]

On Jun 9, 2015, at 3:02 PM, David Miller <davem@davemloft.net> wrote:
> Anyways, please put proper memory barriers into your testcase and
> let's see if the problem still triggers.

Okay, I edited my test case changing the asm to "membar #Sync; stda %%f0, [%0] 0xf0; membar #Sync". That should be entirely over-doing it, right? Problem still occurs within seconds.

And, don't forget Problem #2 -- which didn't even involve weird ASIs at all (at least, not in the test task itself).

On Jun 9, 2015, at 4:45 PM, David Miller <davem@davemloft.net> wrote:
> Just FYI, I commented out the usleep() in your test program and have
> been running your:
> 
> seq 64 | xargs -n1 -P64 /bin/sh -c 'while ./a.out; do : ; done'
> 
> test, and it's been running flawlessly for 2 hours on my T4.
> 
> This is with Linus's current tree.
> 
> I'll fire up my T3 later and try to reproduce it there.

Gah! Thanks for testing it on your setup!...but that's really unfortunate that you can't reproduce the problem!

What distro are you running? Maybe I should try the same thing you're using...

Anyone else reading this able to reproduce problems with my test program (or not?)

Re: Linux Sparc FPU register corruption

[David Miller]

From: James Y Knight <jyknight@google.com>
Date: Tue, 9 Jun 2015 17:54:10 -0400

> What distro are you running? Maybe I should try the same thing you're using...

Debian 7.0

Re: Linux Sparc FPU register corruption

[Aurelien Jarno]

On 2015-06-09 12:02, David Miller wrote:
> From: James Y Knight <jyknight@google.com>
> Date: Tue, 9 Jun 2015 08:13:58 -0400
> 
> > Um, but my test isn't testing what is being stored to memory at
> > all. It is storing to memory and **never loading from the memory
> > after**. Why would writing FROM fp registers TO memory corrupt the
> > *registers* due to a missing memory barrier?
> 
> The memory barrier is necessary for two reasons, only one of them is
> to handle the asynchronousness of the memory operations.
> 
> The other reason is that there are strict rules for accessing the FPU
> register file around block loads and stores.
> 
> Therefore if you don't do the proper memory barriers you can get
> corrupted FPU register as well as memory contents.
> 
> And complicating things even more, what you can get away with is
> different on every single cpu variant.  That's why I really wish

So it means the userland code doesn't run the same on the various
CPU. How are we supposed to do with static binaries?

> debian didn't disable multiarch as that makes the code use the
> UltraSPARC-I/II/III memcpy, which might not be %100 kosher on
> Niagara-T1 and later.

The UltraSPARC-I/II/III memcpy code might have issues, but it clearly
works a lot better than the Niagara-T1 code on a Niagara-T1 machine.
Disabling multiarch support improves a lot the stability on these
machines.

Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: Linux Sparc FPU register corruption

[David Miller]

From: Aurelien Jarno <aurelien@aurel32.net>
Date: Wed, 10 Jun 2015 09:50:06 +0200

> So it means the userland code doesn't run the same on the various
> CPU. How are we supposed to do with static binaries?

Multiarch works perfectly fine in static binaries, just the same as it
does with dynamically linked executables.

Normally static binaries do not use PLT entries, but with multiarch
it does, so that the proper routine can be resolved at run time just
as it would via the dynamic linker.

> Disabling multiarch support improves a lot the stability on these
> machines.

By disabling it you are creating an even worse situation, for the
reasons I've discussed already, plus guess what I test when I'm
doing development?

Re: Linux Sparc FPU register corruption

[Aurelien Jarno]

On 2015-06-10 02:18, David Miller wrote:
> > Disabling multiarch support improves a lot the stability on these
> > machines.
> 
> By disabling it you are creating an even worse situation, for the
> reasons I've discussed already, plus guess what I test when I'm
> doing development?

How could it be worse? With the Niagara T1 memcpy routines, the machine
is not usable, as the processes crashes regularly with a segmentation
fault. With the default sparc v9 memcpy routines, the machine becomes
usable.

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Re: Linux Sparc FPU register corruption

[James Y Knight]

> On Jun 10, 2015, at 5:18 AM, David Miller <davem@davemloft.net> wrote:
> 
> From: Aurelien Jarno <aurelien@aurel32.net>
> Date: Wed, 10 Jun 2015 09:50:06 +0200
> 
>> So it means the userland code doesn't run the same on the various
>> CPU. How are we supposed to do with static binaries?
> 
> Multiarch works perfectly fine in static binaries, just the same as it
> does with dynamically linked executables.
> 
> Normally static binaries do not use PLT entries, but with multiarch
> it does, so that the proper routine can be resolved at run time just
> as it would via the dynamic linker.
> 
>> Disabling multiarch support improves a lot the stability on these
>> machines.
> 
> By disabling it you are creating an even worse situation, for the
> reasons I've discussed already, plus guess what I test when I'm
> doing development?

There's really no point in arguing about enabling/disabling multiarch glibc in Debian right now... It seems clear there's a bug -- or two -- in the kernel. I'm really having trouble imagining how it could *possibly* be glibc's fault that kernel addresses are randomly appearing in the floating point registers in my test program. So, let's just figure out how to fix the problem first...

Once the reliability bug is fixed, then there will be no argument -- nobody will be against enabling the sparc multiarch routines in Debian's glibc then.

Also, given the behavior of the ASI_BLK_P stores in my test program, it seems hard to imagine that memcpy-niagara2.S could possibly be reliable, since it also uses the same "stda src, [addr] ASI_BLK_P" instruction and will run into the same random errors upon pagefault.


But separately from the reliability issue, it seems rather unfortunate that the 'default' sparcv9 and sparc64 routines aren't actually coded to the base sparcv9 standard instruction set. It seems like probably the base routines should limit themselves to normal LDX/STX or LDDF/STDF instructions, and leave things like LDBLOCKF (which the docs mark CPU-specific, and deprecated, and potentially to be removed from future chips), for when a specific processor is targeted.

Re: Linux Sparc FPU register corruption

[David Miller]

From: Aurelien Jarno <aurelien@aurel32.net>
Date: Wed, 10 Jun 2015 11:33:14 +0200

> On 2015-06-10 02:18, David Miller wrote:
>> > Disabling multiarch support improves a lot the stability on these
>> > machines.
>> 
>> By disabling it you are creating an even worse situation, for the
>> reasons I've discussed already, plus guess what I test when I'm
>> doing development?
> 
> How could it be worse? With the Niagara T1 memcpy routines, the machine
> is not usable, as the processes crashes regularly with a segmentation
> fault. With the default sparc v9 memcpy routines, the machine becomes
> usable.

I do not think this will happen with current kernel and glibc and
all the bug fixes I've been doing over the past few years.

Re: Linux Sparc FPU register corruption

[David Miller]

From: James Y Knight <jyknight@google.com>
Date: Wed, 10 Jun 2015 10:40:58 -0400

> But separately from the reliability issue, it seems rather
> unfortunate that the 'default' sparcv9 and sparc64 routines aren't
> actually coded to the base sparcv9 standard instruction set. It
> seems like probably the base routines should limit themselves to
> normal LDX/STX or LDDF/STDF instructions, and leave things like
> LDBLOCKF (which the docs mark CPU-specific, and deprecated, and
> potentially to be removed from future chips), for when a specific
> processor is targeted.

All sparc64 cpus support block loads and stores.

There are documents, and then there is reality.

Those instructions cannot be removed without breaking tons of code
out there and the people making changes to the sparc64 cpus are
painfully aware of this.

Re: Linux Sparc FPU register corruption

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 09 Jun 2015 13:45:37 -0700 (PDT)

> Just FYI, I commented out the usleep() in your test program and have
> been running your:
> 
> seq 64 | xargs -n1 -P64 /bin/sh -c 'while ./a.out; do : ; done'
> 
> test, and it's been running flawlessly for 2 hours on my T4.
> 
> This is with Linus's current tree.
> 
> I'll fire up my T3 later and try to reproduce it there.

Good news, I can reproduce it on my T3.

I'll try to debug this.

Re: Linux Sparc FPU register corruption

[David Mattli]

I've submitted a bug to Debian against src:linux requesting that they
enable SLUB on the sparc kernel because in my testing a SLUB kernel
never shows an error with Knight's test program and a SLAB kernel shows
it within seconds.

Enabling SLUB also fixes the nettle FTBFS.

I'd appreciate if you can help by making this change to the linux
package, adding information to the bug or by confirming whether the
proposed workaround works for you.

Here's the src:linux bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bugx9180

And the nettle FTBFS bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bugx9013

Thanks!
David Mattli

Re: Linux Sparc FPU register corruption

[David Miller]

From: David Mattli <dmm@mattli.us>
Date: Thu, 18 Jun 2015 11:51:23 -0500

> 
> I've submitted a bug to Debian against src:linux requesting that they
> enable SLUB on the sparc kernel because in my testing a SLUB kernel
> never shows an error with Knight's test program and a SLAB kernel shows
> it within seconds.
> 
> Enabling SLUB also fixes the nettle FTBFS.
> 
> I'd appreciate if you can help by making this change to the linux
> package, adding information to the bug or by confirming whether the
> proposed workaround works for you.

There is no reason why SLUB vs. SLAB can be the real cause of
this problem.

It is definitely the FPU register handling.

I've already root caused the bug.  Specifically if I change the
memcpy in the kernel to one that doesn't use the FPU, the test
case passes just fine.

I'm working on a fix but this and next week I am travelling.
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in

Re: Linux Sparc FPU register corruption

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Wed, 10 Jun 2015 13:22:07 -0700 (PDT)

> Good news, I can reproduce it on my T3.
> 
> I'll try to debug this.

Just wanted to give a status update on this.  I know what is going
wrong and am working on coming up with a fix.  This is a two decades
old bug.

I wrote a sophisticated tracing mechanism to load all of the FPU state
saves and restores and a special software trap that dumps kernel FPU
saving state which is triggerable from the testcase.

What's really important is to generate this state dump exactly when
the FPU corruption first occurs.  So in the testcase I made it jump
out to the error reporting code from within the inner loop of stores.
So it looks like this:

==========
#if USE_BLKSTORE
  int *mem;
  for (mem = (int*)res; mem < (int*)((char*)res + 0x140000); mem += 16) {
    // 0xf0 = ASI_BLK_P                                                                                                                                                       
    asm volatile("stda %%f0, [%0] 0xf0" : : "r"(mem) : "memory");
    asm volatile("std %%f0, [%0]" : : "r" (&xx[0]) : "memory");
    if (xx[0] != 0x0102030405060708)
            goto do_bug;
  }
#else
 ...
do_bug:
  get_fp_regs(xx);

  int i;
  for(i = 0; i < 32; ++i) {
    if (xx[i] != 0x0102030405060708) {
      // Modifying a low-fp register causes the usleep to not restore the values.                                                                                              
      // asm volatile("fsrc2 %f32, %f0");                                                                                                                                      
      // Calling usleep causes the fp registers to regain their proper values.                                                                                                 
      //usleep(10);                                                                                                                                                            
      __asm__ __volatile__("ta 0x0f");
==========

The "ta 0x0f" is the special software trap that dumps state into the kernel
log.  Part of what gets dumped is a ring buffer of FPU save/restore events,
it looks like this:

FPU fpsaved[0x7] gsr[0x6] xfsr[0x0]
FREGS0 [0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708]
FREGS1 [0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708]
FREGS2 [0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708]
FREGS3 [0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708:0102030405060708]
FREGS4 [0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000]
FREGS5 [0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000]
FREGS6 [0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000]
FREGS7 [0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000:0000000000000000]
FPU_HIST ent(7) [41 03 09 4a 09 4b 08 08]

It dumps the fsr/gsr saved at the top level of the FPU state stack, and the FPU registers
from the top two FPU state stack slots.  Then it dumps the FPU event history log.

The above log encodes to:

==========
        ETRAP
                VIS_ENTRY(fprs=0x4)
                VIS_EXIT
                RTRAP
                RTRAP_KFPU_RESTORE(fpu_saved=0x4)
        RTRAP
        RTRAP_UFPU(fprs=0x4)
ETRAP
==========

The final ETRAP is the one generated by the application for the "ta 0x0f"
instruction.

At a high level the FPU saving code is trying to minimize the amount
of register saves we do when the kernel wants to use the FPU
internally _or_ we're doing a context switch.  The cpu helps us by
having two dirty bits in the %fprs register, one for the lower half of
the fpu (FPRS_DL) and one for the upper half (FPRS_DU).  This register
also has a "float enable" bit (FPRS_FEF).

The only complex aspect of this dirty register state mechanism is that
there is no dirty bit for the %gsr and %fsr registers, if the enable
bit is set at all we have to assume that both of these register might
be dirty.

Each context of execution needs a place to store the FPU registers,
and there needs to be an event which triggers the restoration of the
saved FPU register state.  These two jobs are done by trap entry and
exit.

Trap entry allocates FPU save area slots by incrementing a counter
(current_thread_info()->fpdepth).  Trap return looks to see if any FPU
state was saved during that trap handler and if so writes that saved
FPU state back into the chip.

The problem occurs when we have a situation like trapping from
userspace with just the FPRS_FEF bit set in the %fprs.  This says
that no registers (besides %fsr and %gsr) are dirty.

If we then trap again into the kernel, and use the FPU, we won't
have to save any of the FPU register set, just %fsr and %gsr.
So on trap return the code only restores %fsr and %gsr, because
that is what we determined needed to be saved.

Then we return back into userspace, without restoring the user FPU
registers from the top-level FPU state saving stack.

The simplest fix is to save all registers when we are using anything
deeper than the top-most FPU state saving stack slot.  A patch
implementing that is below and works for the test case.

There are some related similar issues, especially in VISenterhalf,
that I want to address as well since it could cause similar
corruptions.

I'll post when I have a final patch for everything, thanks for your
patience.

diff --git a/arch/sparc/lib/VISsave.S b/arch/sparc/lib/VISsave.S
index b320ae9..eb11136 100644
--- a/arch/sparc/lib/VISsave.S
+++ b/arch/sparc/lib/VISsave.S
@@ -44,9 +44,8 @@ vis1:	ldub		[%g6 + TI_FPSAVED], %g3
 
 	 stx		%g3, [%g6 + TI_GSR]
 2:	add		%g6, %g1, %g3
-	cmp		%o5, FPRS_DU
-	be,pn		%icc, 6f
-	 sll		%g1, 3, %g1
+	mov		FPRS_DU | FPRS_DL | FPRS_FEF, %o5
+	sll		%g1, 3, %g1
 	stb		%o5, [%g3 + TI_FPSAVED]
 	rd		%gsr, %g2
 	add		%g6, %g1, %g3
@@ -81,25 +80,6 @@ vis1:	ldub		[%g6 + TI_FPSAVED], %g3
 80:	jmpl		%g7 + %g0, %g0
 	 nop
 
-6:	ldub		[%g3 + TI_FPSAVED], %o5
-	or		%o5, FPRS_DU, %o5
-	add		%g6, TI_FPREGS+0x80, %g2
-	stb		%o5, [%g3 + TI_FPSAVED]
-
-	sll		%g1, 5, %g1
-	add		%g6, TI_FPREGS+0xc0, %g3
-	wr		%g0, FPRS_FEF, %fprs
-	membar		#Sync
-	stda		%f32, [%g2 + %g1] ASI_BLK_P
-	stda		%f48, [%g3 + %g1] ASI_BLK_P
-	membar		#Sync
-	ba,pt		%xcc, 80f
-	 nop
-
-	.align		32
-80:	jmpl		%g7 + %g0, %g0
-	 nop
-
 	.align		32
 VISenterhalf:
 	ldub		[%g6 + TI_FPDEPTH], %g1

Re: Linux Sparc FPU register corruption

[Sam Ravnborg]

Hi Davem.

> Just wanted to give a status update on this.  I know what is going
> wrong and am working on coming up with a fix.  This is a two decades
> old bug.

Thanks for the informative post - keep up the good work!

	Sam