unaligned accesses in SLAB etc.

unaligned accesses in SLAB etc.

[David Miller]

I'm getting tons of the following on sparc64:

[603965.383447] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603965.396987] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
[603965.410523] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603965.424061] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
[603965.437617] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603970.554394] log_unaligned: 333 callbacks suppressed
[603970.564041] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603970.577576] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
[603970.591122] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603970.604669] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
[603970.618216] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
[603976.515633] log_unaligned: 31 callbacks suppressed
[603976.525092] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603976.540196] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603976.555308] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603976.570411] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603976.585526] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603982.476424] log_unaligned: 43 callbacks suppressed
[603982.485881] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
[603982.501590] Kernel unaligned access at TPC[5470a8] kmem_cache_free+0xc8/0x200
[603982.501605] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
[603982.530382] Kernel unaligned access at TPC[5470a8] kmem_cache_free+0xc8/0x200
[603982.544820] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
[603987.567130] log_unaligned: 11 callbacks suppressed
[603987.576582] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603987.591696] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603987.606811] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603987.621904] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
[603987.637017] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0

Re: unaligned accesses in SLAB etc.

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Sat, 11 Oct 2014 22:15:10 -0400 (EDT)

> 
> I'm getting tons of the following on sparc64:
> 
> [603965.383447] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603965.396987] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> [603965.410523] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0

The unaligned accesses are happening in the SLAB_OBJ_PFMEMALLOC code,
which assumes that all object pointers are "unsigned long" aligned:

static inline void set_obj_pfmemalloc(void **objp)
{
        *objp = (void *)((unsigned long)*objp | SLAB_OBJ_PFMEMALLOC);
        return;
}

etc. etc.

But that code has been there working forever.  Something changed
recently such that this assumption no longer holds.

In all of the cases, the address is 4-byte aligned but not 8-byte
aligned.  And they are vmalloc addresses.

Which made me suspect the percpu commit:

====================
commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
Author: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Date:   Thu Oct 9 15:26:27 2014 -0700

    mm/slab: use percpu allocator for cpu cache
====================

And indeed, reverting this commit fixes the problem.

Re: unaligned accesses in SLAB etc.

[mroos]

> From: David Miller <davem@davemloft.net>
> Date: Sat, 11 Oct 2014 22:15:10 -0400 (EDT)
> 
> > 
> > I'm getting tons of the following on sparc64:
> > 
> > [603965.383447] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> > [603965.396987] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> > [603965.410523] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0

> In all of the cases, the address is 4-byte aligned but not 8-byte
> aligned.  And they are vmalloc addresses.
> 
> Which made me suspect the percpu commit:
> 
> ====================
> commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
> Author: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Date:   Thu Oct 9 15:26:27 2014 -0700
> 
>     mm/slab: use percpu allocator for cpu cache
> ====================
> 
> And indeed, reverting this commit fixes the problem.

I tested Joonsoo Kim's fix and it gets rid of the kernel unaligned 
access messages, yes.

But the instability on UltraSparc II era machines still remains - 
occassional Bus Errors during kernel compilation, messages like this:

sh[11771]: segfault at ffd6a4d1 ip 00000000f7cc5714 (rpc 00000000f7cc562c) sp 00000000ffd69d90 error 30002 in libc-2.19.so[f7c44000+16a000]

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[Joonsoo Kim]

On Mon, Oct 13, 2014 at 11:22:37PM +0300, mroos@linux.ee wrote:
> > From: David Miller <davem@davemloft.net>
> > Date: Sat, 11 Oct 2014 22:15:10 -0400 (EDT)
> > 
> > > 
> > > I'm getting tons of the following on sparc64:
> > > 
> > > [603965.383447] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> > > [603965.396987] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> > > [603965.410523] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> 
> > In all of the cases, the address is 4-byte aligned but not 8-byte
> > aligned.  And they are vmalloc addresses.
> > 
> > Which made me suspect the percpu commit:
> > 
> > ====================
> > commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
> > Author: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > Date:   Thu Oct 9 15:26:27 2014 -0700
> > 
> >     mm/slab: use percpu allocator for cpu cache
> > ====================
> > 
> > And indeed, reverting this commit fixes the problem.
> 
> I tested Joonsoo Kim's fix and it gets rid of the kernel unaligned 
> access messages, yes.
> 
> But the instability on UltraSparc II era machines still remains - 
> occassional Bus Errors during kernel compilation, messages like this:
> 
> sh[11771]: segfault at ffd6a4d1 ip 00000000f7cc5714 (rpc 00000000f7cc562c) sp 00000000ffd69d90 error 30002 in libc-2.19.so[f7c44000+16a000]

Hello, Meelis.

Thanks for testing.

I'd like to know that your another problem is related to commit
bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").
So, if the commit is reverted, your another problem is also gone completely?

Thanks.

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Date: Tue, 14 Oct 2014 08:52:19 +0900

> I'd like to know that your another problem is related to commit
> bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").  So,
> if the commit is reverted, your another problem is also gone
> completely?

The other problem has been present forever.

Re: unaligned accesses in SLAB etc.

[Joonsoo Kim]

On Mon, Oct 13, 2014 at 08:04:16PM -0400, David Miller wrote:
> From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> Date: Tue, 14 Oct 2014 08:52:19 +0900
> 
> > I'd like to know that your another problem is related to commit
> > bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").  So,
> > if the commit is reverted, your another problem is also gone
> > completely?
> 
> The other problem has been present forever.

Okay.
Thanks for notifying me.

Thanks.

Re: unaligned accesses in SLAB etc.

[mroos]

> > I'd like to know that your another problem is related to commit
> > bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").  So,
> > if the commit is reverted, your another problem is also gone
> > completely?
> 
> The other problem has been present forever.

Umm? I am afraid I have been describing it badly. This random 
SIGBUS+SIGSEGV problem is new - I have not seen it before.

I have been able to do kernel compiles for years on sparc64 (modulo 
specific bugs in specific configurations) and 3.17 + start/end swap 
patch seems also stable for most machine. With yesterdays git + align 
patch, it dies with SIGBUS multiple times during compilation so it's a 
new regression for me.

Will try reverting that commit tomorrow.

My only other current sparc64 problems that I am seeing - V210/V440 die 
during bootup if compiled with gcc 4.9 and V480 dies with FATAL 
exceptions during bootups since previous kernel release. Maybe also 
exit_mmap warning - I do not know if they have been fixed, I see them 
rarely.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: mroos@linux.ee
Date: Wed, 15 Oct 2014 00:19:36 +0300 (EEST)

>> > I'd like to know that your another problem is related to commit
>> > bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").  So,
>> > if the commit is reverted, your another problem is also gone
>> > completely?
>> 
>> The other problem has been present forever.
> 
> Umm? I am afraid I have been describing it badly. This random 
> SIGBUS+SIGSEGV problem is new - I have not seen it before.

Sorry, I thought it was the same bug that causes git corruptions
for you.  I misunderstood.

> I have been able to do kernel compiles for years on sparc64 (modulo 
> specific bugs in specific configurations) and 3.17 + start/end swap 
> patch seems also stable for most machine. With yesterdays git + align 
> patch, it dies with SIGBUS multiple times during compilation so it's a 
> new regression for me.
> 
> Will try reverting that commit tomorrow.

If that fails, please try to bisect, it will help us a lot.

> My only other current sparc64 problems that I am seeing - V210/V440 die 
> during bootup if compiled with gcc 4.9 and V480 dies with FATAL 
> exceptions during bootups since previous kernel release. Maybe also 
> exit_mmap warning - I do not know if they have been fixed, I see them 
> rarely.

The gcc-4.9 case is interesting, are you saying that a gcc-4.9 compiled
kernel works fine on other systems?

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > My only other current sparc64 problems that I am seeing - V210/V440 die 
> > during bootup if compiled with gcc 4.9 and V480 dies with FATAL 
> > exceptions during bootups since previous kernel release. Maybe also 
> > exit_mmap warning - I do not know if they have been fixed, I see them 
> > rarely.
> 
> The gcc-4.9 case is interesting, are you saying that a gcc-4.9 compiled
> kernel works fine on other systems?

Yes, all USII based systems work fine with Debian gcc-4.9, as does 
T2000. Of USIII* systems, V210 and V440 exhibit the boot hang with 
gcc-4.9 and V480 crashes wit FATAL exception during boot that is 
probably earlier than the gcc boot hang so I do not know about V480 and 
gcc-4.9. V240 not tested because of fan failures, V245 is in the queue 
for setup but not tested so far.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Wed, 15 Oct 2014 11:04:49 +0300 (EEST)

>> > My only other current sparc64 problems that I am seeing - V210/V440 die 
>> > during bootup if compiled with gcc 4.9 and V480 dies with FATAL 
>> > exceptions during bootups since previous kernel release. Maybe also 
>> > exit_mmap warning - I do not know if they have been fixed, I see them 
>> > rarely.
>> 
>> The gcc-4.9 case is interesting, are you saying that a gcc-4.9 compiled
>> kernel works fine on other systems?
> 
> Yes, all USII based systems work fine with Debian gcc-4.9, as does 
> T2000. Of USIII* systems, V210 and V440 exhibit the boot hang with 
> gcc-4.9 and V480 crashes wit FATAL exception during boot that is 
> probably earlier than the gcc boot hang so I do not know about V480 and 
> gcc-4.9. V240 not tested because of fan failures, V245 is in the queue 
> for setup but not tested so far.

Ok, on the V210/V440 can you boot with "-p" on the kernel boot command
line and post the log?  Let's start by seeing how far it gets, maybe
we can figure out roughly where it dies.

A boot hang should be relatively easy to diagnose and pinpoint.

Thanks.

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> >> The gcc-4.9 case is interesting, are you saying that a gcc-4.9 compiled
> >> kernel works fine on other systems?
> > 
> > Yes, all USII based systems work fine with Debian gcc-4.9, as does 
> > T2000. Of USIII* systems, V210 and V440 exhibit the boot hang with 
> > gcc-4.9 and V480 crashes wit FATAL exception during boot that is 
> > probably earlier than the gcc boot hang so I do not know about V480 and 
> > gcc-4.9. V240 not tested because of fan failures, V245 is in the queue 
> > for setup but not tested so far.
> 
> Ok, on the V210/V440 can you boot with "-p" on the kernel boot command
> line and post the log?  Let's start by seeing how far it gets, maybe
> we can figure out roughly where it dies.

http://www.spinics.net/lists/sparclinux/msg12238.html and 
http://www.spinics.net/lists/sparclinux/msg12468.html are my relevant 
posts about it. Should I get something more? It would be easy because of 
ALOM.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Wed, 15 Oct 2014 23:11:34 +0300 (EEST)

>> >> The gcc-4.9 case is interesting, are you saying that a gcc-4.9 compiled
>> >> kernel works fine on other systems?
>> > 
>> > Yes, all USII based systems work fine with Debian gcc-4.9, as does 
>> > T2000. Of USIII* systems, V210 and V440 exhibit the boot hang with 
>> > gcc-4.9 and V480 crashes wit FATAL exception during boot that is 
>> > probably earlier than the gcc boot hang so I do not know about V480 and 
>> > gcc-4.9. V240 not tested because of fan failures, V245 is in the queue 
>> > for setup but not tested so far.
>> 
>> Ok, on the V210/V440 can you boot with "-p" on the kernel boot command
>> line and post the log?  Let's start by seeing how far it gets, maybe
>> we can figure out roughly where it dies.
> 
> http://www.spinics.net/lists/sparclinux/msg12238.html and 
> http://www.spinics.net/lists/sparclinux/msg12468.html are my relevant 
> posts about it. Should I get something more? It would be easy because of 
> ALOM.

Less information than I had hoped :-/

I thought it was hanging "during boot" meaning before we try to
execute userspace.  When in fact it seems to die exactly when we start
running the init process.

Wrt. disassembly of fault_in_user_windows(), that's not likely the
cause because if it were being miscompiled it would equally not work
on the other systems.

Something in the UltraSPARC-III specific code paths is going wrong
(either it is miscompiled, or the code makes an assumption that isn't
valid which has happened in the past).

Do you happen to have both gcc-4.9 and a previously working compiler
on these systems?  If you do, we can build a kernel with gcc-4.9 and
then selectively compile certain failes with the older working
compiler to narrow down what compiles into something non-working with
gcc-4.9

I would start with the following files:

	arch/sparc/mm/init_64.c
	arch/sparc/mm/tlb.c
	arch/sparc/mm/tsb.c
	arch/sparc/mm/fault_64.c

And failing that, go for various files under arch/sparc/kernel/ such as:

	arch/sparc/kernel/process_64.c
	arch/sparc/kernel/smp_64.c
	arch/sparc/kernel/sys_sparc_64.c
	arch/sparc/kernel/sys_sparc32.c
	arch/sparc/kernel/traps_64.c

Hopefully, this should be a simply matter of doing a complete build
with gcc-4.9, then removing the object file we want to selectively
build with the older compiler and then going:

	make CC="gcc-4.6" arch/sparc/mm/init_64.o

then relinking with plain 'make'.

If the build system rebuilds the object file on you when you try
to relink the final kernel image, we'll have to do some of this
by hand to make the test.

Thanks.

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> Do you happen to have both gcc-4.9 and a previously working compiler
> on these systems?  If you do, we can build a kernel with gcc-4.9 and
> then selectively compile certain failes with the older working
> compiler to narrow down what compiles into something non-working with
> gcc-4.9

Yes, I kept gcc-4.6 to help resolving it.

[...]

> Hopefully, this should be a simply matter of doing a complete build
> with gcc-4.9, then removing the object file we want to selectively
> build with the older compiler and then going:
> 
> 	make CC="gcc-4.6" arch/sparc/mm/init_64.o
> 
> then relinking with plain 'make'.
> 
> If the build system rebuilds the object file on you when you try
> to relink the final kernel image, we'll have to do some of this
> by hand to make the test.

Unfortunately it starts a full rebuild with plain make after compiling 
some files with gcc-4.6 - detects CC change?

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > Hopefully, this should be a simply matter of doing a complete build
> > with gcc-4.9, then removing the object file we want to selectively
> > build with the older compiler and then going:
> > 
> > 	make CC="gcc-4.6" arch/sparc/mm/init_64.o
> > 
> > then relinking with plain 'make'.
> > 
> > If the build system rebuilds the object file on you when you try
> > to relink the final kernel image, we'll have to do some of this
> > by hand to make the test.
> 
> Unfortunately it starts a full rebuild with plain make after compiling 
> some files with gcc-4.6 - detects CC change?

Figured out from make V=1 how to call gcc-4.6 directly, so far my 
bisection shows that it one or probably more of arch/sparc/kernel/*.c 
but probably more than 1 - 2 halfs of it both failed. Still bisecting.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Thu, 16 Oct 2014 23:11:49 +0300 (EEST)

>> > Hopefully, this should be a simply matter of doing a complete build
>> > with gcc-4.9, then removing the object file we want to selectively
>> > build with the older compiler and then going:
>> > 
>> > 	make CC="gcc-4.6" arch/sparc/mm/init_64.o
>> > 
>> > then relinking with plain 'make'.
>> > 
>> > If the build system rebuilds the object file on you when you try
>> > to relink the final kernel image, we'll have to do some of this
>> > by hand to make the test.
>> 
>> Unfortunately it starts a full rebuild with plain make after compiling 
>> some files with gcc-4.6 - detects CC change?
> 
> Figured out from make V=1 how to call gcc-4.6 directly, so far my 
> bisection shows that it one or probably more of arch/sparc/kernel/*.c 
> but probably more than 1 - 2 halfs of it both failed. Still bisecting.

Thanks a lot for working this out.

I'm going to also try to setup a test environment so I can try this
gcc-4.9 stuff on my T4-2 as well.

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 936 bytes --]

Removed non-sparc64 people from CC to not spam them. Kept the lists 
though.

> Do you happen to have both gcc-4.9 and a previously working compiler
> on these systems?  If you do, we can build a kernel with gcc-4.9 and
> then selectively compile certain failes with the older working
> compiler to narrow down what compiles into something non-working with
> gcc-4.9
> 
> I would start with the following files:
> 
> 	arch/sparc/mm/init_64.c
> 	arch/sparc/mm/tlb.c
> 	arch/sparc/mm/tsb.c
> 	arch/sparc/mm/fault_64.c
> 
> And failing that, go for various files under arch/sparc/kernel/ such as:
> 
> 	arch/sparc/kernel/process_64.c
> 	arch/sparc/kernel/smp_64.c
> 	arch/sparc/kernel/sys_sparc_64.c
> 	arch/sparc/kernel/sys_sparc32.c
> 	arch/sparc/kernel/traps_64.c

arch/sparc/kernel/setup_64.c is the only culprit.

Attached are 2 versions of the object file as of v3.17-rc1-22-g480cadc 
that I tested.

-- 
Meelis Roos (mroos@linux.ee)

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 15992 bytes --]

[-- Attachment #3: Type: APPLICATION/octet-stream, Size: 15472 bytes --]

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Fri, 17 Oct 2014 00:38:20 +0300 (EEST)

> arch/sparc/kernel/setup_64.c is the only culprit.
> 
> Attached are 2 versions of the object file as of v3.17-rc1-22-g480cadc 
> that I tested.

Just to confirm, a gcc-4.9 compiled kernel works if just setup_64.c
is built with gcc-4.6, is that correct?

I'm looking through these two object files now, thanks!

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> From: Meelis Roos <mroos@linux.ee>
> Date: Fri, 17 Oct 2014 00:38:20 +0300 (EEST)
> 
> > arch/sparc/kernel/setup_64.c is the only culprit.
> > 
> > Attached are 2 versions of the object file as of v3.17-rc1-22-g480cadc 
> > that I tested.
> 
> Just to confirm, a gcc-4.9 compiled kernel works if just setup_64.c
> is built with gcc-4.6, is that correct?

Yes, exactly.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Sun, 19 Oct 2014 10:07:03 +0300 (EEST)

>> From: Meelis Roos <mroos@linux.ee>
>> Date: Fri, 17 Oct 2014 00:38:20 +0300 (EEST)
>> 
>> > arch/sparc/kernel/setup_64.c is the only culprit.
>> > 
>> > Attached are 2 versions of the object file as of v3.17-rc1-22-g480cadc 
>> > that I tested.
>> 
>> Just to confirm, a gcc-4.9 compiled kernel works if just setup_64.c
>> is built with gcc-4.6, is that correct?
> 
> Yes, exactly.

Just an update, I have an environment where I can perfectly reproduce
this.  I have a gcc-4.9 SVN built that compiles kernels which crash
the same way it does for you.

I'll let you know when I make more progress.

Re: unaligned accesses in SLAB etc.

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Mon, 20 Oct 2014 14:57:46 -0400 (EDT)

> Just an update, I have an environment where I can perfectly reproduce
> this.  I have a gcc-4.9 SVN built that compiles kernels which crash
> the same way it does for you.
> 
> I'll let you know when I make more progress.

Whilst I don't have access to my reproducer machine until tomorrow in
order to test this myself, I wanted to toss this patch your way so you
could get a head start on me.

The issue is not that gcc-4.9 miscompiles anything, the issue is that
we had an existing bug that is exposed by gcc-4.9 simply allocating
registers in a different order.

per_cpu_patch() is the function that matters.  I verified this by
pulling that function out of setup_64.c and into it's own separate
foo.c file, and only building that source file with gcc-4.9

I poured over the assembler several times over the course of a day or
so, and I'm pretty sure the generated code is fine.  I even extracted
the assembler into a userland test-case and stepped through it for
the code paths that Ultra-III systems trigger.

What happens is that the inner-most registers are corrupted by the
first one of the TLB misses triggered by this code patching.  These
TLB misses are serviced by the firmware because we are still using the
firmware's trap table this early on, and if the code path in the
firmware to service that TLB miss is deep enough we get a register
spill.

This is the top-most of the initial kernel stack's call chain, the
per_cpu_patch() function is invoked right from head_64.S.

What we've traditionally done is save away the firmware's stack
pointer, and jump onto that stack when we make firmware calls.  But
there is absolutely no reason to do that, and it means that by doing
so we have always risked modifying registers erroneously at that
boundary at the top of the initial kernel stack.

So let's get rid of the CIF stack, and just call into the firwmare
using the normal kernel stack.

diff --git a/arch/sparc/include/asm/oplib_64.h b/arch/sparc/include/asm/oplib_64.h
index f346824..741f24a 100644
--- a/arch/sparc/include/asm/oplib_64.h
+++ b/arch/sparc/include/asm/oplib_64.h
@@ -62,7 +62,7 @@ struct linux_mem_p1275 {
 /* You must call prom_init() before using any of the library services,
  * preferably as early as possible.  Pass it the romvec pointer.
  */
-void prom_init(void *cif_handler, void *cif_stack);
+void prom_init(void *cif_handler);
 
 /* Boot argument acquisition, returns the boot command line string. */
 char *prom_getbootargs(void);
diff --git a/arch/sparc/kernel/head_64.S b/arch/sparc/kernel/head_64.S
index 4fdeb80..b8d67c5 100644
--- a/arch/sparc/kernel/head_64.S
+++ b/arch/sparc/kernel/head_64.S
@@ -672,14 +672,12 @@ tlb_fixup_done:
 	sethi	%hi(init_thread_union), %g6
 	or	%g6, %lo(init_thread_union), %g6
 	ldx	[%g6 + TI_TASK], %g4
-	mov	%sp, %l6
 
 	wr	%g0, ASI_P, %asi
 	mov	1, %g1
 	sllx	%g1, THREAD_SHIFT, %g1
 	sub	%g1, (STACKFRAME_SZ + STACK_BIAS), %g1
 	add	%g6, %g1, %sp
-	mov	0, %fp
 
 	/* Set per-cpu pointer initially to zero, this makes
 	 * the boot-cpu use the in-kernel-image per-cpu areas
@@ -706,7 +704,6 @@ tlb_fixup_done:
 	 nop
 #endif
 
-	mov	%l6, %o1			! OpenPROM stack
 	call	prom_init
 	 mov	%l7, %o0			! OpenPROM cif handler
 
diff --git a/arch/sparc/prom/cif.S b/arch/sparc/prom/cif.S
index 9c86b4b..8050f38 100644
--- a/arch/sparc/prom/cif.S
+++ b/arch/sparc/prom/cif.S
@@ -11,11 +11,10 @@
 	.text
 	.globl	prom_cif_direct
 prom_cif_direct:
+	save	%sp, -192, %sp
 	sethi	%hi(p1275buf), %o1
 	or	%o1, %lo(p1275buf), %o1
-	ldx	[%o1 + 0x0010], %o2	! prom_cif_stack
-	save	%o2, -192, %sp
-	ldx	[%i1 + 0x0008], %l2	! prom_cif_handler
+	ldx	[%o1 + 0x0008], %l2	! prom_cif_handler
 	mov	%g4, %l0
 	mov	%g5, %l1
 	mov	%g6, %l3
diff --git a/arch/sparc/prom/init_64.c b/arch/sparc/prom/init_64.c
index d95db75..110b0d7 100644
--- a/arch/sparc/prom/init_64.c
+++ b/arch/sparc/prom/init_64.c
@@ -26,13 +26,13 @@ phandle prom_chosen_node;
  * It gets passed the pointer to the PROM vector.
  */
 
-extern void prom_cif_init(void *, void *);
+extern void prom_cif_init(void *);
 
-void __init prom_init(void *cif_handler, void *cif_stack)
+void __init prom_init(void *cif_handler)
 {
 	phandle node;
 
-	prom_cif_init(cif_handler, cif_stack);
+	prom_cif_init(cif_handler);
 
 	prom_chosen_node = prom_finddevice(prom_chosen_path);
 	if (!prom_chosen_node || (s32)prom_chosen_node == -1)
diff --git a/arch/sparc/prom/p1275.c b/arch/sparc/prom/p1275.c
index b2340f0..545d8bb 100644
--- a/arch/sparc/prom/p1275.c
+++ b/arch/sparc/prom/p1275.c
@@ -20,7 +20,6 @@
 struct {
 	long prom_callback;			/* 0x00 */
 	void (*prom_cif_handler)(long *);	/* 0x08 */
-	unsigned long prom_cif_stack;		/* 0x10 */
 } p1275buf;
 
 extern void prom_world(int);
@@ -52,5 +51,4 @@ void p1275_cmd_direct(unsigned long *args)
 void prom_cif_init(void *cif_handler, void *cif_stack)
 {
 	p1275buf.prom_cif_handler = (void (*)(long *))cif_handler;
-	p1275buf.prom_cif_stack = (unsigned long)cif_stack;
 }

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> Whilst I don't have access to my reproducer machine until tomorrow in
> order to test this myself, I wanted to toss this patch your way so you
> could get a head start on me.

Unfortunately it fails earlier during the boot:

[   25.232318] clocksource: mult[53555555] shift[24]
[   25.288559] clockevent: mult[3126e98] shift[32]
[   25.342813] Console: colour dummy device 80x25
[   25.395990] console [tty0] enabled
[   25.436726] bootconsole [earlyprom0] disabled
ERROR: Last Trap: Memory Address not Aligned

ok

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Thu, 23 Oct 2014 02:22:36 +0300 (EEST)

>> Whilst I don't have access to my reproducer machine until tomorrow in
>> order to test this myself, I wanted to toss this patch your way so you
>> could get a head start on me.
> 
> Unfortunately it fails earlier during the boot:
> 
> [   25.232318] clocksource: mult[53555555] shift[24]
> [   25.288559] clockevent: mult[3126e98] shift[32]
> [   25.342813] Console: colour dummy device 80x25
> [   25.395990] console [tty0] enabled
> [   25.436726] bootconsole [earlyprom0] disabled
> ERROR: Last Trap: Memory Address not Aligned
> 
> ok

I can reproduce and I know what the problem is, fixed patch coming up
shortly.

Thanks for all of your help so far!

Re: unaligned accesses in SLAB etc.

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 23 Oct 2014 13:02:58 -0400 (EDT)

> From: Meelis Roos <mroos@linux.ee>
> Date: Thu, 23 Oct 2014 02:22:36 +0300 (EEST)
> 
>>> Whilst I don't have access to my reproducer machine until tomorrow in
>>> order to test this myself, I wanted to toss this patch your way so you
>>> could get a head start on me.
>> 
>> Unfortunately it fails earlier during the boot:
>> 
>> [   25.232318] clocksource: mult[53555555] shift[24]
>> [   25.288559] clockevent: mult[3126e98] shift[32]
>> [   25.342813] Console: colour dummy device 80x25
>> [   25.395990] console [tty0] enabled
>> [   25.436726] bootconsole [earlyprom0] disabled
>> ERROR: Last Trap: Memory Address not Aligned
>> 
>> ok
> 
> I can reproduce and I know what the problem is, fixed patch coming up
> shortly.

Ok, please test this patch below.

I tested the full matrix of pre-gcc-4.9, gcc-4.9, hypervisor boot
without HOTPLUG_CPU enabled, hypervisor boot with HOTPLUG_CPU enabled,
pre-hypervisor boot.

====================
[PATCH] sparc64: Fix register corruption in top-most kernel stack frame during boot.

Meelis Roos reported that kernels built with gcc-4.9 do not boot, we
eventually narrowed this down to only impacting machines using
UltraSPARC-III and derivitive cpus.

The crash happens right when the first user process is spawned:

[   54.451346] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[   54.451346]
[   54.571516] CPU: 1 PID: 1 Comm: init Not tainted 3.16.0-rc2-00211-gd7933ab #96
[   54.666431] Call Trace:
[   54.698453]  [0000000000762f8c] panic+0xb0/0x224
[   54.759071]  [000000000045cf68] do_exit+0x948/0x960
[   54.823123]  [000000000042cbc0] fault_in_user_windows+0xe0/0x100
[   54.902036]  [0000000000404ad0] __handle_user_windows+0x0/0x10
[   54.978662] Press Stop-A (L1-A) to return to the boot prom
[   55.050713] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004

Further investigation showed that compiling only per_cpu_patch() with
an older compiler fixes the boot.

Detailed analysis showed that the function is not being miscompiled by
gcc-4.9, but it is using a different register allocation ordering.

With the gcc-4.9 compiled function, something during the code patching
causes some of the %i* input registers to get corrupted.  Perhaps
we have a TLB miss path into the firmware that is deep enough to
cause a register window spill and subsequent restore when we get
back from the TLB miss trap.

Let's plug this up by doing two things:

1) Stop using the firmware stack for client interface calls into
   the firmware.  Just use the kernel's stack.

2) As soon as we can, call into a new function "start_early_boot()"
   to put a one-register-window buffer between the firmware's
   deepest stack frame and the top-most initial kernel one.

Reported-by: Meelis Roos <mroos@linux.ee>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc/include/asm/oplib_64.h |  3 ++-
 arch/sparc/kernel/entry.h         |  3 ---
 arch/sparc/kernel/head_64.S       | 38 +-------------------------------------
 arch/sparc/kernel/hvtramp.S       |  1 -
 arch/sparc/kernel/setup_64.c      | 28 ++++++++++++++++++++--------
 arch/sparc/kernel/trampoline_64.S | 12 +++++++-----
 arch/sparc/prom/cif.S             |  5 ++---
 arch/sparc/prom/init_64.c         |  6 +++---
 arch/sparc/prom/p1275.c           |  2 --
 9 files changed, 35 insertions(+), 63 deletions(-)

diff --git a/arch/sparc/include/asm/oplib_64.h b/arch/sparc/include/asm/oplib_64.h
index f346824..2e3a4ad 100644
--- a/arch/sparc/include/asm/oplib_64.h
+++ b/arch/sparc/include/asm/oplib_64.h
@@ -62,7 +62,8 @@ struct linux_mem_p1275 {
 /* You must call prom_init() before using any of the library services,
  * preferably as early as possible.  Pass it the romvec pointer.
  */
-void prom_init(void *cif_handler, void *cif_stack);
+void prom_init(void *cif_handler);
+void prom_init_report(void);
 
 /* Boot argument acquisition, returns the boot command line string. */
 char *prom_getbootargs(void);
diff --git a/arch/sparc/kernel/entry.h b/arch/sparc/kernel/entry.h
index ebaba61..88d322b 100644
--- a/arch/sparc/kernel/entry.h
+++ b/arch/sparc/kernel/entry.h
@@ -65,13 +65,10 @@ struct pause_patch_entry {
 extern struct pause_patch_entry __pause_3insn_patch,
 	__pause_3insn_patch_end;
 
-void __init per_cpu_patch(void);
 void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *,
 			     struct sun4v_1insn_patch_entry *);
 void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *,
 			     struct sun4v_2insn_patch_entry *);
-void __init sun4v_patch(void);
-void __init boot_cpu_id_too_large(int cpu);
 extern unsigned int dcache_parity_tl1_occurred;
 extern unsigned int icache_parity_tl1_occurred;
 
diff --git a/arch/sparc/kernel/head_64.S b/arch/sparc/kernel/head_64.S
index 4fdeb80..43e27ce 100644
--- a/arch/sparc/kernel/head_64.S
+++ b/arch/sparc/kernel/head_64.S
@@ -672,14 +672,12 @@ tlb_fixup_done:
 	sethi	%hi(init_thread_union), %g6
 	or	%g6, %lo(init_thread_union), %g6
 	ldx	[%g6 + TI_TASK], %g4
-	mov	%sp, %l6
 
 	wr	%g0, ASI_P, %asi
 	mov	1, %g1
 	sllx	%g1, THREAD_SHIFT, %g1
 	sub	%g1, (STACKFRAME_SZ + STACK_BIAS), %g1
 	add	%g6, %g1, %sp
-	mov	0, %fp
 
 	/* Set per-cpu pointer initially to zero, this makes
 	 * the boot-cpu use the in-kernel-image per-cpu areas
@@ -706,44 +704,10 @@ tlb_fixup_done:
 	 nop
 #endif
 
-	mov	%l6, %o1			! OpenPROM stack
 	call	prom_init
 	 mov	%l7, %o0			! OpenPROM cif handler
 
-	/* Initialize current_thread_info()->cpu as early as possible.
-	 * In order to do that accurately we have to patch up the get_cpuid()
-	 * assembler sequences.  And that, in turn, requires that we know
-	 * if we are on a Starfire box or not.  While we're here, patch up
-	 * the sun4v sequences as well.
-	 */
-	call	check_if_starfire
-	 nop
-	call	per_cpu_patch
-	 nop
-	call	sun4v_patch
-	 nop
-
-#ifdef CONFIG_SMP
-	call	hard_smp_processor_id
-	 nop
-	cmp	%o0, NR_CPUS
-	blu,pt	%xcc, 1f
-	 nop
-	call	boot_cpu_id_too_large
-	 nop
-	/* Not reached... */
-
-1:
-#else
-	mov	0, %o0
-#endif
-	sth	%o0, [%g6 + TI_CPU]
-
-	call	prom_init_report
-	 nop
-
-	/* Off we go.... */
-	call	start_kernel
+	call start_early_boot
 	 nop
 	/* Not reached... */
 
diff --git a/arch/sparc/kernel/hvtramp.S b/arch/sparc/kernel/hvtramp.S
index b7ddcdd..cdbfec2 100644
--- a/arch/sparc/kernel/hvtramp.S
+++ b/arch/sparc/kernel/hvtramp.S
@@ -109,7 +109,6 @@ hv_cpu_startup:
 	sllx		%g5, THREAD_SHIFT, %g5
 	sub		%g5, (STACKFRAME_SZ + STACK_BIAS), %g5
 	add		%g6, %g5, %sp
-	mov		0, %fp
 
 	call		init_irqwork_curcpu
 	 nop
diff --git a/arch/sparc/kernel/setup_64.c b/arch/sparc/kernel/setup_64.c
index e629b83..c38d19f 100644
--- a/arch/sparc/kernel/setup_64.c
+++ b/arch/sparc/kernel/setup_64.c
@@ -30,6 +30,7 @@
 #include <linux/cpu.h>
 #include <linux/initrd.h>
 #include <linux/module.h>
+#include <linux/start_kernel.h>
 
 #include <asm/io.h>
 #include <asm/processor.h>
@@ -162,7 +163,7 @@ char reboot_command[COMMAND_LINE_SIZE];
 
 static struct pt_regs fake_swapper_regs = { { 0, }, 0, 0, 0, 0 };
 
-void __init per_cpu_patch(void)
+static void __init per_cpu_patch(void)
 {
 	struct cpuid_patch_entry *p;
 	unsigned long ver;
@@ -254,7 +255,7 @@ void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *start,
 	}
 }
 
-void __init sun4v_patch(void)
+static void __init sun4v_patch(void)
 {
 	extern void sun4v_hvapi_init(void);
 
@@ -323,14 +324,25 @@ static void __init pause_patch(void)
 	}
 }
 
-#ifdef CONFIG_SMP
-void __init boot_cpu_id_too_large(int cpu)
+void __init start_early_boot(void)
 {
-	prom_printf("Serious problem, boot cpu id (%d) >= NR_CPUS (%d)\n",
-		    cpu, NR_CPUS);
-	prom_halt();
+	int cpu;
+
+	check_if_starfire();
+	per_cpu_patch();
+	sun4v_patch();
+
+	cpu = hard_smp_processor_id();
+	if (cpu >= NR_CPUS) {
+		prom_printf("Serious problem, boot cpu id (%d) >= NR_CPUS (%d)\n",
+			    cpu, NR_CPUS);
+		prom_halt();
+	}
+	current_thread_info()->cpu = cpu;
+
+	prom_init_report();
+	start_kernel();
 }
-#endif
 
 /* On Ultra, we support all of the v8 capabilities. */
 unsigned long sparc64_elf_hwcap = (HWCAP_SPARC_FLUSH | HWCAP_SPARC_STBAR |
diff --git a/arch/sparc/kernel/trampoline_64.S b/arch/sparc/kernel/trampoline_64.S
index 737f8cb..88ede1d 100644
--- a/arch/sparc/kernel/trampoline_64.S
+++ b/arch/sparc/kernel/trampoline_64.S
@@ -109,10 +109,13 @@ startup_continue:
 	brnz,pn		%g1, 1b
 	 nop
 
-	sethi		%hi(p1275buf), %g2
-	or		%g2, %lo(p1275buf), %g2
-	ldx		[%g2 + 0x10], %l2
-	add		%l2, -(192 + 128), %sp
+	/* Get onto temporary stack which will be in the locked
+	 * kernel image.
+	 */
+	sethi		%hi(tramp_stack), %g1
+	or		%g1, %lo(tramp_stack), %g1
+	add		%g1, TRAMP_STACK_SIZE, %g1
+	sub		%g1, STACKFRAME_SZ + STACK_BIAS + 256, %sp
 	flushw
 
 	/* Setup the loop variables:
@@ -394,7 +397,6 @@ after_lock_tlb:
 	sllx		%g5, THREAD_SHIFT, %g5
 	sub		%g5, (STACKFRAME_SZ + STACK_BIAS), %g5
 	add		%g6, %g5, %sp
-	mov		0, %fp
 
 	rdpr		%pstate, %o1
 	or		%o1, PSTATE_IE, %o1
diff --git a/arch/sparc/prom/cif.S b/arch/sparc/prom/cif.S
index 9c86b4b..8050f38 100644
--- a/arch/sparc/prom/cif.S
+++ b/arch/sparc/prom/cif.S
@@ -11,11 +11,10 @@
 	.text
 	.globl	prom_cif_direct
 prom_cif_direct:
+	save	%sp, -192, %sp
 	sethi	%hi(p1275buf), %o1
 	or	%o1, %lo(p1275buf), %o1
-	ldx	[%o1 + 0x0010], %o2	! prom_cif_stack
-	save	%o2, -192, %sp
-	ldx	[%i1 + 0x0008], %l2	! prom_cif_handler
+	ldx	[%o1 + 0x0008], %l2	! prom_cif_handler
 	mov	%g4, %l0
 	mov	%g5, %l1
 	mov	%g6, %l3
diff --git a/arch/sparc/prom/init_64.c b/arch/sparc/prom/init_64.c
index d95db75..110b0d7 100644
--- a/arch/sparc/prom/init_64.c
+++ b/arch/sparc/prom/init_64.c
@@ -26,13 +26,13 @@ phandle prom_chosen_node;
  * It gets passed the pointer to the PROM vector.
  */
 
-extern void prom_cif_init(void *, void *);
+extern void prom_cif_init(void *);
 
-void __init prom_init(void *cif_handler, void *cif_stack)
+void __init prom_init(void *cif_handler)
 {
 	phandle node;
 
-	prom_cif_init(cif_handler, cif_stack);
+	prom_cif_init(cif_handler);
 
 	prom_chosen_node = prom_finddevice(prom_chosen_path);
 	if (!prom_chosen_node || (s32)prom_chosen_node == -1)
diff --git a/arch/sparc/prom/p1275.c b/arch/sparc/prom/p1275.c
index b2340f0..545d8bb 100644
--- a/arch/sparc/prom/p1275.c
+++ b/arch/sparc/prom/p1275.c
@@ -20,7 +20,6 @@
 struct {
 	long prom_callback;			/* 0x00 */
 	void (*prom_cif_handler)(long *);	/* 0x08 */
-	unsigned long prom_cif_stack;		/* 0x10 */
 } p1275buf;
 
 extern void prom_world(int);
@@ -52,5 +51,4 @@ void p1275_cmd_direct(unsigned long *args)
 void prom_cif_init(void *cif_handler, void *cif_stack)
 {
 	p1275buf.prom_cif_handler = (void (*)(long *))cif_handler;
-	p1275buf.prom_cif_stack = (unsigned long)cif_stack;
 }
-- 
1.8.1.2

Re: unaligned accesses in SLAB etc.

[Sam Ravnborg]

A minor detail.

> [PATCH] sparc64: Fix register corruption in top-most kernel stack frame during boot.
> 
> -	call	start_kernel
> +	call start_early_boot

Maybe add a comment about stack use - as per your nice patch description.

> +void __init start_early_boot(void)

This will likely result in sparse complaining about:
fuction not declared - should it be static?

A prototype in include/asm/setup.h would be nice.

Other than this - very nice detective work!

	Sam

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Sam Ravnborg <sam@ravnborg.org>
Date: Fri, 24 Oct 2014 06:54:55 +0200

> A minor detail.
> 
>> [PATCH] sparc64: Fix register corruption in top-most kernel stack frame during boot.
>> 
>> -	call	start_kernel
>> +	call start_early_boot
> 
> Maybe add a comment about stack use - as per your nice patch description.

Added.

>> +void __init start_early_boot(void)
> 
> This will likely result in sparse complaining about:
> fuction not declared - should it be static?
> 
> A prototype in include/asm/setup.h would be nice.

Done.

This is the final patch I'll be using and submitting to -stable as well.

====================
[PATCH] sparc64: Fix register corruption in top-most kernel stack frame during boot.

Meelis Roos reported that kernels built with gcc-4.9 do not boot, we
eventually narrowed this down to only impacting machines using
UltraSPARC-III and derivitive cpus.

The crash happens right when the first user process is spawned:

[   54.451346] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004
[   54.451346]
[   54.571516] CPU: 1 PID: 1 Comm: init Not tainted 3.16.0-rc2-00211-gd7933ab #96
[   54.666431] Call Trace:
[   54.698453]  [0000000000762f8c] panic+0xb0/0x224
[   54.759071]  [000000000045cf68] do_exit+0x948/0x960
[   54.823123]  [000000000042cbc0] fault_in_user_windows+0xe0/0x100
[   54.902036]  [0000000000404ad0] __handle_user_windows+0x0/0x10
[   54.978662] Press Stop-A (L1-A) to return to the boot prom
[   55.050713] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004

Further investigation showed that compiling only per_cpu_patch() with
an older compiler fixes the boot.

Detailed analysis showed that the function is not being miscompiled by
gcc-4.9, but it is using a different register allocation ordering.

With the gcc-4.9 compiled function, something during the code patching
causes some of the %i* input registers to get corrupted.  Perhaps
we have a TLB miss path into the firmware that is deep enough to
cause a register window spill and subsequent restore when we get
back from the TLB miss trap.

Let's plug this up by doing two things:

1) Stop using the firmware stack for client interface calls into
   the firmware.  Just use the kernel's stack.

2) As soon as we can, call into a new function "start_early_boot()"
   to put a one-register-window buffer between the firmware's
   deepest stack frame and the top-most initial kernel one.

Reported-by: Meelis Roos <mroos@linux.ee>
Tested-by: Meelis Roos <mroos@linux.ee>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc/include/asm/oplib_64.h |  3 ++-
 arch/sparc/include/asm/setup.h    |  2 ++
 arch/sparc/kernel/entry.h         |  3 ---
 arch/sparc/kernel/head_64.S       | 40 ++++-----------------------------------
 arch/sparc/kernel/hvtramp.S       |  1 -
 arch/sparc/kernel/setup_64.c      | 28 +++++++++++++++++++--------
 arch/sparc/kernel/trampoline_64.S | 12 +++++++-----
 arch/sparc/prom/cif.S             |  5 ++---
 arch/sparc/prom/init_64.c         |  6 +++---
 arch/sparc/prom/p1275.c           |  2 --
 10 files changed, 40 insertions(+), 62 deletions(-)

diff --git a/arch/sparc/include/asm/oplib_64.h b/arch/sparc/include/asm/oplib_64.h
index f346824..2e3a4ad 100644
--- a/arch/sparc/include/asm/oplib_64.h
+++ b/arch/sparc/include/asm/oplib_64.h
@@ -62,7 +62,8 @@ struct linux_mem_p1275 {
 /* You must call prom_init() before using any of the library services,
  * preferably as early as possible.  Pass it the romvec pointer.
  */
-void prom_init(void *cif_handler, void *cif_stack);
+void prom_init(void *cif_handler);
+void prom_init_report(void);
 
 /* Boot argument acquisition, returns the boot command line string. */
 char *prom_getbootargs(void);
diff --git a/arch/sparc/include/asm/setup.h b/arch/sparc/include/asm/setup.h
index f5fffd8..29d64b1 100644
--- a/arch/sparc/include/asm/setup.h
+++ b/arch/sparc/include/asm/setup.h
@@ -48,6 +48,8 @@ unsigned long safe_compute_effective_address(struct pt_regs *, unsigned int);
 #endif
 
 #ifdef CONFIG_SPARC64
+void __init start_early_boot(void);
+
 /* unaligned_64.c */
 int handle_ldf_stq(u32 insn, struct pt_regs *regs);
 void handle_ld_nf(u32 insn, struct pt_regs *regs);
diff --git a/arch/sparc/kernel/entry.h b/arch/sparc/kernel/entry.h
index ebaba61..88d322b 100644
--- a/arch/sparc/kernel/entry.h
+++ b/arch/sparc/kernel/entry.h
@@ -65,13 +65,10 @@ struct pause_patch_entry {
 extern struct pause_patch_entry __pause_3insn_patch,
 	__pause_3insn_patch_end;
 
-void __init per_cpu_patch(void);
 void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *,
 			     struct sun4v_1insn_patch_entry *);
 void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *,
 			     struct sun4v_2insn_patch_entry *);
-void __init sun4v_patch(void);
-void __init boot_cpu_id_too_large(int cpu);
 extern unsigned int dcache_parity_tl1_occurred;
 extern unsigned int icache_parity_tl1_occurred;
 
diff --git a/arch/sparc/kernel/head_64.S b/arch/sparc/kernel/head_64.S
index 4fdeb80..3d61fca 100644
--- a/arch/sparc/kernel/head_64.S
+++ b/arch/sparc/kernel/head_64.S
@@ -672,14 +672,12 @@ tlb_fixup_done:
 	sethi	%hi(init_thread_union), %g6
 	or	%g6, %lo(init_thread_union), %g6
 	ldx	[%g6 + TI_TASK], %g4
-	mov	%sp, %l6
 
 	wr	%g0, ASI_P, %asi
 	mov	1, %g1
 	sllx	%g1, THREAD_SHIFT, %g1
 	sub	%g1, (STACKFRAME_SZ + STACK_BIAS), %g1
 	add	%g6, %g1, %sp
-	mov	0, %fp
 
 	/* Set per-cpu pointer initially to zero, this makes
 	 * the boot-cpu use the in-kernel-image per-cpu areas
@@ -706,44 +704,14 @@ tlb_fixup_done:
 	 nop
 #endif
 
-	mov	%l6, %o1			! OpenPROM stack
 	call	prom_init
 	 mov	%l7, %o0			! OpenPROM cif handler
 
-	/* Initialize current_thread_info()->cpu as early as possible.
-	 * In order to do that accurately we have to patch up the get_cpuid()
-	 * assembler sequences.  And that, in turn, requires that we know
-	 * if we are on a Starfire box or not.  While we're here, patch up
-	 * the sun4v sequences as well.
+	/* To create a one-register-window buffer between the kernel's
+	 * initial stack and the last stack frame we use from the firmware,
+	 * do the rest of the boot from a C helper function.
 	 */
-	call	check_if_starfire
-	 nop
-	call	per_cpu_patch
-	 nop
-	call	sun4v_patch
-	 nop
-
-#ifdef CONFIG_SMP
-	call	hard_smp_processor_id
-	 nop
-	cmp	%o0, NR_CPUS
-	blu,pt	%xcc, 1f
-	 nop
-	call	boot_cpu_id_too_large
-	 nop
-	/* Not reached... */
-
-1:
-#else
-	mov	0, %o0
-#endif
-	sth	%o0, [%g6 + TI_CPU]
-
-	call	prom_init_report
-	 nop
-
-	/* Off we go.... */
-	call	start_kernel
+	call	start_early_boot
 	 nop
 	/* Not reached... */
 
diff --git a/arch/sparc/kernel/hvtramp.S b/arch/sparc/kernel/hvtramp.S
index b7ddcdd..cdbfec2 100644
--- a/arch/sparc/kernel/hvtramp.S
+++ b/arch/sparc/kernel/hvtramp.S
@@ -109,7 +109,6 @@ hv_cpu_startup:
 	sllx		%g5, THREAD_SHIFT, %g5
 	sub		%g5, (STACKFRAME_SZ + STACK_BIAS), %g5
 	add		%g6, %g5, %sp
-	mov		0, %fp
 
 	call		init_irqwork_curcpu
 	 nop
diff --git a/arch/sparc/kernel/setup_64.c b/arch/sparc/kernel/setup_64.c
index e629b83..c38d19f 100644
--- a/arch/sparc/kernel/setup_64.c
+++ b/arch/sparc/kernel/setup_64.c
@@ -30,6 +30,7 @@
 #include <linux/cpu.h>
 #include <linux/initrd.h>
 #include <linux/module.h>
+#include <linux/start_kernel.h>
 
 #include <asm/io.h>
 #include <asm/processor.h>
@@ -162,7 +163,7 @@ char reboot_command[COMMAND_LINE_SIZE];
 
 static struct pt_regs fake_swapper_regs = { { 0, }, 0, 0, 0, 0 };
 
-void __init per_cpu_patch(void)
+static void __init per_cpu_patch(void)
 {
 	struct cpuid_patch_entry *p;
 	unsigned long ver;
@@ -254,7 +255,7 @@ void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *start,
 	}
 }
 
-void __init sun4v_patch(void)
+static void __init sun4v_patch(void)
 {
 	extern void sun4v_hvapi_init(void);
 
@@ -323,14 +324,25 @@ static void __init pause_patch(void)
 	}
 }
 
-#ifdef CONFIG_SMP
-void __init boot_cpu_id_too_large(int cpu)
+void __init start_early_boot(void)
 {
-	prom_printf("Serious problem, boot cpu id (%d) >= NR_CPUS (%d)\n",
-		    cpu, NR_CPUS);
-	prom_halt();
+	int cpu;
+
+	check_if_starfire();
+	per_cpu_patch();
+	sun4v_patch();
+
+	cpu = hard_smp_processor_id();
+	if (cpu >= NR_CPUS) {
+		prom_printf("Serious problem, boot cpu id (%d) >= NR_CPUS (%d)\n",
+			    cpu, NR_CPUS);
+		prom_halt();
+	}
+	current_thread_info()->cpu = cpu;
+
+	prom_init_report();
+	start_kernel();
 }
-#endif
 
 /* On Ultra, we support all of the v8 capabilities. */
 unsigned long sparc64_elf_hwcap = (HWCAP_SPARC_FLUSH | HWCAP_SPARC_STBAR |
diff --git a/arch/sparc/kernel/trampoline_64.S b/arch/sparc/kernel/trampoline_64.S
index 737f8cb..88ede1d 100644
--- a/arch/sparc/kernel/trampoline_64.S
+++ b/arch/sparc/kernel/trampoline_64.S
@@ -109,10 +109,13 @@ startup_continue:
 	brnz,pn		%g1, 1b
 	 nop
 
-	sethi		%hi(p1275buf), %g2
-	or		%g2, %lo(p1275buf), %g2
-	ldx		[%g2 + 0x10], %l2
-	add		%l2, -(192 + 128), %sp
+	/* Get onto temporary stack which will be in the locked
+	 * kernel image.
+	 */
+	sethi		%hi(tramp_stack), %g1
+	or		%g1, %lo(tramp_stack), %g1
+	add		%g1, TRAMP_STACK_SIZE, %g1
+	sub		%g1, STACKFRAME_SZ + STACK_BIAS + 256, %sp
 	flushw
 
 	/* Setup the loop variables:
@@ -394,7 +397,6 @@ after_lock_tlb:
 	sllx		%g5, THREAD_SHIFT, %g5
 	sub		%g5, (STACKFRAME_SZ + STACK_BIAS), %g5
 	add		%g6, %g5, %sp
-	mov		0, %fp
 
 	rdpr		%pstate, %o1
 	or		%o1, PSTATE_IE, %o1
diff --git a/arch/sparc/prom/cif.S b/arch/sparc/prom/cif.S
index 9c86b4b..8050f38 100644
--- a/arch/sparc/prom/cif.S
+++ b/arch/sparc/prom/cif.S
@@ -11,11 +11,10 @@
 	.text
 	.globl	prom_cif_direct
 prom_cif_direct:
+	save	%sp, -192, %sp
 	sethi	%hi(p1275buf), %o1
 	or	%o1, %lo(p1275buf), %o1
-	ldx	[%o1 + 0x0010], %o2	! prom_cif_stack
-	save	%o2, -192, %sp
-	ldx	[%i1 + 0x0008], %l2	! prom_cif_handler
+	ldx	[%o1 + 0x0008], %l2	! prom_cif_handler
 	mov	%g4, %l0
 	mov	%g5, %l1
 	mov	%g6, %l3
diff --git a/arch/sparc/prom/init_64.c b/arch/sparc/prom/init_64.c
index d95db75..110b0d7 100644
--- a/arch/sparc/prom/init_64.c
+++ b/arch/sparc/prom/init_64.c
@@ -26,13 +26,13 @@ phandle prom_chosen_node;
  * It gets passed the pointer to the PROM vector.
  */
 
-extern void prom_cif_init(void *, void *);
+extern void prom_cif_init(void *);
 
-void __init prom_init(void *cif_handler, void *cif_stack)
+void __init prom_init(void *cif_handler)
 {
 	phandle node;
 
-	prom_cif_init(cif_handler, cif_stack);
+	prom_cif_init(cif_handler);
 
 	prom_chosen_node = prom_finddevice(prom_chosen_path);
 	if (!prom_chosen_node || (s32)prom_chosen_node == -1)
diff --git a/arch/sparc/prom/p1275.c b/arch/sparc/prom/p1275.c
index b2340f0..545d8bb 100644
--- a/arch/sparc/prom/p1275.c
+++ b/arch/sparc/prom/p1275.c
@@ -20,7 +20,6 @@
 struct {
 	long prom_callback;			/* 0x00 */
 	void (*prom_cif_handler)(long *);	/* 0x08 */
-	unsigned long prom_cif_stack;		/* 0x10 */
 } p1275buf;
 
 extern void prom_world(int);
@@ -52,5 +51,4 @@ void p1275_cmd_direct(unsigned long *args)
 void prom_cif_init(void *cif_handler, void *cif_stack)
 {
 	p1275buf.prom_cif_handler = (void (*)(long *))cif_handler;
-	p1275buf.prom_cif_stack = (unsigned long)cif_stack;
 }
-- 
1.8.1.2

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > I can reproduce and I know what the problem is, fixed patch coming up
> > shortly.
> 
> Ok, please test this patch below.

Thank you, it works fine on my V210, V440 and newly added V245.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Fri, 24 Oct 2014 10:49:12 +0300 (EEST)

>> > I can reproduce and I know what the problem is, fixed patch coming up
>> > shortly.
>> 
>> Ok, please test this patch below.
> 
> Thank you, it works fine on my V210, V440 and newly added V245.

Thanks so much for testing.

I'll clean this up based upon Sam's feedback, add your Tested-by:,
and push it around.

Thanks again!

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> >> > I'd like to know that your another problem is related to commit
> >> > bf0dea23a9c0 ("mm/slab: use percpu allocator for cpu cache").  So,
> >> > if the commit is reverted, your another problem is also gone
> >> > completely?
> >> 
> >> The other problem has been present forever.
> > 
> > Umm? I am afraid I have been describing it badly. This random 
> > SIGBUS+SIGSEGV problem is new - I have not seen it before.
> 
> Sorry, I thought it was the same bug that causes git corruptions
> for you.  I misunderstood.
> 
> > I have been able to do kernel compiles for years on sparc64 (modulo 
> > specific bugs in specific configurations) and 3.17 + start/end swap 
> > patch seems also stable for most machine. With yesterdays git + align 
> > patch, it dies with SIGBUS multiple times during compilation so it's a 
> > new regression for me.
> > 
> > Will try reverting that commit tomorrow.
> 
> If that fails, please try to bisect, it will help us a lot.

Commit bf0dea23a9c0 is working OK with no revert needed (checked out 
this revision and it tested OK).

So far I know that the breakage seems to have happened between
cadbb58039f7cab1def9c931012ab04c953a6997 (first sparc commit of 
the batch, working OK on V100) and 
bdcf81b658ebc4c2640c3c2c55c8b31c601b6996 (last sparc commit before the 
merge, breaks on E3500). Will continue bisecting the sparc64 commits.

Also, I noticed that when the problem happens, it's deterministic - with 
some kernels, sshd dies reproducibly on login. With most kernels, 
building kernel breaks in one specific location, not randomly.

scripts/Makefile.build:352: recipe for target 'sound/modules.order' failed
make[1]: *** [sound/modules.order] Bus error
make[1]: *** Deleting file 'sound/modules.order'
Makefile:929: recipe for target 'sound' failed

Will tell when I get more details.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Thu, 16 Oct 2014 10:02:57 +0300 (EEST)

> scripts/Makefile.build:352: recipe for target 'sound/modules.order' failed
> make[1]: *** [sound/modules.order] Bus error
> make[1]: *** Deleting file 'sound/modules.order'
> Makefile:929: recipe for target 'sound' failed

I just reproduced this on my Sun Blade 2500, so it can trigger on UltraSPARC-IIIi
systems too.

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > scripts/Makefile.build:352: recipe for target 'sound/modules.order' failed
> > make[1]: *** [sound/modules.order] Bus error
> > make[1]: *** Deleting file 'sound/modules.order'
> > Makefile:929: recipe for target 'sound' failed
> 
> I just reproduced this on my Sun Blade 2500, so it can trigger on UltraSPARC-IIIi
> systems too.

My bisection led to the folloowing commit but it seems irrelevant (I 
have no sun4v on these machines):

4ccb9272892c33ef1c19a783cfa87103b30c2784 is the first bad commit
commit 4ccb9272892c33ef1c19a783cfa87103b30c2784
Author: bob picco <bpicco@meloft.net>
Date:   Tue Sep 16 09:26:47 2014 -0400

    sparc64: sun4v TLB error power off events


However, the following chunk sound slightly suspicious:

+       if (fault_code & FAULT_CODE_BAD_RA)
+               goto do_sigbus;
+

because SIGNUS is what I got. For some machines, it killed chekroot 
during startup, for some shells under some circumstances, for some sshd.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Thu, 16 Oct 2014 23:16:44 +0300 (EEST)

>> > scripts/Makefile.build:352: recipe for target 'sound/modules.order' failed
>> > make[1]: *** [sound/modules.order] Bus error
>> > make[1]: *** Deleting file 'sound/modules.order'
>> > Makefile:929: recipe for target 'sound' failed
>> 
>> I just reproduced this on my Sun Blade 2500, so it can trigger on UltraSPARC-IIIi
>> systems too.
> 
> My bisection led to the folloowing commit but it seems irrelevant (I 
> have no sun4v on these machines):
> 
> 4ccb9272892c33ef1c19a783cfa87103b30c2784 is the first bad commit
> commit 4ccb9272892c33ef1c19a783cfa87103b30c2784
> Author: bob picco <bpicco@meloft.net>
> Date:   Tue Sep 16 09:26:47 2014 -0400
> 
>     sparc64: sun4v TLB error power off events
> 
> 
> However, the following chunk sound slightly suspicious:
> 
> +       if (fault_code & FAULT_CODE_BAD_RA)
> +               goto do_sigbus;
> +
> 
> because SIGNUS is what I got. For some machines, it killed chekroot 
> during startup, for some shells under some circumstances, for some sshd.

Good catch!

So I'm going to audit all the code paths to make sure we don't put garbage
into the fault_code value.

Re: unaligned accesses in SLAB etc.

[David Miller]

From: David Miller <davem@redhat.com>
Date: Thu, 16 Oct 2014 16:20:01 -0400 (EDT)

> So I'm going to audit all the code paths to make sure we don't put garbage
> into the fault_code value.

There are two code paths where we can put garbage into the fault_code
value.  And for the dtlb_prot.S case, the value we put in there is
TLB_TAG_ACCESS which is 0x30, which include bit 0x20 which is that
FAULT_CODE_BAD_RA indication which is erroneously triggering.

The other path is via hugepage TLB misses, for the situation where
we haven't allocated the huge TSB for the thread yet.  That might
explain some other longer-term problems we've had.

I'm about to test the following fix:

diff --git a/arch/sparc/kernel/dtlb_prot.S b/arch/sparc/kernel/dtlb_prot.S
index b2c2c5b..d668ca14 100644
--- a/arch/sparc/kernel/dtlb_prot.S
+++ b/arch/sparc/kernel/dtlb_prot.S
@@ -24,11 +24,11 @@
 	mov		TLB_TAG_ACCESS, %g4		! For reload of vaddr
 
 /* PROT ** ICACHE line 2: More real fault processing */
+	ldxa		[%g4] ASI_DMMU, %g5		! Put tagaccess in %g5
 	bgu,pn		%xcc, winfix_trampoline		! Yes, perform winfixup
-	 ldxa		[%g4] ASI_DMMU, %g5		! Put tagaccess in %g5
-	ba,pt		%xcc, sparc64_realfault_common	! Nope, normal fault
 	 mov		FAULT_CODE_DTLB | FAULT_CODE_WRITE, %g4
-	nop
+	ba,pt		%xcc, sparc64_realfault_common	! Nope, normal fault
+	 nop
 	nop
 	nop
 	nop
diff --git a/arch/sparc/kernel/tsb.S b/arch/sparc/kernel/tsb.S
index 14158d4..be98685 100644
--- a/arch/sparc/kernel/tsb.S
+++ b/arch/sparc/kernel/tsb.S
@@ -162,10 +162,10 @@ tsb_miss_page_table_walk_sun4v_fastpath:
 	nop
 	.previous
 
-	rdpr	%tl, %g3
-	cmp	%g3, 1
+	rdpr	%tl, %g7
+	cmp	%g7, 1
 	bne,pn	%xcc, winfix_trampoline
-	 nop
+	 mov	%g3, %g4
 	ba,pt	%xcc, etrap
 	 rd	%pc, %g7
 	call	hugetlb_setup

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> From: David Miller <davem@redhat.com>
> Date: Thu, 16 Oct 2014 16:20:01 -0400 (EDT)
> 
> > So I'm going to audit all the code paths to make sure we don't put garbage
> > into the fault_code value.
> 
> There are two code paths where we can put garbage into the fault_code
> value.  And for the dtlb_prot.S case, the value we put in there is
> TLB_TAG_ACCESS which is 0x30, which include bit 0x20 which is that
> FAULT_CODE_BAD_RA indication which is erroneously triggering.
> 
> The other path is via hugepage TLB misses, for the situation where
> we haven't allocated the huge TSB for the thread yet.  That might
> explain some other longer-term problems we've had.
> 
> I'm about to test the following fix:

Thank you - it seems to work fine for me on E3500 on top of 
3.17.0-07551-g052db7e + slab alignment fix.

However, on top of mainline HEAD 3.17.0-09670-g0429fbc it explodes with 
scheduler BUG - just reported to LKML + sched maintainers.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Fri, 17 Oct 2014 14:12:09 +0300 (EEST)

> However, on top of mainline HEAD 3.17.0-09670-g0429fbc it explodes with 
> scheduler BUG - just reported to LKML + sched maintainers.

task_stack_end_corrupted() cannot work properly on sparc64.

It stores the magic value at "task_thread_info(p) + 1", but on
sparc64 that's where we store the nested array of FPU register
saves.

In fact this facility could be corrupting FPU register state in
certain circumstances.

The current sparc64 design is intentional, the CPU stack grows down
toward the thread_info, and the FPU stack saving area grows up from
the end of thread_info.

I don't want to define the array size of the fpregs save area
explicitly and thereby placing an artificial limit there.

Re: unaligned accesses in SLAB etc.

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Sat, 18 Oct 2014 13:59:07 -0400 (EDT)

> I don't want to define the array size of the fpregs save area
> explicitly and thereby placing an artificial limit there.

Nevermind, it seems we have a hard limit of 7 FPU save areas anyways.

Meelis, please try this patch:

diff --git a/arch/sparc/include/asm/thread_info_64.h b/arch/sparc/include/asm/thread_info_64.h
index f85dc85..cc6275c 100644
--- a/arch/sparc/include/asm/thread_info_64.h
+++ b/arch/sparc/include/asm/thread_info_64.h
@@ -63,7 +63,8 @@ struct thread_info {
 	struct pt_regs		*kern_una_regs;
 	unsigned int		kern_una_insn;
 
-	unsigned long		fpregs[0] __attribute__ ((aligned(64)));
+	unsigned long		fpregs[(7 * 256) / sizeof(unsigned long)]
+		__attribute__ ((aligned(64)));
 };
 
 #endif /* !(__ASSEMBLY__) */

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > I don't want to define the array size of the fpregs save area
> > explicitly and thereby placing an artificial limit there.
> 
> Nevermind, it seems we have a hard limit of 7 FPU save areas anyways.
> 
> Meelis, please try this patch:

Works fine with 3.17.0-09670-g0429fbc + fault patch.

Will try current git next to find any new problems :)

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > > I don't want to define the array size of the fpregs save area
> > > explicitly and thereby placing an artificial limit there.
> > 
> > Nevermind, it seems we have a hard limit of 7 FPU save areas anyways.
> > 
> > Meelis, please try this patch:
> 
> Works fine with 3.17.0-09670-g0429fbc + fault patch.
> 
> Will try current git next to find any new problems :)

Works on all 3 machines, with latest git (only had to apply the no-ipv6 
patch on one of them). Thank you for the good work!

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Sun, 19 Oct 2014 20:12:43 +0300 (EEST)

>> > > I don't want to define the array size of the fpregs save area
>> > > explicitly and thereby placing an artificial limit there.
>> > 
>> > Nevermind, it seems we have a hard limit of 7 FPU save areas anyways.
>> > 
>> > Meelis, please try this patch:
>> 
>> Works fine with 3.17.0-09670-g0429fbc + fault patch.
>> 
>> Will try current git next to find any new problems :)
> 
> Works on all 3 machines, with latest git (only had to apply the no-ipv6 
> patch on one of them). Thank you for the good work!

Thanks for testing.

Hopefully we can kill the gcc-4.9 bug next, and then see if that
exit_mmap() crash is still happening.

Re: unaligned accesses in SLAB etc.

[Sam Ravnborg]

On Sat, Oct 18, 2014 at 02:23:35PM -0400, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Sat, 18 Oct 2014 13:59:07 -0400 (EDT)
> 
> > I don't want to define the array size of the fpregs save area
> > explicitly and thereby placing an artificial limit there.
> 
> Nevermind, it seems we have a hard limit of 7 FPU save areas anyways.
> 
> Meelis, please try this patch:
> 
> diff --git a/arch/sparc/include/asm/thread_info_64.h b/arch/sparc/include/asm/thread_info_64.h
> index f85dc85..cc6275c 100644
> --- a/arch/sparc/include/asm/thread_info_64.h
> +++ b/arch/sparc/include/asm/thread_info_64.h
> @@ -63,7 +63,8 @@ struct thread_info {
>  	struct pt_regs		*kern_una_regs;
>  	unsigned int		kern_una_insn;
>  
> -	unsigned long		fpregs[0] __attribute__ ((aligned(64)));
> +	unsigned long		fpregs[(7 * 256) / sizeof(unsigned long)]

This part:

> +		__attribute__ ((aligned(64)));

Could be written as __aligned(64)

	Sam

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Sam Ravnborg <sam@ravnborg.org>
Date: Sun, 19 Oct 2014 17:32:20 +0200

> This part:
> 
>> +		__attribute__ ((aligned(64)));
> 
> Could be written as __aligned(64)

I'll try to remember to sweep this up in sparc-next, thanks Sam.

We probably use this long-hand form in a lot of other places in
the sparc code too, so I'll try to do a full sweep.

Thanks again.

Re: unaligned accesses in SLAB etc.

[Sam Ravnborg]

On Sun, Oct 19, 2014 at 01:27:37PM -0400, David Miller wrote:
> From: Sam Ravnborg <sam@ravnborg.org>
> Date: Sun, 19 Oct 2014 17:32:20 +0200
> 
> > This part:
> > 
> >> +		__attribute__ ((aligned(64)));
> > 
> > Could be written as __aligned(64)
> 
> I'll try to remember to sweep this up in sparc-next, thanks Sam.
> 
> We probably use this long-hand form in a lot of other places in
> the sparc code too, so I'll try to do a full sweep.

Another related one would be a full sweep of "__asm__ __volatile__"
to the shorter version "asm volatile".

The latter is used in a few places in sparc already - so toolchain supports it.

I got hits in:
include/asm/irqflags_32.h:      asm volatile("rd        %%psr, %0" : "=r" (flags));
include/asm/processor_64.h:#define cpu_relax()  asm volatile("\n99:\n\t"                        \
kernel/kprobes.c:       asm volatile(".global kretprobe_trampoline\n"

But this would touch 93 files. Thats too much crunch :-(

	Sam

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> I just reproduced this on my Sun Blade 2500, so it can trigger on UltraSPARC-IIIi
> systems too.

I looked it up - V210 and V440 are also IIIi, not plain III. So I do not 
have information about real USIII, sorry for confusion.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[Meelis Roos]

> > I just reproduced this on my Sun Blade 2500, so it can trigger on UltraSPARC-IIIi
> > systems too.
> 
> I looked it up - V210 and V440 are also IIIi, not plain III. So I do not 
> have information about real USIII, sorry for confusion.

Brr, I just understood I confused 2 problems with the same subject. You 
are talking about SIGBUS problem that is also happening on IIIi, my last 
comment is about gcc-4.9 problem so please just ignore it.

-- 
Meelis Roos (mroos@linux.ee)

Re: unaligned accesses in SLAB etc.

[Joonsoo Kim]

2014-10-12 11:15 GMT+09:00 David Miller <davem@davemloft.net>:
>
> I'm getting tons of the following on sparc64:
>
> [603965.383447] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603965.396987] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> [603965.410523] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603965.424061] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> [603965.437617] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603970.554394] log_unaligned: 333 callbacks suppressed
> [603970.564041] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603970.577576] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> [603970.591122] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603970.604669] Kernel unaligned access at TPC[546b60] free_block+0xa0/0x1a0
> [603970.618216] Kernel unaligned access at TPC[546b58] free_block+0x98/0x1a0
> [603976.515633] log_unaligned: 31 callbacks suppressed
> [603976.525092] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603976.540196] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603976.555308] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603976.570411] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603976.585526] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603982.476424] log_unaligned: 43 callbacks suppressed
> [603982.485881] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
> [603982.501590] Kernel unaligned access at TPC[5470a8] kmem_cache_free+0xc8/0x200
> [603982.501605] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
> [603982.530382] Kernel unaligned access at TPC[5470a8] kmem_cache_free+0xc8/0x200
> [603982.544820] Kernel unaligned access at TPC[549378] kmem_cache_alloc+0xd8/0x1e0
> [603987.567130] log_unaligned: 11 callbacks suppressed
> [603987.576582] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603987.591696] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603987.606811] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603987.621904] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0
> [603987.637017] Kernel unaligned access at TPC[548080] cache_alloc_refill+0x180/0x3a0

Hello,

Could you test below patch?
If it fixes your problem, I will send it with proper description.

Thanks.

---------->8----------------
diff --git a/mm/slab.c b/mm/slab.c
index 154aac8..eb2b2ea 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -1992,7 +1992,7 @@ static struct array_cache __percpu *alloc_kmem_cache_cpus(
        struct array_cache __percpu *cpu_cache;

        size = sizeof(void *) * entries + sizeof(struct array_cache);
-       cpu_cache = __alloc_percpu(size, 0);
+       cpu_cache = __alloc_percpu(size, sizeof(void *));

        if (!cpu_cache)
                return NULL;

Re: unaligned accesses in SLAB etc.

[David Miller]

From: Joonsoo Kim <js1304@gmail.com>
Date: Mon, 13 Oct 2014 02:22:15 +0900

> Could you test below patch?
> If it fixes your problem, I will send it with proper description.

It works, I just tested using ARCH_KMALLOC_MINALIGN which would be
better.

Re: unaligned accesses in SLAB etc.

[Joonsoo Kim]

2014-10-13 2:30 GMT+09:00 David Miller <davem@davemloft.net>:
> From: Joonsoo Kim <js1304@gmail.com>
> Date: Mon, 13 Oct 2014 02:22:15 +0900
>
>> Could you test below patch?
>> If it fixes your problem, I will send it with proper description.
>
> It works, I just tested using ARCH_KMALLOC_MINALIGN which would be
> better.

Oops. resend with whole Cc list.

Thanks for testing.
ARCH_KMALLOC_MINALIGN is for object alignment,
but, current problem is caused by alignment of cpu cache array.
I think that my fix is more proper in this situation.
I will send fix tomorrow,
because I'd like to test more and it's 2:42 am. :)

Thanks.