Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[Andi Kleen]

"Nakajima, Jun" <jun.nakajima@intel.com> writes:

>> I would hate to break this again just to save a few hundred bytes in
>> this function. Also the overhead is very low so it is also not
>> interesting to make it conditional for speed reasons.
>
> For maintenance and testing purposes, I think it's still better to make
> it conditional. If the errata are fixed, you might want to kill the
> condition depending on the stepping, for example. During the transition
> time, you need to support both the steppings until old ones go away
> (then remove the workaround).

My understanding is that it will never be fixed on the K7 (current Athlons),
And these will be with us for a long time; more or less forever, just like
the f00f bug :-)

I would agree with you if the patch had some bad impact on common
paths, but I don't think that's the case here. It merely adds a cheap check
to an already slow path.

On the other hand the errata is so unlikely that I doubt it will be frequently
hit anyways. 2.6 kernel hitting it was just very very unlucky, and it
only did so very infrequently.

Just to fix the kernel we could have chosen a different workaround, like
adding an exception table entry to each prefetch and jumping back
(I did that on the x86-64 port originally). But this way is also not that bad
and it fixes hypothetical user space programs that could maybe hit it too.

Of course they may want to also fix it in a different way to run on older
kernels (e.g. handling the signal in user space or avoiding the conditions).
But doing it centrally in the kernel is a bit cleaner and at some point
people have to update their kernels anyways.

-Andi

Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[dada1]

From: "Andi Kleen" <ak@muc.de>
>
> Of course they may want to also fix it in a different way to run on older
> kernels (e.g. handling the signal in user space or avoiding the
conditions).
> But doing it centrally in the kernel is a bit cleaner and at some point
> people have to update their kernels anyways.

Could you be kind enough to post here the example code for a SIGSEGV handler
that would be necessary for old kernels ?

I do think it woul help some people like me, for the future googling on the
prefetch errata.

I do use preftechnta instructions on my programs, and this errata could
explain some strange crashes.

As the program crashing is a huge multi-threaded network application, with
up to 300000 opened TCP sockets, the SIGSEGV fault is usually followed by a
system crash (networks buffers using all of lowmem)

Thanks
Eric Dumazet

Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[Andi Kleen]

On Thu, Sep 11, 2003 at 06:58:23AM +0200, dada1 wrote:
> 
> From: "Andi Kleen" <ak@muc.de>
> >
> > Of course they may want to also fix it in a different way to run on older
> > kernels (e.g. handling the signal in user space or avoiding the
> conditions).
> > But doing it centrally in the kernel is a bit cleaner and at some point
> > people have to update their kernels anyways.
> 
> Could you be kind enough to post here the example code for a SIGSEGV handler
> that would be necessary for old kernels ?

I don't have any. But it would be very similar to the in kernel checking
code (see the is_prefetch function in my patches). Just you feed it
the fields from sigcontext in the signal handler and replace __get_user 
with a normal memory access.

> 
> I do think it woul help some people like me, for the future googling on the
> prefetch errata.
> 
> I do use preftechnta instructions on my programs, and this errata could
> explain some strange crashes.

The bogus faults are very easy to diagnose. When you have a core dump
and disassemble the faulting instruction (in gdb x/i $eip) and it is
a prefetch (prefetch/prefetchw/prefetchnt*) then it could be that.

If it is a different instruction it is unrelated.

It would also only happen when you prefetch ever on unmapped addresses.

> 
> As the program crashing is a huge multi-threaded network application, with
> up to 300000 opened TCP sockets, the SIGSEGV fault is usually followed by a
> system crash (networks buffers using all of lowmem)

That sounds like an unrelated issue.

When user space crashes on this the kernel is unaffected.

In case the 2.6 kernel crashes on this (2.4 does not trigger it)
then you can also run the oops through ksymoops and check if the 
faulting instruction is prefetch. If it isn't  then it is something else.

Network buffers using up all of low mem and then crash 
is likely some OOM handling problem. If you're on 2.4 try an -aa kernel, 
they handle this much better than the marcelo tree. If it's 2.6 then 
I would recommend posting oopses on this list, maybe someone can fix 
it. I suspect 2.6's OOM handling could be still improved.

-Andi

Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[dada1]

>
> I don't have any. But it would be very similar to the in kernel checking
> code (see the is_prefetch function in my patches). Just you feed it
> the fields from sigcontext in the signal handler and replace __get_user
> with a normal memory access.

OK will try... but how test it... sounds not easy.

> > I do use preftechnta instructions on my programs, and this errata could
> > explain some strange crashes.
>
> The bogus faults are very easy to diagnose. When you have a core dump
> and disassemble the faulting instruction (in gdb x/i $eip) and it is
> a prefetch (prefetch/prefetchw/prefetchnt*) then it could be that.

Well, the program is using more than 2Go ram... the core is not written to
disk as the machine hangs just *after*

And this is a remote machine in a Datacenter.

>
> If it is a different instruction it is unrelated.
>
> It would also only happen when you prefetch ever on unmapped addresses.

NULL for example ?

Typical example of code ;

T_cell *ptr, *next ;
for (ptr = list.head ; ptr != NULL ; ptr = next) {
   next = ptr->next ;
   prefetch(next) ;
   some_work(ptr) ;
   }

I may replace NULL by &FakeMappedData   (allways present in memory)

>
> That sounds like an unrelated issue.
>
> When user space crashes on this the kernel is unaffected.

This is not a kernel crash. But total freeze as all memory is used by
network buffers, in no more than 10 seconds.
This application receive smalls TCP messages (about 30 bytes), but the
network stacks allocates 4KB buffers to store this little messages.

No oops, but what can we do, if the freeze lasts eternity ?

I posted a test application some days ago about this problem and got no
answers/feedback.

>
> In case the 2.6 kernel crashes on this (2.4 does not trigger it)
> then you can also run the oops through ksymoops and check if the
> faulting instruction is prefetch. If it isn't  then it is something else.
>
> Network buffers using up all of low mem and then crash
> is likely some OOM handling problem. If you're on 2.4 try an -aa kernel,
> they handle this much better than the marcelo tree. If it's 2.6 then
> I would recommend posting oopses on this list, maybe someone can fix
> it. I suspect 2.6's OOM handling could be still improved.
>
> -Andi
> -

Eric

Network buffer hang was Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[Andi Kleen]

On Thu, Sep 11, 2003 at 07:58:46AM +0200, dada1 wrote:
> 
> >
> > I don't have any. But it would be very similar to the in kernel checking
> > code (see the is_prefetch function in my patches). Just you feed it
> > the fields from sigcontext in the signal handler and replace __get_user
> > with a normal memory access.
> 
> OK will try... but how test it... sounds not easy.

>From your description of the symptoms it sounds like a waste of time.

> 
> Well, the program is using more than 2Go ram... the core is not written to
> disk as the machine hangs just *after*

These bug doesn't cause kernel hangs, just "ordinary" segfaults

(unless the kernel triggers it, but that happens only very rarely)

> >
> > If it is a different instruction it is unrelated.
> >
> > It would also only happen when you prefetch ever on unmapped addresses.
> 
> NULL for example ?

Yes, NULL would work. But only when there are byte sized reads of parts
of the prefetched memory that are not aligned to the cache line size
and hit the second part of a cache line and happens exactly the in
the right timing window after the prefetch

(see Richard's description - it really is quite hard to trigger) 


> 
> Typical example of code ;
> 
> T_cell *ptr, *next ;
> for (ptr = list.head ; ptr != NULL ; ptr = next) {
>    next = ptr->next ;
>    prefetch(next) ;
>    some_work(ptr) ;
>    }
> 
> I may replace NULL by &FakeMappedData   (allways present in memory)

That's certainly safer, but slightly slower (need one more register in
the loop)

If it doesn't trigger I would probably not bother.


> 
> >
> > That sounds like an unrelated issue.
> >
> > When user space crashes on this the kernel is unaffected.
> 
> This is not a kernel crash. But total freeze as all memory is used by
> network buffers, in no more than 10 seconds.

Ok, but then you have to diagnose this freeze. I'm not sure why you
think it must be this prefetch thingy. If the prefetch issue was
hit then you would just get a normal segfault, not a kernel hang.

e.g. you could write some kind of reduced test case for it and
post it to the netdev mailing list (netdev@oss.sgi.com) 

I'm cc'ing it for you.

> This application receive smalls TCP messages (about 30 bytes), but the
> network stacks allocates 4KB buffers to store this little messages.

Most drivers only allocate MTU size in their receive ring
(normally 1.5K on ethernet). This is rounded to 2K by  the memory allocator.

But most drivers support a rx_copybreak parameter. When the received
packet is smaller than rx_copybreak it is copied to a freshly allocated
buffer with the right size.

In addition the 2.4 stack also supports garbage collection in the TCP
receive buffers. This means even when a driver doesn't do the rx_copybreak
trick and the receive queue of a socket fills up it will copy the data
to fresh, right sized packets by itself.

Another limit for this scenario is that the network stack has internal
limits that supposed to avoid this. These are: each socket has a 
fixed receive buffer size and when more data arrives (including packet
metadata and normal wastage) than the receive buffer allows then it is
still dropped. In addition TCP has a global memory limit that also kicks 
in. And the network stack has a global queue limit that prevents
too much data to be queued from the driver to the higher level
parts (/proc/sys/net/core/netdev_max_backlog). Sometimes the queueing
can also be controlled on the driver level with driver specific
knobs.

This all can be tuned by sysctls in /proc/sys. See Documentation/networking/
ip-sysctl.txt for more details. 

Also the latest 2.6 kernel finally has a writable /proc/sys/vm/min_free_kbytes
again. This controls the amount of memory kept free for interrupts.
Increase that.

> I posted a test application some days ago about this problem and got no
> answers/feedback.

Did you post it to netdev?  On linux-kernel such things get often
lost in the noise.

Also I would contact the driver maintainer, it could be really a driver
Issue.

-Andi

Re: Network buffer hang was Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[dada1]

[-- Attachment #1: Type: text/plain, Size: 6753 bytes --]

> > This is not a kernel crash. But total freeze as all memory is used by
> > network buffers, in no more than 10 seconds.
>
> Ok, but then you have to diagnose this freeze. I'm not sure why you
> think it must be this prefetch thingy. If the prefetch issue was
> hit then you would just get a normal segfault, not a kernel hang.

Well, the machine is a bi-athlon, and I use prefetchnta... thats all.

>
> e.g. you could write some kind of reduced test case for it and
> post it to the netdev mailing list (netdev@oss.sgi.com)

Thanks very much. I'm resending my original mail (with a small test program
attached), at the end of this one.

>
> I'm cc'ing it for you.
>
> > This application receive smalls TCP messages (about 30 bytes), but the
> > network stacks allocates 4KB buffers to store this little messages.
>
> Most drivers only allocate MTU size in their receive ring
> (normally 1.5K on ethernet). This is rounded to 2K by  the memory
allocator.
>
> But most drivers support a rx_copybreak parameter. When the received
> packet is smaller than rx_copybreak it is copied to a freshly allocated
> buffer with the right size.

I'm using e1000 driver , on linux-2.6, this driver doesnt use the
rx_copybreak trick.

>
> In addition the 2.4 stack also supports garbage collection in the TCP
> receive buffers. This means even when a driver doesn't do the rx_copybreak
> trick and the receive queue of a socket fills up it will copy the data
> to fresh, right sized packets by itself.
>
> Another limit for this scenario is that the network stack has internal
> limits that supposed to avoid this. These are: each socket has a
> fixed receive buffer size and when more data arrives (including packet
> metadata and normal wastage) than the receive buffer allows then it is
> still dropped. In addition TCP has a global memory limit that also kicks
> in. And the network stack has a global queue limit that prevents
> too much data to be queued from the driver to the higher level
> parts (/proc/sys/net/core/netdev_max_backlog). Sometimes the queueing
> can also be controlled on the driver level with driver specific
> knobs.
>

cat /proc/sys/net/core/netdev_max_backlog
300

> This all can be tuned by sysctls in /proc/sys. See
Documentation/networking/
> ip-sysctl.txt for more details.
>
> Also the latest 2.6 kernel finally has a writable
/proc/sys/vm/min_free_kbytes
> again. This controls the amount of memory kept free for interrupts.
> Increase that.

Hum I didnt knew this one...

cat  /proc/sys/vm/min_free_kbytes
16384

>
> > I posted a test application some days ago about this problem and got no
> > answers/feedback.
>
> Did you post it to netdev?  On linux-kernel such things get often
> lost in the noise.
>
> Also I would contact the driver maintainer, it could be really a driver
> Issue.
>
> -Andi

Here is the copy of the mail I sent the Sep 1st on linux-kernel & linux-net
:

Hi all

I have an annoying problem with a network server (TCP sockets)

On some stress situation,  LowMemory goes close to 0, and the whole machine
freezes.

When the sockets receive a lot of data, and the server is busy, the TCP
stack just can use too many buffers (in LowMem).

TCP stack uses "size-4096" buffers to store the datas, even if only one byte
is coming from the network.

I tried to change /proc/sys/net/ipv4/tcp_mem, without results.
# echo "1000 10000 15000" >/proc/sys/net/ipv4/tcp_mem

You can reproduce the problem with the test program attached.

# gcc -o crash crash.c
# ulimit -n 20000
# ./crash listen  8888 &
# ./crash call 127.0.0.1:8888 &

grep "size-4096 " /proc/slabinfo
size-4096      40015  40015  4096  1  1 : tunables  24  12  0  : slabdata
40015  40015 0

(thats is 160 Mo, far more than the limit given in
/proc/sys/net/ipv4/tcp_mem)

grep TCP /proc/net/sockstat
TCP: inuse 39996 orphan 0 tw 0 alloc 39997  mem 79986

What is the unit of 'mem' field ? Unless it is 2Ko, the numbers are wrong.

 How may I ask the kernel NOT to use more than 'X Mo' to store TCP messages
?

Thanks

Eric Dumazet

/*
 * Program to freeze a linux box, by using all the LOWMEM
 * A bug on the tcp stack may be the reason
 * Use at your own risk !!
 */

/* Principles :
   A listener accepts incoming tcp sockets, write 40 bytes, and does nothing
with them (no reading)
   A writer establish TCP sockets, sends some data (40 bytes), no more
reading/writing
 */
#include <stdio.h>
# include <sys/socket.h>
# include <netinet/tcp.h>
# include <arpa/inet.h>
# include <netdb.h>
# include <unistd.h>
# include <string.h>

/*
 * Usage :
 *              crash listen port
 *              crash call IP:port
 */
void usage(int code)
{
fprintf(stderr, "Usages :\n") ;
fprintf(stderr, "    crash listen port\n") ;
fprintf(stderr, "    crash call IP:port\n") ;
exit(code) ;
}
const char some_data[40] = "some data.... just some data" ;

void do_listener(const char *string)
{
int port = atoi(string) ;
struct sockaddr_in host, from ;
int fdlisten ;
unsigned int total ;
socklen_t fromlen ;
  memset(&host,0, sizeof(host));
  host.sin_family = AF_INET;
  host.sin_port = htons(port);
  fdlisten = socket(AF_INET, SOCK_STREAM, 0) ;
  if (bind(fdlisten, (struct sockaddr *)&host, sizeof(host)) == -1) {
        perror("bind") ;
        return ;
        }
listen(fdlisten, 10) ;
for (total=0;;total++) {
        int nfd ;
        fromlen = sizeof(from) ;
        nfd = accept(fdlisten, (struct sockaddr *)&from, &fromlen) ;
        if (nfd == -1) break ;
        write(nfd, some_data, sizeof(some_data)) ;
        }
printf("total=%u\n", total) ;
pause() ;
}

void do_caller(const char *string)
{
union {
   int i ;
   char c[4] ;
        } u ;
struct sockaddr_in dest;
int a1, a2, a3, a4, port ;
unsigned int total ;
sscanf(string, "%d.%d.%d.%d:%d", &a1, &a2, &a3, &a4, &port) ;
u.c[0] = a1 ; u.c[1] = a2 ; u.c[2] = a3 ; u.c[3] = a4 ;
for (total=0;;total++) {
    int fd ;
        memset(&dest, 0, sizeof(dest)) ;
        dest.sin_family = AF_INET ;
        dest.sin_port = htons(port) ;
        dest.sin_addr.s_addr = u.i ;
    fd = socket(AF_INET, SOCK_STREAM, 0) ;
        if (fd == -1) break ;
        if (connect(fd, (struct sockaddr *)&dest, sizeof(dest)) == -1) {
                perror("connect") ;
                break ;
                }
        write(fd, some_data, sizeof(some_data)) ;
        }
printf("total=%u\n", total) ;
pause() ;
}

int main(int argc, char *argv[])
{
int listener ;
int caller ;
if (argc != 3) {
        usage(1);
        }
listener = !strcmp(argv[1], "listen") ;
caller = !strcmp(argv[1], "call") ;
if (listener) {
        do_listener(argv[2]) ;
        }
else if (caller) {
        do_caller(argv[2]) ;
        }
else usage(2) ;
return 0 ;
}
/********************************************************************/



[-- Attachment #2: crash.c --]
[-- Type: text/plain, Size: 2624 bytes --]

/*
 * Program to freeze a linux box, by using all the LOWMEM
 * A bug on the tcp stack may be the reason 
 * Use at your own risk !!
 */

/* Principles :
   A listener accepts incoming tcp sockets, write 40 bytes, and does nothing with them (no reading)
   A writer establish TCP sockets, sends some data (40 bytes), no more reading/writing
 */
#include <stdio.h>
# include <sys/socket.h>
# include <netinet/tcp.h>
# include <arpa/inet.h>
# include <netdb.h>
# include <unistd.h>
# include <string.h>

/*
 * Usage :
 *              crash listen port
 *              crash call IP:port
 */
void usage(int code)
{
fprintf(stderr, "Usages :\n") ;
fprintf(stderr, "    crash listen port\n") ;
fprintf(stderr, "    crash call IP:port\n") ;
exit(code) ;
}
const char some_data[40] = "some data.... just some data" ;

void do_listener(const char *string)
{
int port = atoi(string) ;
struct sockaddr_in host, from ;
int fdlisten ;
unsigned int total ;
socklen_t fromlen ;
  memset(&host,0, sizeof(host));
  host.sin_family = AF_INET;
  host.sin_port = htons(port);
  fdlisten = socket(AF_INET, SOCK_STREAM, 0) ;
  if (bind(fdlisten, (struct sockaddr *)&host, sizeof(host)) == -1) {
	perror("bind") ;
	return ;
	}
listen(fdlisten, 10) ;
for (total=0;;total++) {
	int nfd ;
	fromlen = sizeof(from) ;
	nfd = accept(fdlisten, (struct sockaddr *)&from, &fromlen) ;
	if (nfd == -1) break ;
	write(nfd, some_data, sizeof(some_data)) ;
	}
printf("total=%u\n", total) ;
pause() ;
}

void do_caller(const char *string)
{
union {
   int i ;
   char c[4] ;
	} u ;
struct sockaddr_in dest;
int a1, a2, a3, a4, port ;
unsigned int total ;
sscanf(string, "%d.%d.%d.%d:%d", &a1, &a2, &a3, &a4, &port) ;
u.c[0] = a1 ; u.c[1] = a2 ; u.c[2] = a3 ; u.c[3] = a4 ;
for (total=0;;total++) {
    int fd ;
	memset(&dest, 0, sizeof(dest)) ;
	dest.sin_family = AF_INET ;
   	dest.sin_port = htons(port) ;
   	dest.sin_addr.s_addr = u.i ;
    fd = socket(AF_INET, SOCK_STREAM, 0) ;
	if (fd == -1) break ;
	if (connect(fd, (struct sockaddr *)&dest, sizeof(dest)) == -1) {
		perror("connect") ;
		break ;
		}
	write(fd, some_data, sizeof(some_data)) ;
	}
printf("total=%u\n", total) ;
pause() ;
}

int main(int argc, char *argv[])
{
int listener ;
int caller ;
if (argc != 3) {
	usage(1);
	}
listener = !strcmp(argv[1], "listen") ;
caller = !strcmp(argv[1], "call") ;
if (listener) {
	do_listener(argv[2]) ;
	}
else if (caller) {
	do_caller(argv[2]) ;
	}
else usage(2) ;
return 0 ;
}
/********************************************************************/

Re: Network buffer hang was Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[Ben Greear]

dada1 wrote:
>>>This is not a kernel crash. But total freeze as all memory is used by
>>>network buffers, in no more than 10 seconds.

I believe I have hit this as well, by starting 500kbps streams on 500
vlans at once.  My machine locks so hard that I have not been able to
get any info out of it...but the realtek driver shows and error message
about not being able to allocate any buffers if I ping it....

I was using kernel 2.4.21 or so, and had tcp buffers turned up
quite high.  I have seen this on P-IV and AMD (single Athlon) systems,
as well as a VIA C3.


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network buffer hang was Re: [PATCH] 2.6 workaround for Athlon/Opteron prefetch errata

[David S. Miller]

On Thu, 11 Sep 2003 18:46:24 -0700
Ben Greear <greearb@candelatech.com> wrote:

> I was using kernel 2.4.21 or so, and had tcp buffers turned up
> quite high.

You seal your own fate by setting this too high.  It's meant
to keep TCP from commiting too much system memory to itself.