Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Chuck Ebbert]

In-Reply-To: <200607180821.45346.a1426z@gawab.com>

On Tue, 18 Jul 2006 08:21:45 +0300. Al Boldi wrote:

> Going one step further,
> with #define arch_stack_align(x) (x)
> all blips/hits/weirdness are gone
> 
> Which means that either arch_stack_align isn't necessary at all, or 
> randomization isn't working as intended.
> 
> Can somebody prove me wrong here?

Your program seems highly sensitive to any changes, e.g. with the
following code, results with and without the commented lines are
different.  (I changed i to 5555555 because my cpu is slower than
yours and changed main() to call it 10 times.)  This on an AMD
Turion64 1.6GHz running an i386 kernel with stock arch_stack_align()
and randomize_va_space == 1.

void fn()
{
        double x = 0.0, y = 0.0;
        long i = 5555555;
//      static int printed = 0;
//
//      if (!printed) {
//              printed++;
//              printf("&x = %p, &y = %p\n", &x, &y);
//      }

        elapsed(1);
        while (i--)
                fn2(&x,&y);
        printf("%4lu ", elapsed(0));
}

$ ./tst.ex
&x = 0xbfb32d90, &y = 0xbfb32d98
  10    6   10   10    6   10    7   10   10   10   10   10   10   10   10   10   10   10   10   10 msec
$ ./tst.ex
   7   10    6    6    6    6   10   10    6    6    6   10   10    6    6    6    6   10    6    6 msec

BTW when compiled with gcc 4.1.1 using -O3 it just prints all zeros,
so I had to use 3.3.3.
-- 
Chuck

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Chuck Ebbert wrote:
> In-Reply-To: <200607180821.45346.a1426z@gawab.com>
>
> On Tue, 18 Jul 2006 08:21:45 +0300. Al Boldi wrote:
> > Going one step further,
> > with #define arch_stack_align(x) (x)
> > all blips/hits/weirdness are gone
> >
> > Which means that either arch_stack_align isn't necessary at all, or
> > randomization isn't working as intended.
> >
> > Can somebody prove me wrong here?
>
> Your program seems highly sensitive to any changes,

Extremely sensitive.  You may have noticed the strange repetitions in main 
instead of a for loop.  It's like that for a reason:  compile semantics.

> e.g. with the
> following code, results with and without the commented lines are
> different.  (I changed i to 5555555 because my cpu is slower than
> yours and changed main() to call it 10 times.)  This on an AMD
> Turion64 1.6GHz running an i386 kernel with stock arch_stack_align()
> and randomize_va_space == 1.
>
> void fn()
> {
>         double x = 0.0, y = 0.0;
>         long i = 5555555;
> //      static int printed = 0;
> //
> //      if (!printed) {
> //              printed++;
> //              printf("&x = %p, &y = %p\n", &x, &y);
> //      }
>
>         elapsed(1);
>         while (i--)
>                 fn2(&x,&y);
>         printf("%4lu ", elapsed(0));
> }
>
> $ ./tst.ex
> &x = 0xbfb32d90, &y = 0xbfb32d98
>   10    6   10   10    6   10    7   10   10   10   10   10   10   10   10
>   10   10   10   10   10 msec $ ./tst.ex
>    7   10    6    6    6    6   10   10    6    6    6   10   10    6    6
>    6    6   10    6    6 msec

So what happens when this is renamed/ sh -c'd/ randomize off'd/ while'd/ 
compiled w/ -Os?  Keep in mind, we want to surface a kernel problem, not a 
compiler problem, even though the compiler may have a problem.

Do you get the same kind of weirdness?

Mind you, only i686+ show this problem, i3/4/586 seem ok.  Don't know about AMD.

> BTW when compiled with gcc 4.1.1 using -O3 it just prints all zeros,
> so I had to use 3.3.3.

With your changes on:

stock kernel, gcc.322 -O3 doesn't show a slowdown.

stock kernel, randomize_va_space=0, gcc.322 -Os tstExec.c, while :; do ./a.out; done
&x = 0xbffff874, &y = 0xbffff86c   28   28   28   27   27   27   27   30   28   27 msec
&x = 0xbffff874, &y = 0xbffff86c   27   27   27   27   27   29   27   27   27   28 msec
&x = 0xbffff874, &y = 0xbffff86c   27   27   27   27   29   27   28   28   27   27 msec
&x = 0xbffff874, &y = 0xbffff86c   28   27   27   29   27   27   27   28   27   27 msec
&x = 0xbffff874, &y = 0xbffff86c   27   30   27   28   27   27   27   27   27   27 msec
&x = 0xbffff874, &y = 0xbffff86c   27   29   27   27   28   27   27   27   29   27 msec

stock kernel, randomize_va_space=1, gcc.322 -Os tstExec.c, while :; do ./a.out; done
&x = 0xbfe2e614, &y = 0xbfe2e60c   29   28   27   29   27   27   27   28   27   28 msec
&x = 0xbfd6a104, &y = 0xbfd6a0fc   55   56   56   55   57   59   55   56   55   55 msec
&x = 0xbf91d454, &y = 0xbf91d44c   27   27   27   28   27   28   27   29   27   27 msec
&x = 0xbf941e84, &y = 0xbf941e7c   55   56   56   56   56   56   57   59   55   55 msec
&x = 0xbfa75834, &y = 0xbfa7582c   28   27   27   27   27   27   27   27   27   27 msec
&x = 0xbfb58634, &y = 0xbfb5862c   27   30   27   28   27   27   27   27   28   27 msec

stock kernel, randomize_va_space=0, gcc.322 -Os tstExec.c, while :; do ./a.out; done
  28   28   28   29   27   27   27   28   27   27 msec
  27   27   27   27   29   28   27   27   27   27 msec
  27   29   28   29   27   28   28   27   27   27 msec
  27   31   27   28   27   27   27   27   27   29 msec
  28   27   28   27   27   27   27   27   29   27 msec
  27   28   29   29   29   27   29   27   27   27 msec

stock kernel, randomize_va_space=1, gcc.322 -Os tstExec.c, while :; do ./a.out; done
  55   54   54   55   55   56   56   56   58   54 msec
  55   55   56   56   56   56   57   56   55   55 msec
  27   27   28   27   27   27   28   27   28   27 msec
  30   28   27   27   27   28   27   27   27   28 msec
  27   27   27   28   27   27   28   30   28   27 msec
  55   55   55   56   55   56   55   58   57   55 msec
  55   55   56   55   56   58   56   55   55   55 msec

Can you confirm these numbers?

Thanks for your input!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Al Boldi wrote:
> Chuck Ebbert wrote:
> > On Tue, 18 Jul 2006 08:21:45 +0300. Al Boldi wrote:
> > > Going one step further,
> > > with #define arch_stack_align(x) (x)
> > > all blips/hits/weirdness are gone
> > >
> > > Which means that either arch_stack_align isn't necessary at all, or
> > > randomization isn't working as intended.
> > >
> > > Can somebody prove me wrong here?
> >
> > Your program seems highly sensitive to any changes,
>
> Extremely sensitive.  You may have noticed the strange repetitions in main
> instead of a for loop.  It's like that for a reason:  compile semantics.
>
> > $ ./tst.ex
> > &x = 0xbfb32d90, &y = 0xbfb32d98
> >   10    6   10   10    6   10    7   10   10   10   10   10   10   10  
>
> With your changes on:
>
> stock kernel, randomize_va_space=0, gcc.322 -Os tstExec.c,
> while :; do ./a.out; done
> &x = 0xbffff874, &y = 0xbffff86c   28   28
> &x = 0xbffff874, &y = 0xbffff86c   27   27  
> &x = 0xbffff874, &y = 0xbffff86c   27   27
> &x = 0xbffff874, &y = 0xbffff86c   28   27
> &x = 0xbffff874, &y = 0xbffff86c   27   30
> &x = 0xbffff874, &y = 0xbffff86c   27   29
>
> stock kernel, randomize_va_space=1, gcc.322 -Os tstExec.c,
> while :; do ./a.out; done
> &x = 0xbfe2e614, &y = 0xbfe2e60c   29   28
> &x = 0xbfd6a104, &y = 0xbfd6a0fc   55   56  
> &x = 0xbf91d454, &y = 0xbf91d44c   27   27
> &x = 0xbf941e84, &y = 0xbf941e7c   55   56
> &x = 0xbfa75834, &y = 0xbfa7582c   28   27
> &x = 0xbfb58634, &y = 0xbfb5862c   27   30

After closer inspection, it looks like addresses ending with 3c,7c,bc,fc 
cause a slowdown on P4, while addresses ending with 1c,3c,5c,7c,9c,bc,dc,fc 
cause a slowdown on P2.

Any easy way to instruct the kernel to skip those addresses?

Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Chuck Ebbert wrote:
> On Mon, 24 Jul 2006 18:57:46 +0300, Al Boldi wrote:
> > > With your changes on:
> > >
> > > stock kernel, randomize_va_space=0, gcc.322 -Os tstExec.c,
> > > while :; do ./a.out; done
> > > &x = 0xbffff874, &y = 0xbffff86c   28   28
> > > &x = 0xbffff874, &y = 0xbffff86c   27   27
> > > &x = 0xbffff874, &y = 0xbffff86c   27   27
> > > &x = 0xbffff874, &y = 0xbffff86c   28   27
> > > &x = 0xbffff874, &y = 0xbffff86c   27   30
> > > &x = 0xbffff874, &y = 0xbffff86c   27   29
> > >
> > > stock kernel, randomize_va_space=1, gcc.322 -Os tstExec.c,
> > > while :; do ./a.out; done
> > > &x = 0xbfe2e614, &y = 0xbfe2e60c   29   28
> > > &x = 0xbfd6a104, &y = 0xbfd6a0fc   55   56
> > > &x = 0xbf91d454, &y = 0xbf91d44c   27   27
> > > &x = 0xbf941e84, &y = 0xbf941e7c   55   56
> > > &x = 0xbfa75834, &y = 0xbfa7582c   28   27
> > > &x = 0xbfb58634, &y = 0xbfb5862c   27   30
> >
> > After closer inspection, it looks like addresses ending with 3c,7c,bc,fc
> > cause a slowdown on P4, while addresses ending with
> > 1c,3c,5c,7c,9c,bc,dc,fc cause a slowdown on P2.
>
> Those addresses cause 'y' to span a cacheline (P4 = 64 bytes, P2 = 32.)
> Even when the kernel aligns to 128 bytes this could happen depending
> on how deeply you nest functions.
>
> > Any easy way to instruct the kernel to skip those addresses?
>
> First, I think you need to define locals in order of decreasing size.
> IOW 'x' and 'y' need to be first inside fn(), but that may not work
> when things get inlined.  So using the '-malign-double' GCC option,
> or forcing alignment with '__attribute__ ((aligned(8)))' for each variable
> might be better.
>
> Then you have to make sure the stack is aligned. See
> '-mpreferred-stack-boundary'.

This would imply a recompile, what about precompiled dists?  Do they compile 
the sources this way?

> I still think the kernel should be aligning the stack to 128 bytes anyway.

I think so too, but can you see how randomization aggravates the situation?


Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Chuck Ebbert]

In-Reply-To: <200607241857.46594.a1426z@gawab.com>

On Mon, 24 Jul 2006 18:57:46 +0300, Al Boldi wrote:
>
> > With your changes on:
> >
> > stock kernel, randomize_va_space=0, gcc.322 -Os tstExec.c,
> > while :; do ./a.out; done
> > &x = 0xbffff874, &y = 0xbffff86c   28   28
> > &x = 0xbffff874, &y = 0xbffff86c   27   27  
> > &x = 0xbffff874, &y = 0xbffff86c   27   27
> > &x = 0xbffff874, &y = 0xbffff86c   28   27
> > &x = 0xbffff874, &y = 0xbffff86c   27   30
> > &x = 0xbffff874, &y = 0xbffff86c   27   29
> >
> > stock kernel, randomize_va_space=1, gcc.322 -Os tstExec.c,
> > while :; do ./a.out; done
> > &x = 0xbfe2e614, &y = 0xbfe2e60c   29   28
> > &x = 0xbfd6a104, &y = 0xbfd6a0fc   55   56  
> > &x = 0xbf91d454, &y = 0xbf91d44c   27   27
> > &x = 0xbf941e84, &y = 0xbf941e7c   55   56
> > &x = 0xbfa75834, &y = 0xbfa7582c   28   27
> > &x = 0xbfb58634, &y = 0xbfb5862c   27   30
>
> After closer inspection, it looks like addresses ending with 3c,7c,bc,fc 
> cause a slowdown on P4, while addresses ending with 1c,3c,5c,7c,9c,bc,dc,fc 
> cause a slowdown on P2.
>

Those addresses cause 'y' to span a cacheline (P4 = 64 bytes, P2 = 32.)
Even when the kernel aligns to 128 bytes this could happen depending
on how deeply you nest functions.

> Any easy way to instruct the kernel to skip those addresses?

First, I think you need to define locals in order of decreasing size.
IOW 'x' and 'y' need to be first inside fn(), but that may not work
when things get inlined.  So using the '-malign-double' GCC option,
or forcing alignment with '__attribute__ ((aligned(8)))' for each variable
might be better.

Then you have to make sure the stack is aligned. See
'-mpreferred-stack-boundary'.

I still think the kernel should be aligning the stack to 128 bytes anyway.

-- 
Chuck

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Paulo Marques wrote:
> Al Boldi wrote:
> >[...] > void fn() {
> >
> > 	long i = 9999999;
> > 	double x,y;
> >
> > 	elapsed(1);
> > 	while (i--) fn2(&x,&y);
> > 	printf("%4lu ",elapsed(0));
> > }
>
> You are not initializing x and y and with -Os at least my gcc really
> uses floating point load/store operations to handle that code.

Thanks for pointing that out.

I was really waiting for someone to critique this, but keep in mind this code 
tries to surface a performance problem, and any modification changes the 
semantics of the compiled code, which then may yield different results.

> Maybe the coprocessor has a hard time normalizing certain garbage on the
> stack, but without/with randomization the data comes from other
> addresses and you're just lucky with the contents.

Good point, but this random garbage makes the test even more realistic, as 
this code would thus cover more variations without actually coding for it.

> Does this also happens if you add a "x=0, y=0;" line to that function?

with arch_stack_align using 0xf
gcc -Os tstExec.c
randomization on
causes 2x blips/hits
randomization off
causes no blips/hits
mv a.out tstExec
causes continuous 2x slowdown
sh -c ./tstExec
causes slowdown to disappear (can somebody explain this weirdness?)

with arch_stack_align using 0x7f
all weirdness is gone
gcc -O3 tstExec.c
randomization on
causes some minor blips/hits
randomization off
causes even less blips/hits

Going one step further,
with #define arch_stack_align(x) (x)
all blips/hits/weirdness are gone

Which means that either arch_stack_align isn't necessary at all, or 
randomization isn't working as intended.

Can somebody prove me wrong here?

Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Paulo Marques]

Al Boldi wrote:
>[...] > void fn() {
> 
> 	long i = 9999999;
> 	double x,y;
> 
> 	elapsed(1);
> 	while (i--) fn2(&x,&y);
> 	printf("%4lu ",elapsed(0));
> }

You are not initializing x and y and with -Os at least my gcc really 
uses floating point load/store operations to handle that code.

Maybe the coprocessor has a hard time normalizing certain garbage on the 
stack, but without/with randomization the data comes from other 
addresses and you're just lucky with the contents.

Does this also happens if you add a "x=0, y=0;" line to that function?

-- 
Paulo Marques - www.grupopie.com

Pointy-Haired Boss: I don't see anything that could stand in our way.
            Dilbert: Sanity? Reality? The laws of physics?

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Chuck Ebbert wrote:
> On Sat, 15 Jul 2006 17:09:45 +0300, Al Boldi wrote:
> > Randomization on.  Executable runs with 8x blips/hits.
> > Randomization off.  Executable runs without blips/hits.
> > With randomization off, a mere rename causes an 8x-slowdown to occur. 
> > Run this renamed executable through sh -c ./tstExec, and the slowdown
> > disappears.  Really weired :)
>
> Does this help at all?  I don't have a space heater^W^WPentium IV
> to test on.

P4 is not a space heater, it's a stove.

> --- 2.6.18-rc1-nb.orig/arch/i386/kernel/process.c
> +++ 2.6.18-rc1-nb/arch/i386/kernel/process.c
> @@ -890,5 +890,5 @@ unsigned long arch_align_stack(unsigned
>  {
>  	if (randomize_va_space)
>  		sp -= get_random_int() % 8192;
> -	return sp & ~0xf;
> +	return sp & ~0x7f;
>  }

Thanks! That did the trick!

The weirdness is completely gone, as are the continuous slowdowns, but I 
still get these:

on mdk9.1/gcc3.2.2/2.6.17.4
# gcc -O3 tstExec.c
# echo 0 > /proc/sys/kernel/randomize_va_space
# while :; do ./a.out;done

   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5    7    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    6 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec
   5    5    5    5    5    5    5    5    6    5 msec

on mdk9.1/gcc3.2.2/2.6.17.4
# gcc -O3 tstExec.c
# echo 1 > /proc/sys/kernel/randomize_va_space
# while :; do ./a.out;done

   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    6    5    5   13    5   13    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5   11    5   13    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    6    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5    6    5   13    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5    6    5   13    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5    6    5   13    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5   13    5    6    5 msec
   5    5    5    5    5    5   13    5    7    5 msec
   5    5    5    5    5    5   13    5   13    5 msec
   5    5    5    5    5    5    6    5    6    5 msec
   5    5    5    5    5    5    6    5    6    5 msec

on rhel4/gcc4.0.1/2.6.17.4
# gcc -O3 tstExec.c
# echo 0 > /proc/sys/kernel/randomize_va_space
# while :; do ./a.out;done

   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   6    6    6    6    6    6    6    6    6    6 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    6    5    5    5    5    5    5    5    5 msec

on rhel4/gcc4.0.1/2.6.17.4
# gcc -O3 tstExec.c
# echo 1 > /proc/sys/kernel/randomize_va_space
# while :; do ./a.out;done

  13   13   13   13   13   13   13   13   13   13 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   12    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
  13   13   13   13   13   13   13   13   13   13 msec
  13   13   13   13   13   13   13   13   13   14 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   14 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec
   5    5    5    5    5    5    5    5    5    5 msec
  13   13   13   13   13   13   13   13   13   13 msec

Thanks again!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Chuck Ebbert]

In-Reply-To: <200607151709.45870.a1426z@gawab.com>

On Sat, 15 Jul 2006 17:09:45 +0300, Al Boldi wrote:

> Randomization on.  Executable runs with 8x blips/hits.
> Randomization off.  Executable runs without blips/hits.
> With randomization off, a mere rename causes an 8x-slowdown to occur.  Run 
> this renamed executable through sh -c ./tstExec, and the slowdown 
> disappears.  Really weired :)

Does this help at all?  I don't have a space heater^W^WPentium IV
to test on.

--- 2.6.18-rc1-nb.orig/arch/i386/kernel/process.c
+++ 2.6.18-rc1-nb/arch/i386/kernel/process.c
@@ -890,5 +890,5 @@ unsigned long arch_align_stack(unsigned 
 {
 	if (randomize_va_space)
 		sp -= get_random_int() % 8192;
-	return sp & ~0xf;
+	return sp & ~0x7f;
 }
-- 
Chuck
 "You can't read a newspaper if you can't read."  --George W. Bush

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Arjan van de Ven wrote:
> On Sat, 2006-07-15 at 17:09 +0300, Al Boldi wrote:
> > Arjan van de Ven wrote:
> > > On Sat, 2006-07-15 at 14:29 +0300, Al Boldi wrote:
> > > > Arjan van de Ven wrote:
> > > > > > BTW, why does randomize_stack_top() mod against (8192*1024)
> > > > > > instead of (8192) like arch_align_stack()?
> > > > >
> > > > >  because it wants to randomize for 8Mb, unlike arch_align_stack
> > > > > which wants to randomize the last 8Kb within this 8Mb ;)
> > > >
> > > > Randomizing twice?
> > >
> > > a VMA can only be randomized in 4Kb (well page size) granularity, so
> > > the 8Mb randomization can only work in that 4Kb unit, the "second"
> > > randomization can work in 16 byte granularity.
> > >
> > > > There is even a case where a mere rename or running through an extra
> > > > shell causes a slowdown.  And that's with randomization turned off.
> > >
> > > randomization off will slow stuff down yes... you get cache alias
> > > contention that way.
>
> a question.. do you have prelink installed/active on your system? that
> may very well mess with timings...

No, not that I am aware of.  How can I find out?
I confirmed this on a standard rhel4 virgin-install.  Does rhel4 have prelink 
enabled by default?

But even so, I don't think this is a timing issue, as the slowdowns are so 
huge it is obvious even without timing.

And 2.4.31 doesn't have this problem.

Did you try it?

Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Arjan van de Ven]

On Sat, 2006-07-15 at 17:09 +0300, Al Boldi wrote:
> Arjan van de Ven wrote:
> > On Sat, 2006-07-15 at 14:29 +0300, Al Boldi wrote:
> > > Arjan van de Ven wrote:
> > > > > BTW, why does randomize_stack_top() mod against (8192*1024) instead
> > > > > of (8192) like arch_align_stack()?
> > > >
> > > >  because it wants to randomize for 8Mb, unlike arch_align_stack which
> > > > wants to randomize the last 8Kb within this 8Mb ;)
> > >
> > > Randomizing twice?
> >
> > a VMA can only be randomized in 4Kb (well page size) granularity, so the
> > 8Mb randomization can only work in that 4Kb unit, the "second"
> > randomization can work in 16 byte granularity.
> >
> > > There is even a case where a mere rename or running through an extra
> > > shell causes a slowdown.  And that's with randomization turned off.
> >
> > randomization off will slow stuff down yes... you get cache alias
> > contention that way.

a question.. do you have prelink installed/active on your system? that
may very well mess with timings...

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Arjan van de Ven wrote:
> On Sat, 2006-07-15 at 14:29 +0300, Al Boldi wrote:
> > Arjan van de Ven wrote:
> > > > BTW, why does randomize_stack_top() mod against (8192*1024) instead
> > > > of (8192) like arch_align_stack()?
> > >
> > >  because it wants to randomize for 8Mb, unlike arch_align_stack which
> > > wants to randomize the last 8Kb within this 8Mb ;)
> >
> > Randomizing twice?
>
> a VMA can only be randomized in 4Kb (well page size) granularity, so the
> 8Mb randomization can only work in that 4Kb unit, the "second"
> randomization can work in 16 byte granularity.
>
> > There is even a case where a mere rename or running through an extra
> > shell causes a slowdown.  And that's with randomization turned off.
>
> randomization off will slow stuff down yes... you get cache alias
> contention that way.

Randomization on.  Executable runs with 8x blips/hits.
Randomization off.  Executable runs without blips/hits.
With randomization off, a mere rename causes an 8x-slowdown to occur.  Run 
this renamed executable through sh -c ./tstExec, and the slowdown 
disappears.  Really weired :)

> > 2.4.31 doesn't show these slowdowns.
>
> 2.4.31 randomizes the stack with 8Kb.
>
> > What is 2.6 doing?
>
> you're not providing a lot of info ;)
>
> why do you suspect randomization as cause for whatever slowdown you are
> seeing?

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips/hits to go away, 
most of the time.

> What kind of slowdown are you seeing?

see below.

Beware, this is highly compiler/glibc/distribution dependent.
My test environment is mdk9.1, gcc-3.2.2-3, 2.6.17.4, compile with -Os switch
Confirmed on rhel4, gcc-4.0.1, 2.6.9-5.EL, compile with no switches

Thanks!

--
Al

---
#include <stdio.h>
#include <sys/time.h>

unsigned long elapsed(int start) {

	static struct timeval s,e;
	if (start) return gettimeofday(&s, NULL);
	gettimeofday(&e, NULL);
	return ((e.tv_sec - s.tv_sec) * 1000 + (e.tv_usec - s.tv_usec) / 1000);
}

void fn2(double *x, double *y) {

	*x = *y;
}

void fn() {

	long i = 9999999;
	double x,y;

	elapsed(1);
	while (i--) fn2(&x,&y);
	printf("%4lu ",elapsed(0));
}

int main(int argc, char **argv) {

	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	fn();
	printf("msec\n");
	
	return 0;
}

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Arjan van de Ven]

On Sat, 2006-07-15 at 14:29 +0300, Al Boldi wrote:
> Arjan van de Ven wrote:
> > > BTW, why does randomize_stack_top() mod against (8192*1024) instead of
> > > (8192) like arch_align_stack()?
> >
> >  because it wants to randomize for 8Mb, unlike arch_align_stack which
> > wants to randomize the last 8Kb within this 8Mb ;)
> 
> Randomizing twice?

a VMA can only be randomized in 4Kb (well page size) granularity, so the
8Mb randomization can only work in that 4Kb unit, the "second"
randomization can work in 16 byte granularity.

> There is even a case where a mere rename or running through an extra shell 
> causes a slowdown.  And that's with randomization turned off.

randomization off will slow stuff down yes... you get cache alias
contention that way.




> 2.4.31 doesn't show these slowdowns.

2.4.31 randomizes the stack with 8Kb.

> What is 2.6 doing?

you're not providing a lot of info ;)

why do you suspect randomization as cause for whatever slowdown you are
seeing? What kind of slowdown are you seeing?

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Arjan van de Ven wrote:
> > BTW, why does randomize_stack_top() mod against (8192*1024) instead of
> > (8192) like arch_align_stack()?
>
>  because it wants to randomize for 8Mb, unlike arch_align_stack which
> wants to randomize the last 8Kb within this 8Mb ;)

Randomizing twice?

Anyway, I tried different combinations of turning off randomization in both 
functions and got mixed results, so it looks like there is some interaction 
here.

Trying different compiler versions and switches also show different results.

Calling these slowdowns blips is really an understatement, as there are cases 
which lock into 800% hits. i.e: processes that, when repeatedly called, lock 
into a continuous 8x slowdown on i686P4.

There is even a case where a mere rename or running through an extra shell 
causes a slowdown.  And that's with randomization turned off.

2.4.31 doesn't show these slowdowns.

What is 2.6 doing?


Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Arjan van de Ven]

> BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) 
> like arch_align_stack()?
> 

 because it wants to randomize for 8Mb, unlike arch_align_stack which
wants to randomize the last 8Kb within this 8Mb ;)

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Frank van Maarseveen wrote:
> On Wed, Jul 12, 2006 at 06:03:18PM +0200, Andi Kleen wrote:
> > Al Boldi <a1426z@gawab.com> writes:
> > > Frank van Maarseveen wrote:
> > > > Do not randomize stack location unless current->personality permits
> > > > it.
>
> [...]
>
> > > It still blips on my system.
> > >
> > > echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.
> > >
> > > ???
> >
> > fs/binfmt_elf.c:randomize_stack_top would need the same check
>
> Actually, randomize_stack_top() checks
>
> 	if (current->flags & PF_RANDOMIZE) {
>
> and it's only called from load_elf_binary() right after this:
>
> 	if (!(current->personality & ADDR_NO_RANDOMIZE) && randomize_va_space)
> 		current->flags |= PF_RANDOMIZE;
>
> Further on create_elf_tables() is called and that one calls
> arch_align_stack() so maybe it is more appropriate to test (current->flags
> & PF_RANDOMIZE) in arch_align_stack() instead of recomputing it.

exec.c uses arch_align_stack() also for non-elf.

BTW, why does randomize_stack_top() mod against (8192*1024) instead of (8192) 
like arch_align_stack()?


Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Frank van Maarseveen]

On Wed, Jul 12, 2006 at 06:03:18PM +0200, Andi Kleen wrote:
> Al Boldi <a1426z@gawab.com> writes:
> 
> > Frank van Maarseveen wrote:
> > >
> > > Do not randomize stack location unless current->personality permits it.
[...]
> > 
> > It still blips on my system.
> > 
> > echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.
> > 
> > ???
> 
> fs/binfmt_elf.c:randomize_stack_top would need the same check

Actually, randomize_stack_top() checks

	if (current->flags & PF_RANDOMIZE) {

and it's only called from load_elf_binary() right after this:

	if (!(current->personality & ADDR_NO_RANDOMIZE) && randomize_va_space)
		current->flags |= PF_RANDOMIZE;

Further on create_elf_tables() is called and that one calls arch_align_stack()
so maybe it is more appropriate to test (current->flags & PF_RANDOMIZE) in
arch_align_stack() instead of recomputing it.

-- 
Frank

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Arjan van de Ven wrote:
> On Wed, 2006-07-12 at 23:12 +0300, Al Boldi wrote:
> > It already checks for it, but makes no difference.
> >
> > Thanks for looking into this!
>
> well glibc also randomizes things a bit (for better cache coloring) ...

Any way to turn it off?

Thanks!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Arjan van de Ven]

On Wed, 2006-07-12 at 23:12 +0300, Al Boldi wrote:

> It already checks for it, but makes no difference.
> 
> Thanks for looking into this!
> 

well glibc also randomizes things a bit (for better cache coloring) ...

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Andi Kleen wrote:
> Al Boldi <a1426z@gawab.com> writes:
> > Frank van Maarseveen wrote:
> > > Do not randomize stack location unless current->personality permits
> > > it.
> > >
> > > Signed-off-by: Frank van Maarseveen <frankvm@frankvm.com>
> > > ---
> > >
> > > The problem seems also present in
> > >
> > >         arch/um/kernel/process_kern.c
> > >         arch/x86_64/kernel/process.c
> > >
> > >  arch/i386/kernel/process.c |    3 ++-
> > >  1 files changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
> > > --- a/arch/i386/kernel/process.c        2006-06-23 16:08:13.000000000
> > > +0200 +++ b/arch/i386/kernel/process.c        2006-07-11
> > > 14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
> > >  #include <linux/kallsyms.h>
> > >  #include <linux/ptrace.h>
> > >  #include <linux/random.h>
> > > +#include <linux/personality.h>
> > >
> > >  #include <asm/uaccess.h>
> > >  #include <asm/pgtable.h>
> > > @@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
> > >
> > >  unsigned long arch_align_stack(unsigned long sp)
> > >  {
> > > -       if (randomize_va_space)
> > > +       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
> > > randomize_va_space) sp -= get_random_int() % 8192;
> > >         return sp & ~0xf;
> > >  }
> >
> > It still blips on my system.
> >
> > echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.
> >
> > ???
>
> fs/binfmt_elf.c:randomize_stack_top would need the same check

It already checks for it, but makes no difference.

Thanks for looking into this!

--
Al

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Andi Kleen]

Al Boldi <a1426z@gawab.com> writes:

> Frank van Maarseveen wrote:
> >
> > Do not randomize stack location unless current->personality permits it.
> >
> > Signed-off-by: Frank van Maarseveen <frankvm@frankvm.com>
> > ---
> >
> > The problem seems also present in
> >
> >         arch/um/kernel/process_kern.c
> >         arch/x86_64/kernel/process.c
> >
> >  arch/i386/kernel/process.c |    3 ++-
> >  1 files changed, 2 insertions(+), 1 deletion(-)
> >
> > diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
> > --- a/arch/i386/kernel/process.c        2006-06-23 16:08:13.000000000
> > +0200 +++ b/arch/i386/kernel/process.c        2006-07-11
> > 14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
> >  #include <linux/kallsyms.h>
> >  #include <linux/ptrace.h>
> >  #include <linux/random.h>
> > +#include <linux/personality.h>
> >
> >  #include <asm/uaccess.h>
> >  #include <asm/pgtable.h>
> > @@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
> >
> >  unsigned long arch_align_stack(unsigned long sp)
> >  {
> > -       if (randomize_va_space)
> > +       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
> > randomize_va_space) sp -= get_random_int() % 8192;
> >         return sp & ~0xf;
> >  }
> 
> It still blips on my system.
> 
> echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.
> 
> ???

fs/binfmt_elf.c:randomize_stack_top would need the same check
-Andi

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Arjan van de Ven]

On Tue, 2006-07-11 at 17:22 +0200, Frank van Maarseveen wrote:
> Do not randomize stack location unless current->personality permits it.
> 
> Signed-off-by: Frank van Maarseveen <frankvm@frankvm.com>
> ---

hmm I could have sworn this was taken care of already, 

Acked-by: Arjan van de Ven <arjan@linux.intel.com>


thinking about it... prior to adding randomization, there already was 8K
randomization of the stack pointer.. we probably only made the new
randomization conditional on the personality..
(and to be honest for performance reasons you really really want that 8K
at least on the netburst architectures)

Re: [PATCH] x86: Don't randomize stack unless current->personality permits it

[Al Boldi]

Frank van Maarseveen wrote:
>
> Do not randomize stack location unless current->personality permits it.
>
> Signed-off-by: Frank van Maarseveen <frankvm@frankvm.com>
> ---
>
> The problem seems also present in
>
>         arch/um/kernel/process_kern.c
>         arch/x86_64/kernel/process.c
>
>  arch/i386/kernel/process.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletion(-)
>
> diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
> --- a/arch/i386/kernel/process.c        2006-06-23 16:08:13.000000000
> +0200 +++ b/arch/i386/kernel/process.c        2006-07-11
> 14:39:20.000000000 +0200 @@ -38,6 +38,7 @@
>  #include <linux/kallsyms.h>
>  #include <linux/ptrace.h>
>  #include <linux/random.h>
> +#include <linux/personality.h>
>
>  #include <asm/uaccess.h>
>  #include <asm/pgtable.h>
> @@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
>
>  unsigned long arch_align_stack(unsigned long sp)
>  {
> -       if (randomize_va_space)
> +       if (!(current->personality & ADDR_NO_RANDOMIZE) &&
> randomize_va_space) sp -= get_random_int() % 8192;
>         return sp & ~0xf;
>  }

It still blips on my system.

echo 0 > /proc/sys/kernel/randomize_va_space makes the blips go away.

???

Thanks!

--
Al

[PATCH] x86: Don't randomize stack unless current->personality permits it

[Frank van Maarseveen]

Do not randomize stack location unless current->personality permits it.

Signed-off-by: Frank van Maarseveen <frankvm@frankvm.com>
---

The problem seems also present in

	arch/um/kernel/process_kern.c
	arch/x86_64/kernel/process.c

 arch/i386/kernel/process.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -rup a/arch/i386/kernel/process.c b/arch/i386/kernel/process.c
--- a/arch/i386/kernel/process.c	2006-06-23 16:08:13.000000000 +0200
+++ b/arch/i386/kernel/process.c	2006-07-11 14:39:20.000000000 +0200
@@ -38,6 +38,7 @@
 #include <linux/kallsyms.h>
 #include <linux/ptrace.h>
 #include <linux/random.h>
+#include <linux/personality.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
@@ -898,7 +899,7 @@ asmlinkage int sys_get_thread_area(struc
 
 unsigned long arch_align_stack(unsigned long sp)
 {
-	if (randomize_va_space)
+	if (!(current->personality & ADDR_NO_RANDOMIZE) && randomize_va_space)
 		sp -= get_random_int() % 8192;
 	return sp & ~0xf;
 }