Feedback on 4.9 performance after PTI fixes

Feedback on 4.9 performance after PTI fixes

[Willy Tarreau]

Hi,

I managed to take a bit of time to run some more tests on PTI both
native and hosted in KVM, on stable versions built with
CONFIG_PAGE_TABLE_ISOLATION=y. Here it's 4.9.75, used both on the
host and the VM. I could compare pti=on/off both in the host and the
VM. A single CPU was exposed in the VM.

It was running on my laptop (core i7 3320M at 2.6 GHz, 3.3 GHz single
core turbo).

The test was run on haproxy's ability to forward connections. The
results are below :

   Host  |  Guest  | conn/s  | ratio_to_host |  ratio_to_VM | Notes
---------+---------+---------+---------------+--------------+----------------
 pti=off |   -     |  27400  |     100.0%    |      -       | host reference
 pti=off | pti=off |  24200  |      88.3%    |    100.0%    | VM reference 
 pti=off | pti=on  |  13300  |      48.5%    |     55.0%    |
 pti=on  |   -     |  23800  |      86.9%    |      -	    | protected host
 pti=on  | pti=off |  23100  |      84.3%    |     95.5%    |
 pti=on  | pti=on  |  13300  |      48.5%    |     55.0%    |

The ratio_to_host column shows the performance relative to the host
with pti=off. The ratio_to_VM column shows the performance relative to
the VM running with pti=off in a host also having pti=off (ie:
performance before upgrading the systems).

On this test we see a few things :
  - the performance impact on the native host is around 13%

  - the highest performance impact on VMs comes from having PTI on the
    guest kernel (-45%). At this point it makes no difference whether
    the host kernel has it or not.

  - the host kernel's protection has a very limited impact on the guest
    system's performance (-4.5%), which is probably nice for some cloud
    users who might want to take the risk of turning the protection off
    on their VMs.

The impact inside VMs is quite big but it's not where we usuall install
processes sensitive to syscall performance. I could find an even higher
impact on a packet generation program dropping from 2.5 Mpps to 600kpps
in the VM after the fix, but it doesn't make much sense to do this in
VMs so I don't really care.

I have not yet tried the retpoline patches.

Regards,
Willy

Re: Feedback on 4.9 performance after PTI fixes

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On Sun, 2018-01-07 at 11:18 +0100, Willy Tarreau wrote:
>   - the highest performance impact on VMs comes from having PTI on the
>     guest kernel (-45%). At this point it makes no difference whether
>     the host kernel has it or not.

Hi Willy,

out of curiosity, is the pcid/invpcid flags exposed to and used by your guest
CPU? It might very well that the PCID optimisations are not used by the guests
here, and it might be worth either checking on bare metal or with the PCID
optimisations enabled.

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Feedback on 4.9 performance after PTI fixes

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Mon, 2018-01-08 at 18:07 +0100, Yves-Alexis Perez wrote:
> On Sun, 2018-01-07 at 11:18 +0100, Willy Tarreau wrote:
> >   - the highest performance impact on VMs comes from having PTI on the
> >     guest kernel (-45%). At this point it makes no difference whether
> >     the host kernel has it or not.
> 
> Hi Willy,
> 
> out of curiosity, is the pcid/invpcid flags exposed to and used by your guest
> CPU? It might very well that the PCID optimisations are not used by the guests
> here, and it might be worth either checking on bare metal or with the PCID
> optimisations enabled.

More details on this: https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: Feedback on 4.9 performance after PTI fixes

[David Laight]

From: Willy Tarreau
> Sent: 07 January 2018 10:19
...
> The impact inside VMs is quite big but it's not where we usuall install
> processes sensitive to syscall performance. I could find an even higher
> impact on a packet generation program dropping from 2.5 Mpps to 600kpps
> in the VM after the fix, but it doesn't make much sense to do this in
> VMs so I don't really care.

Why not?
It will be related to the cost of sending (and probably receiving)
network traffic in a VM.
This is something that is done a lot.

Maybe not packet generation, but a UDP/IP benchmark inside a VM would
be sensible.
It may well be that moderate ethernet packet rates cause a massive
performance drop when the host kernel has PTI enabled.

	David

Re: Feedback on 4.9 performance after PTI fixes

[Willy Tarreau]

Hi Yves-Alexis,

On Mon, Jan 08, 2018 at 06:07:54PM +0100, Yves-Alexis Perez wrote:
> On Sun, 2018-01-07 at 11:18 +0100, Willy Tarreau wrote:
> >   - the highest performance impact on VMs comes from having PTI on the
> >     guest kernel (-45%). At this point it makes no difference whether
> >     the host kernel has it or not.
> 
> Hi Willy,
> 
> out of curiosity, is the pcid/invpcid flags exposed to and used by your guest
> CPU? It might very well that the PCID optimisations are not used by the guests
> here, and it might be worth either checking on bare metal or with the PCID
> optimisations enabled.

You're totally right, I discovered during my later developments that
indeed PCID is not exposed there. So we take the hit of a full TLB
flush twice per syscall.

Willy

Re: Feedback on 4.9 performance after PTI fixes

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

On Mon, 2018-01-08 at 19:26 +0100, Willy Tarreau wrote:
> You're totally right, I discovered during my later developments that
> indeed PCID is not exposed there. So we take the hit of a full TLB
> flush twice per syscall.

So I really think it might make sense to redo the tests with PCID, because the
 assumptions you're basing your patch series on might actually not hold.

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Feedback on 4.9 performance after PTI fixes

[Willy Tarreau]

On Mon, Jan 08, 2018 at 09:26:10PM +0100, Yves-Alexis Perez wrote:
> On Mon, 2018-01-08 at 19:26 +0100, Willy Tarreau wrote:
> > You're totally right, I discovered during my later developments that
> > indeed PCID is not exposed there. So we take the hit of a full TLB
> > flush twice per syscall.
> 
> So I really think it might make sense to redo the tests with PCID, because the
>  assumptions you're basing your patch series on might actually not hold.

I'll have to do it on the bare-metal server soon anyway.

Cheers,
Willy

Re: Feedback on 4.9 performance after PTI fixes

[Pavlos Parissis]

[-- Attachment #1.1: Type: text/plain, Size: 3402 bytes --]

On 8 January 2018 at 21:39, Willy Tarreau <w@1wt.eu> wrote:
> On Mon, Jan 08, 2018 at 09:26:10PM +0100, Yves-Alexis Perez wrote:
>> On Mon, 2018-01-08 at 19:26 +0100, Willy Tarreau wrote:
>>> You're totally right, I discovered during my later developments that indeed
>>> PCID is not exposed there. So we take the hit of a full TLB flush twice per
>>> syscall.
>> 
>> So I really think it might make sense to redo the tests with PCID, because
>> the assumptions you're basing your patch series on might actually not hold.
> 
> I'll have to do it on the bare-metal server soon anyway.
> 
> Cheers, Willy

Today, I performed a stress test against various kernels using HAPEE 1.7-r1
(Enterprise version of HAProxy) on CentOS 7.4 and here is the executive summary:

RedHat Kernel with patches increases latency by 275%, drops capacity by 34% and
completely saturates CPU resources. Kernels 4.9.75 and 4.14.12 doesn't increase
latency and doesn't drop the capacity compared to 4.9.56. Kernel 4.14.12 doesn't
bring significant improvements over 4.9.75. Moreover, kernel packages came with
some microcode updates which did something on the CPUs during the boot process. I
don't have a lot of details about those microcode updates.

Here some info about the test environment:

I used one server as a test haproxy node and configured a pool with a single
server, where another haproxy server was acting as a server. I used another
machine as a generator to send 80K/sec HTTP requests. I didn't have the
time to set up a distributed stress test and as a result, I could only generate
80K/sec requests - A single HAProxy node can serve much more requests.
The server, which was behind haproxy, served all requests from memory and without
any external dependencies(Disk/Network IO). The request and the response fit in
one TCP packet.

All servers were bare-metal, had Intel E5-2620v3 @ 2.40GHz (with PCID support) and
connected to the same switch with 10GbE interfaces. Due to network restrictions, I
couldn't use the 2nd interface and a result the traffic traveled twice from one
interface. As a generator tool, I used wrk2, and I set 80K/second as maximum
request rate.

I have also compared two production haproxy servers running RedHat kernels with
and without patches and the server with the patches spent 50% more at CPU at
system and softirq levels. Those productions server have been handling customer
traffic since Friday, so I have 100% confidence in the CPU increase.
I will do the same for two production servers with 4.9.56 and 4.9.75 kernels.

Moreover, I have observed similar drop of performance with RedHat kernel on
different workloads, such as MySQL, cassandra and graphite. On the later, I
observed a big performance win with 4.14.12.

Several people mentioned to me that kernel 4.14 should provide much better
performance due to the Longer-lived TLB Entries with PCID feature. So far, I
haven't seen in my haproxy test. This does not mean that it doesn't have better
performance; absence of evidence is not evidence of absence :-)

All above are specific to my workload and setup; other workloads may see less or
more impact on the performance.

Last but not least, I would like to take this opportunity to say a huge THANK YOU
to all developers/engineers involved in making Linux more secure and safe to use.

Cheers,
Pavlos


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Feedback on 4.9 performance after PTI fixes

[Willy Tarreau]

Hi again,

updating the table after Yves-Alexis' comment on PCID. Rerunning the test
with -cpu=Haswell to enable PCID gave me much better numbers :

On Sun, Jan 07, 2018 at 11:18:56AM +0100, Willy Tarreau wrote:
> Hi,
> 
> I managed to take a bit of time to run some more tests on PTI both
> native and hosted in KVM, on stable versions built with
> CONFIG_PAGE_TABLE_ISOLATION=y. Here it's 4.9.75, used both on the
> host and the VM. I could compare pti=on/off both in the host and the
> VM. A single CPU was exposed in the VM.
> 
> It was running on my laptop (core i7 3320M at 2.6 GHz, 3.3 GHz single
> core turbo).
> 
> The test was run on haproxy's ability to forward connections. The
> results are below :
> 
>    Host  |  Guest  | conn/s  | ratio_to_host |  ratio_to_VM | Notes
> ---------+---------+---------+---------------+--------------+----------------
>  pti=off |   -     |  27400  |     100.0%    |      -       | host reference
>  pti=off | pti=off |  24200  |      88.3%    |    100.0%    | VM reference 
>  pti=off | pti=on  |  13300  |      48.5%    |     55.0%    |
>  pti=on  |   -     |  23800  |      86.9%    |      -	      | protected host
>  pti=on  | pti=off |  23100  |      84.3%    |     95.5%    |
>  pti=on  | pti=on  |  13300  |      48.5%    |     55.0%    |

New run :

    Host  |  Guest  | conn/s  | ratio  | Notes
 ---------+---------+---------+--------+----------------
  pti=off | pti=off |  23100  | 100.0% | VM reference without PTI
  pti=off | pti=on  |  19700  |  85.2% | VM with PTI and PCID
  pti=off | pti=on  |  12700  |  55.0% | VM with PTI without PCID

So the performance cut in half was indeed caused by the lack of PCID
here. With it the impact is much less, though still important.

Willy