KVM slow LAMP guest

KVM slow LAMP guest

[Hansa]

Hi,

I have a LAMP server as guest in KVM. Whenever the server is idle for some time it takes about 30 seconds to load a Wordpress site.
If the server is not idle the site shows up in max 5 seconds. I've already turned of power management in the guest by passing

     GRUB_CMDLINE_LINUX_DEFAULT="apm=off"

in /etc/default/grub. This has no effect.
Does KVM do some power management on guests? If so, how do I turn this off for my LAMP guest?

Best, Hansa

Re: KVM slow LAMP guest

[David Matlack]

On Thu, Jun 18, 2015 at 1:25 AM, Hansa <forums@logic-q.nl> wrote:
> Hi,
>
> I have a LAMP server as guest in KVM. Whenever the server is idle for some
> time it takes about 30 seconds to load a Wordpress site.
> If the server is not idle the site shows up in max 5 seconds. I've already
> turned of power management in the guest by passing
>
>     GRUB_CMDLINE_LINUX_DEFAULT="apm=off"
>
> in /etc/default/grub. This has no effect.
> Does KVM do some power management on guests? If so, how do I turn this off
> for my LAMP guest?

KVM doesn't do any power management of guests. But if everything is idle on
the host (including your guest), then host power management could kick in.
Have you tried playing with host pm?

Could you try running your workload with the guest kernel parameter "idle=poll"
and let me know the performance?

Also, if you are running Linux 4.0 or later on the host, could you try running
your workload with the KVM module parameter "halt_poll_ns=500000"?

>
> Best, Hansa
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM slow LAMP guest

[David Matlack]

(adding back the kvm mailing list)

On Tue, Jul 14, 2015 at 1:12 AM, "C. Bröcker"
<c.brocker@intercollect.com> wrote:
> On 14-7-2015 10:04, Hansa wrote:
>>
>> On 13-7-2015 20:57, David Matlack wrote:
>>>
>>> On Thu, Jun 18, 2015 at 10:26 AM, David Matlack <dmatlack@google.com>
>>> wrote:
>>>>
>>>> On Thu, Jun 18, 2015 at 1:25 AM, Hansa <forums@logic-q.nl> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have a LAMP server as guest in KVM. Whenever the server is idle for
>>>>> some
>>>>> time it takes about 30 seconds to load a Wordpress site.
>>>>> If the server is not idle the site shows up in max 5 seconds. I've
>>>>> already
>>>>> turned of power management in the guest by passing
>>>>>
>>>>>      GRUB_CMDLINE_LINUX_DEFAULT="apm=off"
>>>>>
>>>>> in /etc/default/grub. This has no effect.
>>>>> Does KVM do some power management on guests? If so, how do I turn this
>>>>> off
>>>>> for my LAMP guest?
>>>>
>>>> KVM doesn't do any power management of guests. But if everything is idle
>>>> on
>>>> the host (including your guest), then host power management could kick
>>>> in.
>>>> Have you tried playing with host pm?
>>>>
>>>> Could you try running your workload with the guest kernel parameter
>>>> "idle=poll"
>>>> and let me know the performance?
>>>>
>>>> Also, if you are running Linux 4.0 or later on the host, could you try
>>>> running
>>>> your workload with the KVM module parameter "halt_poll_ns=500000"?
>>>
>>> Hansa-- Did you ever get a chance to run your workload with these
>>> changes?
>>
>> Hi David,
>>
>> Sorry I didn't respond to your mail earlier and nice of you to check back.
>> To be honest no I haven't. To much work overload on my side. I've just added
>> the idle=poll to the guests' grub and rebooted the server. I'll let it run a
>> few day's to check if it helps. Btw: I'm running 2.6 linux kernel here so
>> I'm not sure if I can add the "halt_poll_ns" param to KVM.
>>
>> Best, Hansa
>
> Wow! That makes real difference!
> The site was never that fast! Although CPU load is constantly on 100% any
> chance I can tweak this param? And is there a similar param for Windows
> guests?

Since idle=poll gave a noticeable improvement, your workload is a good
candidate for halt_poll_ns. This is a host KVM module parameter. It will
give you a similar speedup, it will work independent of your guest os,
and won't peg your cpu usage at 100% during periods of no activity. If
you don't have halt_poll_ns available (your host kernel older than 4.0),
is upgrading to a newer kernel or backporting the patch [1] an option?

I'm not sure of any more guest params you can tweak (linux or windows), but
I'll let you know if I come up with anything.

[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926

Re: KVM slow LAMP guest

["C. Bröcker"]

On 14-7-2015 19:16, David Matlack wrote:
> (adding back the kvm mailing list)
>
> On Tue, Jul 14, 2015 at 1:12 AM, "C. Bröcker"
> <c.brocker@intercollect.com> wrote:
>> On 14-7-2015 10:04, Hansa wrote:
>>> On 13-7-2015 20:57, David Matlack wrote:
>>>> On Thu, Jun 18, 2015 at 10:26 AM, David Matlack <dmatlack@google.com>
>>>> wrote:
>>>>> On Thu, Jun 18, 2015 at 1:25 AM, Hansa <forums@logic-q.nl> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have a LAMP server as guest in KVM. Whenever the server is idle for
>>>>>> some
>>>>>> time it takes about 30 seconds to load a Wordpress site.
>>>>>> If the server is not idle the site shows up in max 5 seconds. I've
>>>>>> already
>>>>>> turned of power management in the guest by passing
>>>>>>
>>>>>>       GRUB_CMDLINE_LINUX_DEFAULT="apm=off"
>>>>>>
>>>>>> in /etc/default/grub. This has no effect.
>>>>>> Does KVM do some power management on guests? If so, how do I turn this
>>>>>> off
>>>>>> for my LAMP guest?
>>>>> KVM doesn't do any power management of guests. But if everything is idle
>>>>> on
>>>>> the host (including your guest), then host power management could kick
>>>>> in.
>>>>> Have you tried playing with host pm?
>>>>>
>>>>> Could you try running your workload with the guest kernel parameter
>>>>> "idle=poll"
>>>>> and let me know the performance?
>>>>>
>>>>> Also, if you are running Linux 4.0 or later on the host, could you try
>>>>> running
>>>>> your workload with the KVM module parameter "halt_poll_ns=500000"?
>>>> Hansa-- Did you ever get a chance to run your workload with these
>>>> changes?
>>> Hi David,
>>>
>>> Sorry I didn't respond to your mail earlier and nice of you to check back.
>>> To be honest no I haven't. To much work overload on my side. I've just added
>>> the idle=poll to the guests' grub and rebooted the server. I'll let it run a
>>> few day's to check if it helps. Btw: I'm running 2.6 linux kernel here so
>>> I'm not sure if I can add the "halt_poll_ns" param to KVM.
>>>
>>> Best, Hansa
>> Wow! That makes real difference!
>> The site was never that fast! Although CPU load is constantly on 100% any
>> chance I can tweak this param? And is there a similar param for Windows
>> guests?
> Since idle=poll gave a noticeable improvement, your workload is a good
> candidate for halt_poll_ns. This is a host KVM module parameter. It will
> give you a similar speedup, it will work independent of your guest os,
> and won't peg your cpu usage at 100% during periods of no activity. If
> you don't have halt_poll_ns available (your host kernel older than 4.0),
> is upgrading to a newer kernel or backporting the patch [1] an option?
>
> I'm not sure of any more guest params you can tweak (linux or windows), but
> I'll let you know if I come up with anything.
>
> [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926
Sorry David.
That is not an option. It's a production server on 2.6 kernel. 4.0 kernel is not in Base repos and I don't know which impact installing the newer kernel version will have on this server. Any other options?

Re: KVM slow LAMP guest

[Hansa]

On 14-7-2015 23:16, Paolo Bonzini wrote:
> On 14/07/2015 21:54, "Hansa" wrote:
>>> [1]
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926
>>>
>> Sorry David.
>> That is not an option. It's a production server on 2.6 kernel. 4.0
>> kernel is not in Base repos and I don't know which impact installing the
>> newer kernel version will have on this server. Any other options?
> You could backport the patch to the version you have.
>
> What OS is this?  Is it RHEL/CentOS?  If so, halt_poll_ns will be in 6.7
> which will be out in a few days/weeks.
>
> Paolo
CentOS release 6.6 (Final)
-Hansa

Re: KVM slow LAMP guest

["C. Bröcker"]

On 14-7-2015 23:16, Paolo Bonzini wrote:
> On 14/07/2015 21:54, "Hansa" wrote:
>>> [1]
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926
>>>
>> Sorry David.
>> That is not an option. It's a production server on 2.6 kernel. 4.0
>> kernel is not in Base repos and I don't know which impact installing the
>> newer kernel version will have on this server. Any other options?
> You could backport the patch to the version you have.
>
> What OS is this?  Is it RHEL/CentOS?  If so, halt_poll_ns will be in 6.7
> which will be out in a few days/weeks.
>
> Paolo
OK. As said CentOS 6.6.
But where do I put this parameter?
Regards, Hansa

Re: KVM slow LAMP guest

[Paolo Bonzini]

On 15/07/2015 22:02, "C. Bröcker" wrote:
>>
>> What OS is this?  Is it RHEL/CentOS?  If so, halt_poll_ns will be in 6.7
>> which will be out in a few days/weeks.
>>
>> Paolo
> OK. As said CentOS 6.6.
> But where do I put this parameter?

You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
you have the parameter, you have the
/sys/module/kvm/parameters/halt_poll_ns file.

Paolo

> Regards, Hansa

Re: KVM slow LAMP guest

[Hansa]

On 16-7-2015 13:27, Paolo Bonzini wrote:
> On 15/07/2015 22:02, "Hansa" wrote:
>>> What OS is this?  Is it RHEL/CentOS?  If so, halt_poll_ns will be in 6.7
>>> which will be out in a few days/weeks.
>>>
>>> Paolo
>> OK. As said CentOS 6.6.
>> But where do I put this parameter?
> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
> you have the parameter, you have the
> /sys/module/kvm/parameters/halt_poll_ns file.
Thanks Paolo!
I'll await the 6.7 release.
Best, Hansa

Re: KVM slow LAMP guest

[Hansa]

On 16-7-2015 13:27, Paolo Bonzini wrote:
> On 15/07/2015 22:02, "C. Bröcker" wrote:
>>> What OS is this?  Is it RHEL/CentOS?  If so, halt_poll_ns will be in 6.7
>>> which will be out in a few days/weeks.
>>>
>>> Paolo
>> OK. As said CentOS 6.6.
>> But where do I put this parameter?
> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
> you have the parameter, you have the
> /sys/module/kvm/parameters/halt_poll_ns file.
Hi,

I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available.
Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem!

My guess is that a lot people out there have the same problem but do not know the solution. Why not put it in the kernel by default?

Thanks a lot for all the help!
- Hansa

Re: KVM slow LAMP guest

[Wanpeng Li]

On 8/24/15 3:18 AM, Hansa wrote:
> On 16-7-2015 13:27, Paolo Bonzini wrote:
>> On 15/07/2015 22:02, "C. Bröcker" wrote:
>>>> What OS is this?  Is it RHEL/CentOS? If so, halt_poll_ns will be in 
>>>> 6.7
>>>> which will be out in a few days/weeks.
>>>>
>>>> Paolo
>>> OK. As said CentOS 6.6.
>>> But where do I put this parameter?
>> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
>> you have the parameter, you have the
>> /sys/module/kvm/parameters/halt_poll_ns file.
> Hi,
>
> I upgraded to the CentOS 6.7 release which came out last month and as 
> promised the halt_poll_ns parameter was available.
> Last week I tested the availability status every 5 minutes on my 
> Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm 
> pleased to announce that it solves the problem!

How much seconds to load your Wordpress site this time?

Regards,
Wanpeng Li

>
> My guess is that a lot people out there have the same problem but do 
> not know the solution. Why not put it in the kernel by default?
>
> Thanks a lot for all the help!
> - Hansa
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM slow LAMP guest

[Hansa]

On 24-8-2015 1:26, Wanpeng Li wrote:
> On 8/24/15 3:18 AM, Hansa wrote:
>> On 16-7-2015 13:27, Paolo Bonzini wrote:
>>> On 15/07/2015 22:02, "C. Bröcker" wrote:
>>>>> What OS is this?  Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7
>>>>> which will be out in a few days/weeks.
>>>>>
>>>>> Paolo
>>>> OK. As said CentOS 6.6.
>>>> But where do I put this parameter?
>>> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
>>> you have the parameter, you have the
>>> /sys/module/kvm/parameters/halt_poll_ns file.
>> Hi,
>>
>> I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available.
>> Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem!
>
> How much seconds to load your Wordpress site this time?
>
> Regards,
> Wanpeng Li
I've only tested availability using the webmin's "Remote HTTP Service" monitoring tool. Currently it only times out when doing backups. I will do some tests with curl and let you know the result in a day or so.

Re: KVM slow LAMP guest

[Hansa]

On 24-8-2015 1:26, Wanpeng Li wrote:
> On 8/24/15 3:18 AM, Hansa wrote:
>> On 16-7-2015 13:27, Paolo Bonzini wrote:
>>> On 15/07/2015 22:02, "C. Bröcker" wrote:
>>>>> What OS is this?  Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7
>>>>> which will be out in a few days/weeks.
>>>>>
>>>>> Paolo
>>>> OK. As said CentOS 6.6.
>>>> But where do I put this parameter?
>>> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
>>> you have the parameter, you have the
>>> /sys/module/kvm/parameters/halt_poll_ns file.
>> Hi,
>>
>> I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available.
>> Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem!
>
> How much seconds to load your Wordpress site this time?
>
> Regards,
> Wanpeng Li
The average is around 0.4 seconds to load my "heaviest" site on my slowest machine.

On the VM server I issued the command below every eleven minutes:

date >>  curltest-file; _
top -b -n 1 | sed -n '7,12p' >> curltest-file; _
curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file

This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file.

I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar)  issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total.

Cheers and good luck tweaking your sites!
Best, Hansa

Re: KVM slow LAMP guest

[Wanpeng Li]

On 8/25/15 11:42 PM, Hansa wrote:
> On 24-8-2015 1:26, Wanpeng Li wrote:
>> On 8/24/15 3:18 AM, Hansa wrote:
>>> On 16-7-2015 13:27, Paolo Bonzini wrote:
>>>> On 15/07/2015 22:02, "C. Bröcker" wrote:
>>>>>> What OS is this?  Is it RHEL/CentOS? If so, halt_poll_ns will be 
>>>>>> in 6.7
>>>>>> which will be out in a few days/weeks.
>>>>>>
>>>>>> Paolo
>>>>> OK. As said CentOS 6.6.
>>>>> But where do I put this parameter?
>>>> You can add "kvm.halt_poll_ns=500000" to the kernel command line.  If
>>>> you have the parameter, you have the
>>>> /sys/module/kvm/parameters/halt_poll_ns file.
>>> Hi,
>>>
>>> I upgraded to the CentOS 6.7 release which came out last month and 
>>> as promised the halt_poll_ns parameter was available.
>>> Last week I tested the availability status every 5 minutes on my 
>>> Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm 
>>> pleased to announce that it solves the problem!
>>
>> How much seconds to load your Wordpress site this time?
>>
>> Regards,
>> Wanpeng Li
> The average is around 0.4 seconds to load my "heaviest" site on my 
> slowest machine.

Nice!

>
> On the VM server I issued the command below every eleven minutes:
>
> date >>  curltest-file; _
> top -b -n 1 | sed -n '7,12p' >> curltest-file; _
> curl -o /dev/null -s -w"time_total: %{time_total}\\n" 
> https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print 
> strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file
>
> This gives me the total time for displaying my site on a local 
> machine. It also includes a 'top' command to display which processes 
> are running at each sample. All is saved in a file called curltest-file.
>
> I found 7 occurrences in my curltest-file of a time_total larger than 
> 20 seconds. Top however didn't show any significant CPU or IO activity 
> at those sampled times. Further investigations shows me that they are 
> related to a known (gravatar)  issue in the Wordpress Jetpack plugin. 
> I didn't include these samples in the average total.

If you just use halt_poll_ns or both halt_poll_ns and idle=poll in guest?

Regards,
Wanpeng Li

>
> Cheers and good luck tweaking your sites!
> Best, Hansa
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM slow LAMP guest

[Hansa]

On 26-8-2015 0:33, Wanpeng Li wrote:
>> On the VM server I issued the command below every eleven minutes:
>>
>> date >>  curltest-file; _
>> top -b -n 1 | sed -n '7,12p' >> curltest-file; _
>> curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file
>>
>> This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file.
>>
>> I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar)  issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total.
>
> If you just use halt_poll_ns or both halt_poll_ns and idle=poll in guest?
>
I just use kvm.halt_poll_ns=500000
Should I try some different tests?

Re: KVM slow LAMP guest

[Wanpeng Li]

On 8/26/15 6:41 AM, Hansa wrote:
> On 26-8-2015 0:33, Wanpeng Li wrote:
>>> On the VM server I issued the command below every eleven minutes:
>>>
>>> date >>  curltest-file; _
>>> top -b -n 1 | sed -n '7,12p' >> curltest-file; _
>>> curl -o /dev/null -s -w"time_total: %{time_total}\\n"
>>> https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print
>>> strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file
>>>
>>> This gives me the total time for displaying my site on a local
>>> machine. It also includes a 'top' command to display which processes
>>> are running at each sample. All is saved in a file called
>>> curltest-file.
>>>
>>> I found 7 occurrences in my curltest-file of a time_total larger
>>> than 20 seconds. Top however didn't show any significant CPU or IO
>>> activity at those sampled times. Further investigations shows me
>>> that they are related to a known (gravatar)  issue in the Wordpress
>>> Jetpack plugin. I didn't include these samples in the average total.
>>
>> If you just use halt_poll_ns or both halt_poll_ns and idle=poll in
>> guest?
>>
> I just use kvm.halt_poll_ns=500000
> Should I try some different tests?

Looks good to me currently. Per vCPU will consume almost half pCPU's 
capacity in host when add idle=poll in my testing which is not suitable 
for some cloud computing scenarios since vCPUs have high overcommit 
ratio on host.

Regards,
Wanpeng Li