From: Joao Martins <joao.m.martins@oracle.com>
To: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: "Marcelo Tosatti" <mtosatti@redhat.com>,
"Rafael J. Wysocki" <rjw@rjwysocki.net>,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Radim Krčmář" <rkrcmar@redhat.com>,
"Sean Christopherson" <sean.j.christopherson@intel.com>,
"Vitaly Kuznetsov" <vkuznets@redhat.com>,
"Wanpeng Li" <wanpengli@tencent.com>,
"Jim Mattson" <jmattson@google.com>,
"Joerg Roedel" <joro@8bytes.org>,
linux-pm@vger.kernel.org,
"Boris Ostrovsky" <boris.ostrovsky@oracle.com>
Subject: Re: Default governor regardless of cpuidle driver
Date: Thu, 29 Aug 2019 22:12:56 +0100 [thread overview]
Message-ID: <f8c63af5-2509-310d-7ba0-7687b20e3b44@oracle.com> (raw)
In-Reply-To: <2e2a35c8-7f03-d7c8-4701-3bc9d91c1255@linaro.org>
On 8/29/19 9:22 PM, Daniel Lezcano wrote:
> On 29/08/2019 21:11, Joao Martins wrote:
>> On 8/29/19 7:28 PM, Daniel Lezcano wrote:
>>> On 29/08/2019 20:07, Joao Martins wrote:
>>>> On 8/29/19 6:42 PM, Daniel Lezcano wrote:
>>>>> On 29/08/2019 19:16, Joao Martins wrote:
>>>>>> On 8/29/19 4:10 PM, Joao Martins wrote:
>>>>>>> When cpus != maxcpus cpuidle-haltpoll will fail to register all vcpus
>>>>>>> past the online ones and thus fail to register the idle driver.
>>>>>>> This is because cpuidle_add_sysfs() will return with -ENODEV as a
>>>>>>> consequence from get_cpu_device() return no device for a non-existing
>>>>>>> CPU.
>>>>>>>
>>>>>>> Instead switch to cpuidle_register_driver() and manually register each
>>>>>>> of the present cpus through cpuhp_setup_state() callback and future
>>>>>>> ones that get onlined. This mimmics similar logic that intel_idle does.
>>>>>>>
>>>>>>> Fixes: fa86ee90eb11 ("add cpuidle-haltpoll driver")
>>>>>>> Signed-off-by: Joao Martins <joao.m.martins@oracle.com>
>>>>>>> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
>>>>>>> ---
>>>>>>
>>>>>> While testing the above, I found out another issue on the haltpoll series.
>>>>>> But I am not sure what is best suited to cpuidle framework, hence requesting
>>>>>> some advise if below is a reasonable solution or something else is preferred.
>>>>>>
>>>>>> Essentially after haltpoll governor got introduced and regardless of the cpuidle
>>>>>> driver the default governor is gonna be haltpoll for a guest (given haltpoll
>>>>>> governor doesn't get registered for baremetal). Right now, for a KVM guest, the
>>>>>> idle governors have these ratings:
>>>>>>
>>>>>> * ladder -> 10
>>>>>> * teo -> 19
>>>>>> * menu -> 20
>>>>>> * haltpoll -> 21
>>>>>> * ladder + nohz=off -> 25
>>>>>>
>>>>>> When a guest is booted with MWAIT and intel_idle is probed and sucessfully
>>>>>> registered, we will end up with a haltpoll governor being used as opposed to
>>>>>> 'menu' (which used to be the default case). This would prevent IIUC that other
>>>>>> C-states get used other than poll_state (state 0) and state 1.
>>>>>>
>>>>>> Given that haltpoll governor is largely only useful with a cpuidle-haltpoll
>>>>>> it doesn't look reasonable to be the default? What about using haltpoll governor
>>>>>> as default when haltpoll idle driver registers or modload.
>>>>>
>>>>> Are the guest and host kernel the same? IOW compiled with the same
>>>>> kernel config?
>>>>>
>>>> You just need to toggle this (regardless off CONFIG_HALTPOLL_CPUIDLE):
>>>>
>>>> CONFIG_CPU_IDLE_GOV_HALTPOLL=y
>>>>
>>>> And *if you are a KVM guest* it will be the default (unless using nohz=off in
>>>> which case ladder gets the highest rating -- see the listing right above).
>>>>
>>>> Host will just behave differently because the haltpoll governor is checking if
>>>> it is running as kvm guest, and only registering in that case.
>>>
>>> I understood the problem. Actually my question was about if the kernels
>>> are compiled for host and guest, and can be run indifferently.
>>
>> /nods Correct.
>>
>>> In this
>>> case a runtime detection must be done as you propose, otherwise that can
>>> be done at config time. I pretty sure it is the former but before
>>> thinking about the runtime side, I wanted to double check.
>>>
>> Hmm, but even with separate kernels/configs for guest and host I think we would
>> still have the same issue.
>>
>> What I was trying to convey is that even when running with a config solely for
>> KVM guests (that is different than baremetal) you can have today various ways of
>> idling. An Intel x86 kvm guest can have no idle driver (but arch-specific),
>> intel_idle (like baremetal config) and haltpoll. There are usecases for these
>> three, and makes sense to consolidate all.
>>
>> Say you wanted to have a kvm specific config, you would still see the same
>> problem if you happen to compile intel_idle together with haltpoll
>> driver+governor.
>
> Can a guest work with an intel_idle driver?
>
Yes.
If you use Qemu you would add '-overcommit cpu-pm=on' to try it out. ofc,
assuming you're on a relatively recent Qemu (v3.0+) and a fairly recent kernel
version as host (v4.17+).
>> Creating two separate configs here, with and without haltpoll
>> for VMs doesn't sound effective for distros.
>
> Agree
>
>> Perhaps decreasing the rating of
>> haltpoll governor, but while a short term fix it wouldn't give much sensible
>> defaults without the one-off runtime switch.
>
next prev parent reply other threads:[~2019-08-29 21:17 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-29 15:10 [PATCH v2] cpuidle-haltpoll: vcpu hotplug support Joao Martins
2019-08-29 15:27 ` Marcelo Tosatti
2019-09-02 10:48 ` Joao Martins
2019-08-29 17:16 ` Is: Default governor regardless of cpuidle driver Was: " Joao Martins
2019-08-29 17:23 ` Marcelo Tosatti
2019-09-02 21:55 ` Rafael J. Wysocki
2019-09-03 10:13 ` Joao Martins
2019-08-29 17:42 ` Daniel Lezcano
2019-08-29 18:07 ` Joao Martins
2019-08-29 18:28 ` Daniel Lezcano
2019-08-29 19:11 ` Default governor regardless of cpuidle driver Joao Martins
2019-08-29 20:22 ` Daniel Lezcano
2019-08-29 21:12 ` Joao Martins [this message]
2019-08-29 21:51 ` Daniel Lezcano
2019-08-30 11:07 ` Joao Martins
2019-09-02 21:58 ` Rafael J. Wysocki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f8c63af5-2509-310d-7ba0-7687b20e3b44@oracle.com \
--to=joao.m.martins@oracle.com \
--cc=boris.ostrovsky@oracle.com \
--cc=daniel.lezcano@linaro.org \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=mtosatti@redhat.com \
--cc=pbonzini@redhat.com \
--cc=rjw@rjwysocki.net \
--cc=rkrcmar@redhat.com \
--cc=sean.j.christopherson@intel.com \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).