QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH 0/2] i386: Fix interrupt based Async PF enablement
@ 2021-04-01 15:19 Vitaly Kuznetsov
  2021-04-01 15:19 ` [PATCH 1/2] i386: Add 'kvm-asyncpf-int' to kvm_default_props array Vitaly Kuznetsov
                   ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-01 15:19 UTC (permalink / raw)
  To: qemu-devel, Paolo Bonzini
  Cc: Michael S. Tsirkin, Richard Henderson, Eduardo Habkost,
	Dr . David Alan Gilbert

I noticed two issues with 'kvm-asyncpf-int' enablement:
1) We forgot to add to to kvm_default_props[] so it doesn't get enabled
 automatically (unless '-cpu host' is used or the feature is enabled
 manually on the command line)
2) We forgot to disable it for older machine types to preserve migration.
 This went unnoticed because of 1) I believe.

Vitaly Kuznetsov (2):
  i386: Add 'kvm-asyncpf-int' to kvm_default_props array
  i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1

 hw/i386/pc.c      | 1 +
 target/i386/cpu.c | 1 +
 2 files changed, 2 insertions(+)

-- 
2.30.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 1/2] i386: Add 'kvm-asyncpf-int' to kvm_default_props array
  2021-04-01 15:19 [PATCH 0/2] i386: Fix interrupt based Async PF enablement Vitaly Kuznetsov
@ 2021-04-01 15:19 ` Vitaly Kuznetsov
  2021-04-01 15:19 ` [PATCH 2/2] i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1 Vitaly Kuznetsov
  2021-04-01 15:57 ` [PATCH 0/2] i386: Fix interrupt based Async PF enablement Paolo Bonzini
  2 siblings, 0 replies; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-01 15:19 UTC (permalink / raw)
  To: qemu-devel, Paolo Bonzini
  Cc: Michael S. Tsirkin, Richard Henderson, Eduardo Habkost,
	Dr . David Alan Gilbert

Just like all other KVM PV features, 'kvm-asyncpf-int' needs to be
added to all CPU models when KVM is enabled or the feature will always
remain 'off' unless specified explicitly on the command line.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 6b3e9467f177..c7f8a8a8fec0 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -4244,6 +4244,7 @@ static PropValue kvm_default_props[] = {
     { "kvmclock", "on" },
     { "kvm-nopiodelay", "on" },
     { "kvm-asyncpf", "on" },
+    { "kvm-asyncpf-int", "on" },
     { "kvm-steal-time", "on" },
     { "kvm-pv-eoi", "on" },
     { "kvmclock-stable-bit", "on" },
-- 
2.30.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 2/2] i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1
  2021-04-01 15:19 [PATCH 0/2] i386: Fix interrupt based Async PF enablement Vitaly Kuznetsov
  2021-04-01 15:19 ` [PATCH 1/2] i386: Add 'kvm-asyncpf-int' to kvm_default_props array Vitaly Kuznetsov
@ 2021-04-01 15:19 ` Vitaly Kuznetsov
  2021-04-01 15:57 ` [PATCH 0/2] i386: Fix interrupt based Async PF enablement Paolo Bonzini
  2 siblings, 0 replies; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-01 15:19 UTC (permalink / raw)
  To: qemu-devel, Paolo Bonzini
  Cc: Michael S. Tsirkin, Richard Henderson, Eduardo Habkost,
	Dr . David Alan Gilbert

'kvm-asyncpf-int' was implemented in QEMU-5.2 so older machine types
should have it disabled to make migration to an older QEMU which does not
support this feature possible.

The issue went unnoticed probably because we also forgot to add
'kvm-asyncpf-int' to 'kvm_default_props[]' so it was rarely enabled.

Fixes: db5daafab2 ("target/i386: support KVM_FEATURE_ASYNC_PF_INT")
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 hw/i386/pc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 8a84b25a031e..04d5f76bf133 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -103,6 +103,7 @@ const size_t pc_compat_5_2_len = G_N_ELEMENTS(pc_compat_5_2);
 
 GlobalProperty pc_compat_5_1[] = {
     { "ICH9-LPC", "x-smi-cpu-hotplug", "off" },
+    { TYPE_X86_CPU, "kvm-asyncpf-int", "off" },
     { TYPE_X86_CPU, "kvm-msi-ext-dest-id", "off" },
 };
 const size_t pc_compat_5_1_len = G_N_ELEMENTS(pc_compat_5_1);
-- 
2.30.2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-01 15:19 [PATCH 0/2] i386: Fix interrupt based Async PF enablement Vitaly Kuznetsov
  2021-04-01 15:19 ` [PATCH 1/2] i386: Add 'kvm-asyncpf-int' to kvm_default_props array Vitaly Kuznetsov
  2021-04-01 15:19 ` [PATCH 2/2] i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1 Vitaly Kuznetsov
@ 2021-04-01 15:57 ` Paolo Bonzini
  2021-04-06 11:42   ` Vitaly Kuznetsov
  2 siblings, 1 reply; 15+ messages in thread
From: Paolo Bonzini @ 2021-04-01 15:57 UTC (permalink / raw)
  To: Vitaly Kuznetsov, qemu-devel
  Cc: Michael S. Tsirkin, Richard Henderson, Eduardo Habkost,
	Dr . David Alan Gilbert

On 01/04/21 17:19, Vitaly Kuznetsov wrote:
> I noticed two issues with 'kvm-asyncpf-int' enablement:
> 1) We forgot to add to to kvm_default_props[] so it doesn't get enabled
>   automatically (unless '-cpu host' is used or the feature is enabled
>   manually on the command line)
> 2) We forgot to disable it for older machine types to preserve migration.
>   This went unnoticed because of 1) I believe.
> 
> Vitaly Kuznetsov (2):
>    i386: Add 'kvm-asyncpf-int' to kvm_default_props array
>    i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1
> 
>   hw/i386/pc.c      | 1 +
>   target/i386/cpu.c | 1 +
>   2 files changed, 2 insertions(+)
> 

Wasn't this intentional to avoid requiring a new kernel version?

Paolo



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-01 15:57 ` [PATCH 0/2] i386: Fix interrupt based Async PF enablement Paolo Bonzini
@ 2021-04-06 11:42   ` Vitaly Kuznetsov
  2021-04-08 12:46     ` Paolo Bonzini
  0 siblings, 1 reply; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-06 11:42 UTC (permalink / raw)
  To: Paolo Bonzini, qemu-devel
  Cc: Michael S. Tsirkin, Richard Henderson, Eduardo Habkost,
	Dr . David Alan Gilbert

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 01/04/21 17:19, Vitaly Kuznetsov wrote:
>> I noticed two issues with 'kvm-asyncpf-int' enablement:
>> 1) We forgot to add to to kvm_default_props[] so it doesn't get enabled
>>   automatically (unless '-cpu host' is used or the feature is enabled
>>   manually on the command line)
>> 2) We forgot to disable it for older machine types to preserve migration.
>>   This went unnoticed because of 1) I believe.
>> 
>> Vitaly Kuznetsov (2):
>>    i386: Add 'kvm-asyncpf-int' to kvm_default_props array
>>    i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1
>> 
>>   hw/i386/pc.c      | 1 +
>>   target/i386/cpu.c | 1 +
>>   2 files changed, 2 insertions(+)
>> 
>
> Wasn't this intentional to avoid requiring a new kernel version?

I think I forgot the initial plan :-( The problem is that after we
disabled the original APF (#PF based) almost nobody is using the feature
as it needs to be enabled explicitly on the command line.

Several considerations regarding the default: if your kernel doesn't
support the feature you get as much as a warning:

qemu-system-x86_64: warning: host doesn't support requested feature:
CPUID.40000001H:EAX.kvm-asyncpf-int [bit 14]

older machine types are still available (I disable it for <= 5.1 but we
can consider disabling it for 5.2 too). The feature is upstream since
Linux 5.8, I know that QEMU supports much older kernels but this doesn't
probably mean that we can't enable new KVM PV features unless all
supported kernels have it, we'd have to wait many years otherwise. 

-- 
Vitaly



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-06 11:42   ` Vitaly Kuznetsov
@ 2021-04-08 12:46     ` Paolo Bonzini
  2021-04-15 19:14       ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 15+ messages in thread
From: Paolo Bonzini @ 2021-04-08 12:46 UTC (permalink / raw)
  To: Vitaly Kuznetsov, qemu-devel
  Cc: Richard Henderson, Eduardo Habkost, Dr . David Alan Gilbert,
	Michael S. Tsirkin

On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> older machine types are still available (I disable it for <= 5.1 but we
> can consider disabling it for 5.2 too). The feature is upstream since
> Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> probably mean that we can't enable new KVM PV features unless all
> supported kernels have it, we'd have to wait many years otherwise.

Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7, 
though that will go away in 6.1.

We should take the occasion of dropping RHEL7 to be clearer about which 
kernels are supported.

Paolo



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-08 12:46     ` Paolo Bonzini
@ 2021-04-15 19:14       ` Dr. David Alan Gilbert
  2021-04-20 17:35         ` Eduardo Habkost
  0 siblings, 1 reply; 15+ messages in thread
From: Dr. David Alan Gilbert @ 2021-04-15 19:14 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: Vitaly Kuznetsov, Richard Henderson, qemu-devel, Eduardo Habkost,
	Michael S. Tsirkin

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> > older machine types are still available (I disable it for <= 5.1 but we
> > can consider disabling it for 5.2 too). The feature is upstream since
> > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> > probably mean that we can't enable new KVM PV features unless all
> > supported kernels have it, we'd have to wait many years otherwise.
> 
> Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> though that will go away in 6.1.
> 
> We should take the occasion of dropping RHEL7 to be clearer about which
> kernels are supported.

It would be nice to be able to define sets of KVM functonality that we
can either start given machine types with, or provide a separate switch
to limit kvm functionality back to some defined point.  We do trip over
the same things pretty regularly when accidentally turning on new
features.

Dave

> Paolo
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-15 19:14       ` Dr. David Alan Gilbert
@ 2021-04-20 17:35         ` Eduardo Habkost
  2021-04-21  8:38           ` Vitaly Kuznetsov
  0 siblings, 1 reply; 15+ messages in thread
From: Eduardo Habkost @ 2021-04-20 17:35 UTC (permalink / raw)
  To: Dr. David Alan Gilbert
  Cc: Paolo Bonzini, Vitaly Kuznetsov, Richard Henderson, qemu-devel,
	Michael S. Tsirkin

On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> > > older machine types are still available (I disable it for <= 5.1 but we
> > > can consider disabling it for 5.2 too). The feature is upstream since
> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> > > probably mean that we can't enable new KVM PV features unless all
> > > supported kernels have it, we'd have to wait many years otherwise.
> > 
> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> > though that will go away in 6.1.
> > 
> > We should take the occasion of dropping RHEL7 to be clearer about which
> > kernels are supported.
> 
> It would be nice to be able to define sets of KVM functonality that we
> can either start given machine types with, or provide a separate switch
> to limit kvm functionality back to some defined point.  We do trip over
> the same things pretty regularly when accidentally turning on new
> features.

The same idea can apply to the hyperv=on stuff Vitaly is working
on.  Maybe we should consider making a generic version of the
s390x FeatGroup code, use it to define convenient sets of KVM and
hyperv features.

-- 
Eduardo



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-20 17:35         ` Eduardo Habkost
@ 2021-04-21  8:38           ` Vitaly Kuznetsov
  2021-04-21  8:50             ` Daniel P. Berrangé
  0 siblings, 1 reply; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-21  8:38 UTC (permalink / raw)
  To: Eduardo Habkost, Dr. David Alan Gilbert
  Cc: Paolo Bonzini, Richard Henderson, qemu-devel, Michael S. Tsirkin

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
>> > > older machine types are still available (I disable it for <= 5.1 but we
>> > > can consider disabling it for 5.2 too). The feature is upstream since
>> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
>> > > probably mean that we can't enable new KVM PV features unless all
>> > > supported kernels have it, we'd have to wait many years otherwise.
>> > 
>> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
>> > though that will go away in 6.1.
>> > 
>> > We should take the occasion of dropping RHEL7 to be clearer about which
>> > kernels are supported.
>> 
>> It would be nice to be able to define sets of KVM functonality that we
>> can either start given machine types with, or provide a separate switch
>> to limit kvm functionality back to some defined point.  We do trip over
>> the same things pretty regularly when accidentally turning on new
>> features.
>
> The same idea can apply to the hyperv=on stuff Vitaly is working
> on.  Maybe we should consider making a generic version of the
> s390x FeatGroup code, use it to define convenient sets of KVM and
> hyperv features.

True, the more I look at PV features enablement, the more I think that
we're missing something important in the logic. All machine types we
have are generally suposed to work with the oldest supported kernel so
we should wait many years before enabling some of the new PV features
(KVM or Hyper-V) by default.

This also links to our parallel discussion regarding migration
policies. Currently, we can't enable PV features by default based on
their availability on the host because of migration, the set may differ
on the destination host. What if we introduce (and maybe even switch to
it by default) something like

 -migratable opportunistic (stupid name, I know)

which would allow to enable all features supported by the source host
and then somehow checking that the destination host has them all. This
would effectively mean that it is possible to migrate a VM to a
same-or-newer software (both kernel an QEMU) but not the other way
around. This may be a reasonable choice.

-- 
Vitaly



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  8:38           ` Vitaly Kuznetsov
@ 2021-04-21  8:50             ` Daniel P. Berrangé
  2021-04-21  9:23               ` Dr. David Alan Gilbert
  2021-04-21  9:29               ` Vitaly Kuznetsov
  0 siblings, 2 replies; 15+ messages in thread
From: Daniel P. Berrangé @ 2021-04-21  8:50 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Dr. David Alan Gilbert, Paolo Bonzini

On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
> >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> >> > > older machine types are still available (I disable it for <= 5.1 but we
> >> > > can consider disabling it for 5.2 too). The feature is upstream since
> >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> >> > > probably mean that we can't enable new KVM PV features unless all
> >> > > supported kernels have it, we'd have to wait many years otherwise.
> >> > 
> >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> >> > though that will go away in 6.1.
> >> > 
> >> > We should take the occasion of dropping RHEL7 to be clearer about which
> >> > kernels are supported.
> >> 
> >> It would be nice to be able to define sets of KVM functonality that we
> >> can either start given machine types with, or provide a separate switch
> >> to limit kvm functionality back to some defined point.  We do trip over
> >> the same things pretty regularly when accidentally turning on new
> >> features.
> >
> > The same idea can apply to the hyperv=on stuff Vitaly is working
> > on.  Maybe we should consider making a generic version of the
> > s390x FeatGroup code, use it to define convenient sets of KVM and
> > hyperv features.
> 
> True, the more I look at PV features enablement, the more I think that
> we're missing something important in the logic. All machine types we
> have are generally suposed to work with the oldest supported kernel so
> we should wait many years before enabling some of the new PV features
> (KVM or Hyper-V) by default.
> 
> This also links to our parallel discussion regarding migration
> policies. Currently, we can't enable PV features by default based on
> their availability on the host because of migration, the set may differ
> on the destination host. What if we introduce (and maybe even switch to
> it by default) something like
> 
>  -migratable opportunistic (stupid name, I know)
> 
> which would allow to enable all features supported by the source host
> and then somehow checking that the destination host has them all. This
> would effectively mean that it is possible to migrate a VM to a
> same-or-newer software (both kernel an QEMU) but not the other way
> around. This may be a reasonable choice.

I don't think this is usable in pratice. Any large cloud or data center
mgmt app using QEMU relies on migration, so can't opportunistically
use arbitrary new features. They can only use features in the oldest
kernel their deployment cares about. This can be newer than the oldest
that QEMU supports, but still older than the newest that exists.

ie we have situation where:

 - QEMU upstream minimum host is version 7
 - Latest possible host is version 45
 - A particular deployment has a mixture of hosts at version 24 and 37

"-migratable opportunistic"  would let QEMU use features from version 37
despite the deployment needing compatibility with host version 24 still.


It is almost as if we need to have a way to explicitly express a minimum
required host version that VM requires compatibility with, so deployments
can set their own baseline that is newer than QEMU minimum.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  8:50             ` Daniel P. Berrangé
@ 2021-04-21  9:23               ` Dr. David Alan Gilbert
  2021-04-21  9:29               ` Vitaly Kuznetsov
  1 sibling, 0 replies; 15+ messages in thread
From: Dr. David Alan Gilbert @ 2021-04-21  9:23 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Paolo Bonzini, Vitaly Kuznetsov

* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
> > Eduardo Habkost <ehabkost@redhat.com> writes:
> > 
> > > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
> > >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> > >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> > >> > > older machine types are still available (I disable it for <= 5.1 but we
> > >> > > can consider disabling it for 5.2 too). The feature is upstream since
> > >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> > >> > > probably mean that we can't enable new KVM PV features unless all
> > >> > > supported kernels have it, we'd have to wait many years otherwise.
> > >> > 
> > >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> > >> > though that will go away in 6.1.
> > >> > 
> > >> > We should take the occasion of dropping RHEL7 to be clearer about which
> > >> > kernels are supported.
> > >> 
> > >> It would be nice to be able to define sets of KVM functonality that we
> > >> can either start given machine types with, or provide a separate switch
> > >> to limit kvm functionality back to some defined point.  We do trip over
> > >> the same things pretty regularly when accidentally turning on new
> > >> features.
> > >
> > > The same idea can apply to the hyperv=on stuff Vitaly is working
> > > on.  Maybe we should consider making a generic version of the
> > > s390x FeatGroup code, use it to define convenient sets of KVM and
> > > hyperv features.
> > 
> > True, the more I look at PV features enablement, the more I think that
> > we're missing something important in the logic. All machine types we
> > have are generally suposed to work with the oldest supported kernel so
> > we should wait many years before enabling some of the new PV features
> > (KVM or Hyper-V) by default.
> > 
> > This also links to our parallel discussion regarding migration
> > policies. Currently, we can't enable PV features by default based on
> > their availability on the host because of migration, the set may differ
> > on the destination host. What if we introduce (and maybe even switch to
> > it by default) something like
> > 
> >  -migratable opportunistic (stupid name, I know)
> > 
> > which would allow to enable all features supported by the source host
> > and then somehow checking that the destination host has them all. This
> > would effectively mean that it is possible to migrate a VM to a
> > same-or-newer software (both kernel an QEMU) but not the other way
> > around. This may be a reasonable choice.
> 
> I don't think this is usable in pratice. Any large cloud or data center
> mgmt app using QEMU relies on migration, so can't opportunistically
> use arbitrary new features. They can only use features in the oldest
> kernel their deployment cares about. This can be newer than the oldest
> that QEMU supports, but still older than the newest that exists.
> 
> ie we have situation where:
> 
>  - QEMU upstream minimum host is version 7
>  - Latest possible host is version 45
>  - A particular deployment has a mixture of hosts at version 24 and 37
> 
> "-migratable opportunistic"  would let QEMU use features from version 37
> despite the deployment needing compatibility with host version 24 still.
> 
> 
> It is almost as if we need to have a way to explicitly express a minimum
> required host version that VM requires compatibility with, so deployments
> can set their own baseline that is newer than QEMU minimum.

It's not a 'version' - it's just the set of capabilities, and the qemu
needs to check them at startup and fail if they're missing; I think
that's what thats FeatGroup is that was suggested.

Just like we have machine type and CPU version we need a set of PV
features that we rely on the host kernel having, and we should only
expose those PV features to the guest.  It's possible that we might
define some machine types as relying on certain PV features, or that
some PV features wouldn't make sense on some machine types.

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  8:50             ` Daniel P. Berrangé
  2021-04-21  9:23               ` Dr. David Alan Gilbert
@ 2021-04-21  9:29               ` Vitaly Kuznetsov
  2021-04-21  9:34                 ` Dr. David Alan Gilbert
  2021-04-21  9:37                 ` Daniel P. Berrangé
  1 sibling, 2 replies; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-21  9:29 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Dr. David Alan Gilbert, Paolo Bonzini

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>> 
>> > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
>> >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>> >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
>> >> > > older machine types are still available (I disable it for <= 5.1 but we
>> >> > > can consider disabling it for 5.2 too). The feature is upstream since
>> >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
>> >> > > probably mean that we can't enable new KVM PV features unless all
>> >> > > supported kernels have it, we'd have to wait many years otherwise.
>> >> > 
>> >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
>> >> > though that will go away in 6.1.
>> >> > 
>> >> > We should take the occasion of dropping RHEL7 to be clearer about which
>> >> > kernels are supported.
>> >> 
>> >> It would be nice to be able to define sets of KVM functonality that we
>> >> can either start given machine types with, or provide a separate switch
>> >> to limit kvm functionality back to some defined point.  We do trip over
>> >> the same things pretty regularly when accidentally turning on new
>> >> features.
>> >
>> > The same idea can apply to the hyperv=on stuff Vitaly is working
>> > on.  Maybe we should consider making a generic version of the
>> > s390x FeatGroup code, use it to define convenient sets of KVM and
>> > hyperv features.
>> 
>> True, the more I look at PV features enablement, the more I think that
>> we're missing something important in the logic. All machine types we
>> have are generally suposed to work with the oldest supported kernel so
>> we should wait many years before enabling some of the new PV features
>> (KVM or Hyper-V) by default.
>> 
>> This also links to our parallel discussion regarding migration
>> policies. Currently, we can't enable PV features by default based on
>> their availability on the host because of migration, the set may differ
>> on the destination host. What if we introduce (and maybe even switch to
>> it by default) something like
>> 
>>  -migratable opportunistic (stupid name, I know)
>> 
>> which would allow to enable all features supported by the source host
>> and then somehow checking that the destination host has them all. This
>> would effectively mean that it is possible to migrate a VM to a
>> same-or-newer software (both kernel an QEMU) but not the other way
>> around. This may be a reasonable choice.
>
> I don't think this is usable in pratice. Any large cloud or data center
> mgmt app using QEMU relies on migration, so can't opportunistically
> use arbitrary new features. They can only use features in the oldest
> kernel their deployment cares about. This can be newer than the oldest
> that QEMU supports, but still older than the newest that exists.
>
> ie we have situation where:
>
>  - QEMU upstream minimum host is version 7
>  - Latest possible host is version 45
>  - A particular deployment has a mixture of hosts at version 24 and 37
>
> "-migratable opportunistic"  would let QEMU use features from version 37
> despite the deployment needing compatibility with host version 24 still.
>

True; I was not really thinking about 'big' clouds/data centers, these
should have enough resources to carefully set all the required features
and not rely on the 'default'. My thoughts were around using migration
for host upgrade on smaller (several hosts) deployments and in this case
it's probably fairly reasonable to require to start with the oldest host
and upgrade them all if getting new features is one of the upgrade goals.

>
> It is almost as if we need to have a way to explicitly express a minimum
> required host version that VM requires compatibility with, so deployments
> can set their own baseline that is newer than QEMU minimum.

Yes, maybe, but setting the baseline is also a non-trivial task:
e.g. how would users know which PV features they can enable without
going through Linux kernel logs or just trying them on the oldest kernel
they need? This should probably be solved by some upper layer management
app which would collect feature sets from all hosts and come up with a
common subset. I'm not sure if this is done by some tools already.

-- 
Vitaly



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  9:29               ` Vitaly Kuznetsov
@ 2021-04-21  9:34                 ` Dr. David Alan Gilbert
  2021-04-21  9:37                 ` Daniel P. Berrangé
  1 sibling, 0 replies; 15+ messages in thread
From: Dr. David Alan Gilbert @ 2021-04-21  9:34 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Daniel P. Berrangé,
	Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Paolo Bonzini

* Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >> 
> >> > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
> >> >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >> >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> >> >> > > older machine types are still available (I disable it for <= 5.1 but we
> >> >> > > can consider disabling it for 5.2 too). The feature is upstream since
> >> >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> >> >> > > probably mean that we can't enable new KVM PV features unless all
> >> >> > > supported kernels have it, we'd have to wait many years otherwise.
> >> >> > 
> >> >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> >> >> > though that will go away in 6.1.
> >> >> > 
> >> >> > We should take the occasion of dropping RHEL7 to be clearer about which
> >> >> > kernels are supported.
> >> >> 
> >> >> It would be nice to be able to define sets of KVM functonality that we
> >> >> can either start given machine types with, or provide a separate switch
> >> >> to limit kvm functionality back to some defined point.  We do trip over
> >> >> the same things pretty regularly when accidentally turning on new
> >> >> features.
> >> >
> >> > The same idea can apply to the hyperv=on stuff Vitaly is working
> >> > on.  Maybe we should consider making a generic version of the
> >> > s390x FeatGroup code, use it to define convenient sets of KVM and
> >> > hyperv features.
> >> 
> >> True, the more I look at PV features enablement, the more I think that
> >> we're missing something important in the logic. All machine types we
> >> have are generally suposed to work with the oldest supported kernel so
> >> we should wait many years before enabling some of the new PV features
> >> (KVM or Hyper-V) by default.
> >> 
> >> This also links to our parallel discussion regarding migration
> >> policies. Currently, we can't enable PV features by default based on
> >> their availability on the host because of migration, the set may differ
> >> on the destination host. What if we introduce (and maybe even switch to
> >> it by default) something like
> >> 
> >>  -migratable opportunistic (stupid name, I know)
> >> 
> >> which would allow to enable all features supported by the source host
> >> and then somehow checking that the destination host has them all. This
> >> would effectively mean that it is possible to migrate a VM to a
> >> same-or-newer software (both kernel an QEMU) but not the other way
> >> around. This may be a reasonable choice.
> >
> > I don't think this is usable in pratice. Any large cloud or data center
> > mgmt app using QEMU relies on migration, so can't opportunistically
> > use arbitrary new features. They can only use features in the oldest
> > kernel their deployment cares about. This can be newer than the oldest
> > that QEMU supports, but still older than the newest that exists.
> >
> > ie we have situation where:
> >
> >  - QEMU upstream minimum host is version 7
> >  - Latest possible host is version 45
> >  - A particular deployment has a mixture of hosts at version 24 and 37
> >
> > "-migratable opportunistic"  would let QEMU use features from version 37
> > despite the deployment needing compatibility with host version 24 still.
> >
> 
> True; I was not really thinking about 'big' clouds/data centers, these
> should have enough resources to carefully set all the required features
> and not rely on the 'default'. My thoughts were around using migration
> for host upgrade on smaller (several hosts) deployments and in this case
> it's probably fairly reasonable to require to start with the oldest host
> and upgrade them all if getting new features is one of the upgrade goals.

It's not actually that simple.
Small installations tend to have less spare hardware available and/or
flexibility; if you've got say a 3 or 5 host cluster, once you start
upgrading one node you've now got nowhere to go if you hit a problem.

Dave

> >
> > It is almost as if we need to have a way to explicitly express a minimum
> > required host version that VM requires compatibility with, so deployments
> > can set their own baseline that is newer than QEMU minimum.
> 
> Yes, maybe, but setting the baseline is also a non-trivial task:
> e.g. how would users know which PV features they can enable without
> going through Linux kernel logs or just trying them on the oldest kernel
> they need? This should probably be solved by some upper layer management
> app which would collect feature sets from all hosts and come up with a
> common subset. I'm not sure if this is done by some tools already.
> 
> -- 
> Vitaly
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  9:29               ` Vitaly Kuznetsov
  2021-04-21  9:34                 ` Dr. David Alan Gilbert
@ 2021-04-21  9:37                 ` Daniel P. Berrangé
  2021-04-21  9:48                   ` Vitaly Kuznetsov
  1 sibling, 1 reply; 15+ messages in thread
From: Daniel P. Berrangé @ 2021-04-21  9:37 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Dr. David Alan Gilbert, Paolo Bonzini

On Wed, Apr 21, 2021 at 11:29:45AM +0200, Vitaly Kuznetsov wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >> 
> >> > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
> >> >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >> >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
> >> >> > > older machine types are still available (I disable it for <= 5.1 but we
> >> >> > > can consider disabling it for 5.2 too). The feature is upstream since
> >> >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
> >> >> > > probably mean that we can't enable new KVM PV features unless all
> >> >> > > supported kernels have it, we'd have to wait many years otherwise.
> >> >> > 
> >> >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
> >> >> > though that will go away in 6.1.
> >> >> > 
> >> >> > We should take the occasion of dropping RHEL7 to be clearer about which
> >> >> > kernels are supported.
> >> >> 
> >> >> It would be nice to be able to define sets of KVM functonality that we
> >> >> can either start given machine types with, or provide a separate switch
> >> >> to limit kvm functionality back to some defined point.  We do trip over
> >> >> the same things pretty regularly when accidentally turning on new
> >> >> features.
> >> >
> >> > The same idea can apply to the hyperv=on stuff Vitaly is working
> >> > on.  Maybe we should consider making a generic version of the
> >> > s390x FeatGroup code, use it to define convenient sets of KVM and
> >> > hyperv features.
> >> 
> >> True, the more I look at PV features enablement, the more I think that
> >> we're missing something important in the logic. All machine types we
> >> have are generally suposed to work with the oldest supported kernel so
> >> we should wait many years before enabling some of the new PV features
> >> (KVM or Hyper-V) by default.
> >> 
> >> This also links to our parallel discussion regarding migration
> >> policies. Currently, we can't enable PV features by default based on
> >> their availability on the host because of migration, the set may differ
> >> on the destination host. What if we introduce (and maybe even switch to
> >> it by default) something like
> >> 
> >>  -migratable opportunistic (stupid name, I know)
> >> 
> >> which would allow to enable all features supported by the source host
> >> and then somehow checking that the destination host has them all. This
> >> would effectively mean that it is possible to migrate a VM to a
> >> same-or-newer software (both kernel an QEMU) but not the other way
> >> around. This may be a reasonable choice.
> >
> > I don't think this is usable in pratice. Any large cloud or data center
> > mgmt app using QEMU relies on migration, so can't opportunistically
> > use arbitrary new features. They can only use features in the oldest
> > kernel their deployment cares about. This can be newer than the oldest
> > that QEMU supports, but still older than the newest that exists.
> >
> > ie we have situation where:
> >
> >  - QEMU upstream minimum host is version 7
> >  - Latest possible host is version 45
> >  - A particular deployment has a mixture of hosts at version 24 and 37
> >
> > "-migratable opportunistic"  would let QEMU use features from version 37
> > despite the deployment needing compatibility with host version 24 still.
> >
> 
> True; I was not really thinking about 'big' clouds/data centers, these
> should have enough resources to carefully set all the required features
> and not rely on the 'default'. My thoughts were around using migration
> for host upgrade on smaller (several hosts) deployments and in this case
> it's probably fairly reasonable to require to start with the oldest host
> and upgrade them all if getting new features is one of the upgrade goals.


> > It is almost as if we need to have a way to explicitly express a minimum
> > required host version that VM requires compatibility with, so deployments
> > can set their own baseline that is newer than QEMU minimum.
> 
> Yes, maybe, but setting the baseline is also a non-trivial task:
> e.g. how would users know which PV features they can enable without
> going through Linux kernel logs or just trying them on the oldest kernel
> they need? This should probably be solved by some upper layer management
> app which would collect feature sets from all hosts and come up with a
> common subset. I'm not sure if this is done by some tools already.

I specifically didn't talk in terms of features, because the problem you
describe is unreasonable to push onto applications.

Rather QEMU could express host baseline

   - "host-v1"  - features A and B
   - "host-v2"  - features A, B and C
   - "host-v3"  - features A, B, C, D, E and f

The mgmt app / admin only has to know which QEMU host baselines their
hosts support.

Essentially this could be viewed as separating the host kernel dependant
bits out of the machine type, into a separate configuration axis.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 0/2] i386: Fix interrupt based Async PF enablement
  2021-04-21  9:37                 ` Daniel P. Berrangé
@ 2021-04-21  9:48                   ` Vitaly Kuznetsov
  0 siblings, 0 replies; 15+ messages in thread
From: Vitaly Kuznetsov @ 2021-04-21  9:48 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Eduardo Habkost, Michael S. Tsirkin, Richard Henderson,
	qemu-devel, Dr. David Alan Gilbert, Paolo Bonzini

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Apr 21, 2021 at 11:29:45AM +0200, Vitaly Kuznetsov wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > On Wed, Apr 21, 2021 at 10:38:06AM +0200, Vitaly Kuznetsov wrote:
>> >> Eduardo Habkost <ehabkost@redhat.com> writes:
>> >> 
>> >> > On Thu, Apr 15, 2021 at 08:14:30PM +0100, Dr. David Alan Gilbert wrote:
>> >> >> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>> >> >> > On 06/04/21 13:42, Vitaly Kuznetsov wrote:
>> >> >> > > older machine types are still available (I disable it for <= 5.1 but we
>> >> >> > > can consider disabling it for 5.2 too). The feature is upstream since
>> >> >> > > Linux 5.8, I know that QEMU supports much older kernels but this doesn't
>> >> >> > > probably mean that we can't enable new KVM PV features unless all
>> >> >> > > supported kernels have it, we'd have to wait many years otherwise.
>> >> >> > 
>> >> >> > Yes, this is a known problem in fact. :(  In 6.0 we even support RHEL 7,
>> >> >> > though that will go away in 6.1.
>> >> >> > 
>> >> >> > We should take the occasion of dropping RHEL7 to be clearer about which
>> >> >> > kernels are supported.
>> >> >> 
>> >> >> It would be nice to be able to define sets of KVM functonality that we
>> >> >> can either start given machine types with, or provide a separate switch
>> >> >> to limit kvm functionality back to some defined point.  We do trip over
>> >> >> the same things pretty regularly when accidentally turning on new
>> >> >> features.
>> >> >
>> >> > The same idea can apply to the hyperv=on stuff Vitaly is working
>> >> > on.  Maybe we should consider making a generic version of the
>> >> > s390x FeatGroup code, use it to define convenient sets of KVM and
>> >> > hyperv features.
>> >> 
>> >> True, the more I look at PV features enablement, the more I think that
>> >> we're missing something important in the logic. All machine types we
>> >> have are generally suposed to work with the oldest supported kernel so
>> >> we should wait many years before enabling some of the new PV features
>> >> (KVM or Hyper-V) by default.
>> >> 
>> >> This also links to our parallel discussion regarding migration
>> >> policies. Currently, we can't enable PV features by default based on
>> >> their availability on the host because of migration, the set may differ
>> >> on the destination host. What if we introduce (and maybe even switch to
>> >> it by default) something like
>> >> 
>> >>  -migratable opportunistic (stupid name, I know)
>> >> 
>> >> which would allow to enable all features supported by the source host
>> >> and then somehow checking that the destination host has them all. This
>> >> would effectively mean that it is possible to migrate a VM to a
>> >> same-or-newer software (both kernel an QEMU) but not the other way
>> >> around. This may be a reasonable choice.
>> >
>> > I don't think this is usable in pratice. Any large cloud or data center
>> > mgmt app using QEMU relies on migration, so can't opportunistically
>> > use arbitrary new features. They can only use features in the oldest
>> > kernel their deployment cares about. This can be newer than the oldest
>> > that QEMU supports, but still older than the newest that exists.
>> >
>> > ie we have situation where:
>> >
>> >  - QEMU upstream minimum host is version 7
>> >  - Latest possible host is version 45
>> >  - A particular deployment has a mixture of hosts at version 24 and 37
>> >
>> > "-migratable opportunistic"  would let QEMU use features from version 37
>> > despite the deployment needing compatibility with host version 24 still.
>> >
>> 
>> True; I was not really thinking about 'big' clouds/data centers, these
>> should have enough resources to carefully set all the required features
>> and not rely on the 'default'. My thoughts were around using migration
>> for host upgrade on smaller (several hosts) deployments and in this case
>> it's probably fairly reasonable to require to start with the oldest host
>> and upgrade them all if getting new features is one of the upgrade goals.
>
>
>> > It is almost as if we need to have a way to explicitly express a minimum
>> > required host version that VM requires compatibility with, so deployments
>> > can set their own baseline that is newer than QEMU minimum.
>> 
>> Yes, maybe, but setting the baseline is also a non-trivial task:
>> e.g. how would users know which PV features they can enable without
>> going through Linux kernel logs or just trying them on the oldest kernel
>> they need? This should probably be solved by some upper layer management
>> app which would collect feature sets from all hosts and come up with a
>> common subset. I'm not sure if this is done by some tools already.
>
> I specifically didn't talk in terms of features, because the problem you
> describe is unreasonable to push onto applications.
>
> Rather QEMU could express host baseline
>
>    - "host-v1"  - features A and B
>    - "host-v2"  - features A, B and C
>    - "host-v3"  - features A, B, C, D, E and f
>
> The mgmt app / admin only has to know which QEMU host baselines their
> hosts support.
>
> Essentially this could be viewed as separating the host kernel dependant
> bits out of the machine type, into a separate configuration axis.

In case we only think about upstream kernels and assuming PV features
never go away that coud work. Distro kernels, however, exist too and
feature backports are common, so which version should I declare when my
kernel has e.g. features A, B and E ? (There used to be
KVM_GET_API_VERSION ioctl but then we switched to CAPs and this happened
for a reason.)

Personaly, I'd vote for having individual PV features in the config if
it ever gets introduced.

-- 
Vitaly



^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, back to index

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-01 15:19 [PATCH 0/2] i386: Fix interrupt based Async PF enablement Vitaly Kuznetsov
2021-04-01 15:19 ` [PATCH 1/2] i386: Add 'kvm-asyncpf-int' to kvm_default_props array Vitaly Kuznetsov
2021-04-01 15:19 ` [PATCH 2/2] i386: Disable 'kvm-asyncpf-int' feature for machine types <= 5.1 Vitaly Kuznetsov
2021-04-01 15:57 ` [PATCH 0/2] i386: Fix interrupt based Async PF enablement Paolo Bonzini
2021-04-06 11:42   ` Vitaly Kuznetsov
2021-04-08 12:46     ` Paolo Bonzini
2021-04-15 19:14       ` Dr. David Alan Gilbert
2021-04-20 17:35         ` Eduardo Habkost
2021-04-21  8:38           ` Vitaly Kuznetsov
2021-04-21  8:50             ` Daniel P. Berrangé
2021-04-21  9:23               ` Dr. David Alan Gilbert
2021-04-21  9:29               ` Vitaly Kuznetsov
2021-04-21  9:34                 ` Dr. David Alan Gilbert
2021-04-21  9:37                 ` Daniel P. Berrangé
2021-04-21  9:48                   ` Vitaly Kuznetsov

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git
	git clone --mirror https://lore.kernel.org/qemu-devel/2 qemu-devel/git/2.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git