[Xen-devel] max_grant_frames/max_maptrack_frames

[Xen-devel] max_grant_frames/max_maptrack_frames

[Durrant, Paul]

Picking up the discussion from IRC to make it more widely visible...

When per-domain options for maximum grant and maptrack frames came in (in 4.10?) Xen's behaviour w.r.t. to the global command line values (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed

For example, a host running a prior version of Xen with a command line setting gnttab_max_frames=128 would have all of its domUs running with 128 frames. However, after update to a newer Xen, they will only get 32 frames (unless the host is particularly large, in which case they will get 64). Why is this? It's because neither xl.cfg files, nor xl.conf, will specify values (because the scenario is an update from an older installation) and so the hardcoded 32/64 default applies. Hence some domUs with large numbers of PV devices start failing (or at least substantially slow down) and admins start wondering what's going on.

So how best to fix this?

For the sake of a quick fix for the regression, and ease of back-porting, I think it would be best to add a check in domain_create() and create the grant table with parameters which are the larger of the toolstack configured value and the corresponding command line value. This does, however, go against the recent direction of the toolstack getting exactly what it asked for. So for the longer term I am wondering whether there ought to be a way for the toolstack to query the globally configured grant table limits. A GNTTABOP seems the wrong candidate for this, since GNTTABOPs are per-domain, so I'm wondering about a new sysctl to return the value of a named command line parameter.

Thoughts?

  Paul

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] max_grant_frames/max_maptrack_frames

[Jan Beulich]

On 08.11.2019 09:45,  Durrant, Paul  wrote:
> When per-domain options for maximum grant and maptrack frames came in (in 4.10?) Xen's behaviour w.r.t. to the global command line values (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed
> 
> For example, a host running a prior version of Xen with a command line setting gnttab_max_frames=128 would have all of its domUs running with 128 frames. However, after update to a newer Xen, they will only get 32 frames (unless the host is particularly large, in which case they will get 64). Why is this? It's because neither xl.cfg files, nor xl.conf, will specify values (because the scenario is an update from an older installation) and so the hardcoded 32/64 default applies. Hence some domUs with large numbers of PV devices start failing (or at least substantially slow down) and admins start wondering what's going on.
> 
> So how best to fix this?
> 
> For the sake of a quick fix for the regression, and ease of back-porting, I think it would be best to add a check in domain_create() and create the grant table with parameters which are the larger of the toolstack configured value and the corresponding command line value.

How about people simply setting the value in xl.conf, if indeed in can be
set there?

> This does, however, go against the recent direction of the toolstack getting exactly what it asked for. So for the longer term I am wondering whether there ought to be a way for the toolstack to query the globally configured grant table limits. A GNTTABOP seems the wrong candidate for this, since GNTTABOPs are per-domain, so I'm wondering about a new sysctl to return the value of a named command line parameter.

Such a series was already posted (and even had some review, so it's
already at v4, but iirc no update has been provided since May):
https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg02206.html

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] max_grant_frames/max_maptrack_frames

[Jürgen Groß]

On 08.11.19 12:38, Jan Beulich wrote:
> On 08.11.2019 09:45,  Durrant, Paul  wrote:
>> When per-domain options for maximum grant and maptrack frames came in (in 4.10?) Xen's behaviour w.r.t. to the global command line values (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed
>>
>> For example, a host running a prior version of Xen with a command line setting gnttab_max_frames=128 would have all of its domUs running with 128 frames. However, after update to a newer Xen, they will only get 32 frames (unless the host is particularly large, in which case they will get 64). Why is this? It's because neither xl.cfg files, nor xl.conf, will specify values (because the scenario is an update from an older installation) and so the hardcoded 32/64 default applies. Hence some domUs with large numbers of PV devices start failing (or at least substantially slow down) and admins start wondering what's going on.
>>
>> So how best to fix this?
>>
>> For the sake of a quick fix for the regression, and ease of back-porting, I think it would be best to add a check in domain_create() and create the grant table with parameters which are the larger of the toolstack configured value and the corresponding command line value.
> 
> How about people simply setting the value in xl.conf, if indeed in can be
> set there?
> 
>> This does, however, go against the recent direction of the toolstack getting exactly what it asked for. So for the longer term I am wondering whether there ought to be a way for the toolstack to query the globally configured grant table limits. A GNTTABOP seems the wrong candidate for this, since GNTTABOPs are per-domain, so I'm wondering about a new sysctl to return the value of a named command line parameter.
> 
> Such a series was already posted (and even had some review, so it's
> already at v4, but iirc no update has been provided since May):
> https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg02206.html

My "Hypervisor file system" series includes that functionality:

https://patchew.org/Xen/20191002112004.25793-1-jgross@suse.com/


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] max_grant_frames/max_maptrack_frames

[Durrant, Paul]

> -----Original Message-----
> From: Jürgen Groß <jgross@suse.com>
> Sent: 08 November 2019 12:14
> To: Jan Beulich <jbeulich@suse.com>; Durrant, Paul <pdurrant@amazon.com>
> Cc: xen-devel@lists.xenproject.org
> Subject: Re: [Xen-devel] max_grant_frames/max_maptrack_frames
> 
> On 08.11.19 12:38, Jan Beulich wrote:
> > On 08.11.2019 09:45,  Durrant, Paul  wrote:
> >> When per-domain options for maximum grant and maptrack frames came in
> (in 4.10?) Xen's behaviour w.r.t. to the global command line values
> (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed
> >>
> >> For example, a host running a prior version of Xen with a command line
> setting gnttab_max_frames=128 would have all of its domUs running with 128
> frames. However, after update to a newer Xen, they will only get 32 frames
> (unless the host is particularly large, in which case they will get 64).
> Why is this? It's because neither xl.cfg files, nor xl.conf, will specify
> values (because the scenario is an update from an older installation) and
> so the hardcoded 32/64 default applies. Hence some domUs with large
> numbers of PV devices start failing (or at least substantially slow down)
> and admins start wondering what's going on.
> >>
> >> So how best to fix this?
> >>
> >> For the sake of a quick fix for the regression, and ease of back-
> porting, I think it would be best to add a check in domain_create() and
> create the grant table with parameters which are the larger of the
> toolstack configured value and the corresponding command line value.
> >
> > How about people simply setting the value in xl.conf, if indeed in can
> be
> > set there?
> >
> >> This does, however, go against the recent direction of the toolstack
> getting exactly what it asked for. So for the longer term I am wondering
> whether there ought to be a way for the toolstack to query the globally
> configured grant table limits. A GNTTABOP seems the wrong candidate for
> this, since GNTTABOPs are per-domain, so I'm wondering about a new sysctl
> to return the value of a named command line parameter.
> >
> > Such a series was already posted (and even had some review, so it's
> > already at v4, but iirc no update has been provided since May):
> > https://lists.xenproject.org/archives/html/xen-devel/2019-
> 05/msg02206.html
> 
> My "Hypervisor file system" series includes that functionality:
> 
> https://patchew.org/Xen/20191002112004.25793-1-jgross@suse.com/
> 

Oh, even better :-)

  Paul

> 
> Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] max_grant_frames/max_maptrack_frames

[Durrant, Paul]

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: 08 November 2019 11:38
> To: Durrant, Paul <pdurrant@amazon.com>
> Cc: xen-devel@lists.xenproject.org
> Subject: Re: [Xen-devel] max_grant_frames/max_maptrack_frames
> 
> On 08.11.2019 09:45,  Durrant, Paul  wrote:
> > When per-domain options for maximum grant and maptrack frames came in
> (in 4.10?) Xen's behaviour w.r.t. to the global command line values
> (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed
> >
> > For example, a host running a prior version of Xen with a command line
> setting gnttab_max_frames=128 would have all of its domUs running with 128
> frames. However, after update to a newer Xen, they will only get 32 frames
> (unless the host is particularly large, in which case they will get 64).
> Why is this? It's because neither xl.cfg files, nor xl.conf, will specify
> values (because the scenario is an update from an older installation) and
> so the hardcoded 32/64 default applies. Hence some domUs with large
> numbers of PV devices start failing (or at least substantially slow down)
> and admins start wondering what's going on.
> >
> > So how best to fix this?
> >
> > For the sake of a quick fix for the regression, and ease of back-
> porting, I think it would be best to add a check in domain_create() and
> create the grant table with parameters which are the larger of the
> toolstack configured value and the corresponding command line value.
> 
> How about people simply setting the value in xl.conf, if indeed in can be
> set there?

It could be set there, but that's really not the right solution. A set of command line parameters that appropriately configured the host on an older Xen really ought to continue to do the same after installation of the newer Xen, without any additional config requirements.

> 
> > This does, however, go against the recent direction of the toolstack
> getting exactly what it asked for. So for the longer term I am wondering
> whether there ought to be a way for the toolstack to query the globally
> configured grant table limits. A GNTTABOP seems the wrong candidate for
> this, since GNTTABOPs are per-domain, so I'm wondering about a new sysctl
> to return the value of a named command line parameter.
> 
> Such a series was already posted (and even had some review, so it's
> already at v4, but iirc no update has been provided since May):
> https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg02206.html

Ok, I'll take a look. Thanks,

  Paul

> 
> Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] max_grant_frames/max_maptrack_frames

[Jan Beulich]

On 08.11.2019 13:33,  Durrant, Paul  wrote:
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@suse.com>
>> Sent: 08 November 2019 11:38
>> To: Durrant, Paul <pdurrant@amazon.com>
>> Cc: xen-devel@lists.xenproject.org
>> Subject: Re: [Xen-devel] max_grant_frames/max_maptrack_frames
>>
>> On 08.11.2019 09:45,  Durrant, Paul  wrote:
>>> When per-domain options for maximum grant and maptrack frames came in
>> (in 4.10?) Xen's behaviour w.r.t. to the global command line values
>> (gnttab_max_frames and gnttab_max_maptrack_frames respectively) regressed
>>>
>>> For example, a host running a prior version of Xen with a command line
>> setting gnttab_max_frames=128 would have all of its domUs running with 128
>> frames. However, after update to a newer Xen, they will only get 32 frames
>> (unless the host is particularly large, in which case they will get 64).
>> Why is this? It's because neither xl.cfg files, nor xl.conf, will specify
>> values (because the scenario is an update from an older installation) and
>> so the hardcoded 32/64 default applies. Hence some domUs with large
>> numbers of PV devices start failing (or at least substantially slow down)
>> and admins start wondering what's going on.
>>>
>>> So how best to fix this?
>>>
>>> For the sake of a quick fix for the regression, and ease of back-
>> porting, I think it would be best to add a check in domain_create() and
>> create the grant table with parameters which are the larger of the
>> toolstack configured value and the corresponding command line value.
>>
>> How about people simply setting the value in xl.conf, if indeed in can be
>> set there?
> 
> It could be set there, but that's really not the right solution. A set of command line parameters that appropriately configured the host on an older Xen really ought to continue to do the same after installation of the newer Xen, without any additional config requirements.

I guess it depends on the perspective you take: While quite likely the
situation could have been avoided here, it ought to be permissible for
us to decide that we intentionally want to change the meaning of a
command line option (which includes possibly ignoring it in certain
cases). Such a decision would better be documented in the release
notes, yes, but it may still imply other adjustments for host admins
to make during a version upgrade.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel