Re: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames and max_maptrack_frames handling

From: Hans van Kranenburg <hans@knorrie.org>
To: George Dunlap <george.dunlap@citrix.com>,
	"Durrant, Paul" <pdurrant@amazon.com>,
	Ian Jackson <ian.jackson@citrix.com>
Cc: "Juergen Gross" <jgross@suse.com>,
	"Stefano Stabellini" <sstabellini@kernel.org>,
	"Julien Grall" <julien@xen.org>, "Wei Liu" <wl@xen.org>,
	"Paul Durrant" <paul@xen.org>,
	"Andrew Cooper" <Andrew.Cooper3@citrix.com>,
	"Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com>,
	"Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>,
	"Jan Beulich" <jbeulich@suse.com>,
	"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>
Subject: Re: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames and max_maptrack_frames handling
Date: Thu, 28 Nov 2019 16:33:33 +0100	[thread overview]
Message-ID: <5c90ac64-f5ed-744e-0d48-4e3aa2139a30@knorrie.org> (raw)
In-Reply-To: <414ed314-29fc-1b16-9516-790e53a65a4c@citrix.com>

On 11/28/19 3:54 PM, George Dunlap wrote:
> On 11/27/19 10:32 PM, Hans van Kranenburg wrote:
>> Hi all,
>>
>> On 11/27/19 12:13 PM, Durrant, Paul wrote:
>>>> -----Original Message-----
>>>> From: Ian Jackson <ian.jackson@citrix.com>
>>>> Sent: 27 November 2019 11:10
>>>> [...]
>>>> Subject: RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames
>>>> and max_maptrack_frames handling
>>>>
>>>> Durrant, Paul writes ("RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize
>>>> max_grant_frames and max_maptrack_frames handling"):
>>>>>> -----Original Message-----
>>>>>> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of
>>>> Ian
>>>>>> Jackson
>>>>>> I have seen reports of users who ran out of grant/maptrack frames
>>>>>> because of updates to use multiring protocols etc.  The error messages
>>>>>> are not very good and the recommended workaround has been to increase
>>>>>> the default limit on the hypervisor command line.
>>>>>>
>>>>>> It is important that we don't break that workaround!
>>>>>
>>>>> Alas it has apparently been broken for several releases now :-(
>>>>
>>>> I guess at least in Debian (where I have seen this) we haven't
>>>> released with any affected versions yet...
>>>
>>> I believe the problem was introduce in 4.10, so I think it would be prudent to also back-port the final fix to stable trees from then on.
>>
>> Yes, the max grant frame issue has historically always been a painful
>> experience for end users, and Xen 4.11 which we now have in the current
>> Debian stable has made it worse compared to previous versions indeed.
> 
> This rather suggests that the default value isn't very well chosen.
> Ideally some investigation would be done to improve the default sizing;
> end-users shouldn't have to know anything about grant table frames.

Most of the problems started happening a few years ago when using a
newer Linux that got all kinds of multiqueue block stuff for disk and
network enabled on top of an older Xen. (e.g. in Debian using the Linux
4.9 backports kernel on top of Xen 4.4 in Jessie).

The default for the hypervisor option has already been doubled from 32
to 64, which I think is sufficient. However, having the toolstack revert
it back to 32 again is not very helpful, but that's what this thread is
about to solve. :)

A while ago I did some testing:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554#119

I haven't been able to cause nr_frames to go over 64 in any test myself,
and also have never seen values that high in production use. The above
debian bug also does not contain any other report from anyone with a
number above 64. There are reports of users setting it to 256 and then
not caring about it any more, but they didn't report the xen_diag output
back after that, so there's no real data.

Hans

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel