Bug: toolstack allows too low values to be set for shadow_memory

Bug: toolstack allows too low values to be set for shadow_memory

[Hans van Kranenburg]

This bug report is a follow-up to the thread "Domu windows 2012 crash"
on the xen-users list. In there we found out that it is possible to set
a value for shadow_memory that is lower than a safe minimum value.

This became apparent after XSA-304, which caused using more of this type
of memory. Having a hardcoded line like shadow_memory = 8 results in
random crashes of the guest, with the following errors in xl dmesg of
the host:

(XEN) Failed to shatter gfn 105245: -12
(XEN) d75v1 EPT violation 0x19c (--x/rw-) gpa 0x00000105245760 mfn
0x285245 type 0
(XEN) d75v1 Walking EPT tables for GFN 105245:
(XEN) d75v1  epte 9c000004105f9007
(XEN) d75v1  epte 9c000002800000f3
(XEN) d75v1  --- GLA 0x7ff98b40d760
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 75 (vcpu#1) crashed on cpu#4:
(XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    0033:[<00007ff98b40d760>]
(XEN) RFLAGS: 0000000000010216   CONTEXT: hvm guest (d75v1)
(XEN) rax: 0000000000001212   rbx: 000000c714d9da58   rcx: 0000023500001590
(XEN) rdx: 000000c700000001   rsi: 000000c714d9da18   rdi: 000000c714d9db20
(XEN) rbp: 000000c714d9d950   rsp: 000000c714d9d918   r8:  0000023500001470
(XEN) r9:  00000235000014f0   r10: 00007ff99c5c0923   r11: 000000c714d9d970
(XEN) r12: 0000000000000000   r13: 000000c714d9d9d0   r14: 000000c714d9da58
(XEN) r15: 0000000000000006   cr0: 0000000080050031   cr4: 0000000000060678
(XEN) cr3: 00000001d9458002   cr2: 00007ff98b6fa048
(XEN) fsb: 0000000000000000   gsb: 000000c714e9e000   gss: ffffcd015dc40000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033

Why do users have this shadow_memory line in their guest config file?
Well, I guess it's simply because there are not many proper examples of
how to run Windows under Xen, so whoever is searching ze interwobz for
it will end up on a page like...


https://www.howtoforge.com/how-to-run-fully-virtualized-guests-hvm-with-xen-3.2-on-debian-lenny-x86_64

...and then start copy pasting and adjusting until things work. Or...
already have such a config for 10+ years and just only adjust things
when it breaks after upgrading Xen.

Hans van Kranenburg

Re: Bug: toolstack allows too low values to be set for shadow_memory

[Jan Beulich]

On 25.05.2020 17:51, Hans van Kranenburg wrote:
> This bug report is a follow-up to the thread "Domu windows 2012 crash"
> on the xen-users list. In there we found out that it is possible to set
> a value for shadow_memory that is lower than a safe minimum value.
> 
> This became apparent after XSA-304, which caused using more of this type
> of memory. Having a hardcoded line like shadow_memory = 8 results in
> random crashes of the guest,

I don't think it is the tool stack's responsibility to override
admin requested values, or at least not as far a affecting guest
stability goes; host stability of course needs to be guaranteed,
but that's then still the hypervisor's job, not the tool stack's.

Compare this to e.g. setting too small a memory= for a guest to
be able to boot at all, or setting maxmem > memory for a guest
without balloon driver.

Furthermore - what would the suggestion be as to a "safe minimum
value"? Assuming _all_ large pages may potentially get shattered
is surely a waste of memory, unless the admin really knows
guests are going to behave that way. (In your report you also
didn't mention what memory= values the issue was observed with.
Obviously larger memory= also require bumping shadow_memory= at
least from some point onwards.)

Jan

Re: Bug: toolstack allows too low values to be set for shadow_memory

[Hans van Kranenburg]

Hi,

On 5/26/20 9:41 AM, Jan Beulich wrote:
> On 25.05.2020 17:51, Hans van Kranenburg wrote:
>> This bug report is a follow-up to the thread "Domu windows 2012 crash"
>> on the xen-users list. In there we found out that it is possible to set
>> a value for shadow_memory that is lower than a safe minimum value.
>>
>> This became apparent after XSA-304, which caused using more of this type
>> of memory. Having a hardcoded line like shadow_memory = 8 results in
>> random crashes of the guest,
> 
> I don't think it is the tool stack's responsibility to override
> admin requested values, or at least not as far a affecting guest
> stability goes;

This is not primarily a technical issue, or about if software works
correct in a mathematical proven way.

It's a usability issue, so it's about what levels of unknowingly
unloading guns into feet is deemed desirable.

And, if that's happening, how difficult it should be for a user to
actually find out what's wrong.

> host stability of course needs to be guaranteed,
> but that's then still the hypervisor's job, not the tool stack's.
> 
> Compare this to e.g. setting too small a memory= for a guest to
> be able to boot at all, or setting maxmem > memory for a guest
> without balloon driver.
> 
> Furthermore - what would the suggestion be as to a "safe minimum
> value"? Assuming _all_ large pages may potentially get shattered
> is surely a waste of memory, unless the admin really knows
> guests are going to behave that way. (In your report you also
> didn't mention what memory= values the issue was observed with.
> Obviously larger memory= also require bumping shadow_memory= at
> least from some point onwards.)

Thanks,
Hans