Re: [Xen-devel] How to boot domU and dom0 from a device tree

Re: [Xen-devel] How to boot domU and dom0 from a device tree

[Julien Grall]

(Moving from xen-users to xen-devel).

On 11/06/2019 23:18, Stefano Stabellini wrote:
> I managed to reproduced the issue, and I know how to get past it.  Try
> using the raw kernel Image (arch/arm64/boot/Image) instead of Image.gz
> for dom0 and domU. That fixed it for me.
> 
> Julien, I didn't manage to figure out what the issue is exactly, but it
> looks like Image.gz loading is broken at the moment.

Do you mean Image.gz is broken from DomU? Because per the log provided by Denis, 
this is working perfectly for Dom0 as we don't create domain in parallel.

By reading the code I can already spot the reason of the first issue reported by 
Denis. For reminder, this is when Dom0 and DomU are using the same module 
address for the gzip Image.

This is because when probing the kernel for Dom0, the module will get 
uncompressed and the module start/end will be updated to point to the uncompress 
version. Because of that, the probe for DomU kernel will not be able to find the 
module (the start addressed changed).

In this case, I think we only want to uncompress the module one time to avoid 
wasting memory. The solution I have in mind requires some rework in Xen, I would 
actually start by probing the information for all the domains, then uncompress 
the kernels modules, and then finish to build the domain.

For the out of memory problem discussed in this e-mail, I think the problem is 
not because of lack of memory in DomU. The problem is related to the 
inflate/gunzip the code. The code is using an heap (see perform_gunzip) where it 
allocates memory from.

I am assuming the kernels for Dom0 and DomU are exactly the same but they are 
coming from different address. Am I correct? If so, I am a bit unsure this 
worked the first time and not the second time. This probably want some debugging 
to understand the problem. Denis, Stefano, can one of you look at it?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] How to boot domU and dom0 from a device tree

[Stefano Stabellini]

On Wed, 12 Jun 2019, Julien Grall wrote:
> (Moving from xen-users to xen-devel).
> 
> On 11/06/2019 23:18, Stefano Stabellini wrote:
> > I managed to reproduced the issue, and I know how to get past it.  Try
> > using the raw kernel Image (arch/arm64/boot/Image) instead of Image.gz
> > for dom0 and domU. That fixed it for me.
> > 
> > Julien, I didn't manage to figure out what the issue is exactly, but it
> > looks like Image.gz loading is broken at the moment.
> 
> Do you mean Image.gz is broken from DomU? Because per the log provided by
> Denis, this is working perfectly for Dom0 as we don't create domain in
> parallel.
> 
> By reading the code I can already spot the reason of the first issue reported
> by Denis. For reminder, this is when Dom0 and DomU are using the same module
> address for the gzip Image.
> 
> This is because when probing the kernel for Dom0, the module will get
> uncompressed and the module start/end will be updated to point to the
> uncompress version. Because of that, the probe for DomU kernel will not be
> able to find the module (the start addressed changed).
> 
> In this case, I think we only want to uncompress the module one time to avoid
> wasting memory. The solution I have in mind requires some rework in Xen, I
> would actually start by probing the information for all the domains, then
> uncompress the kernels modules, and then finish to build the domain.
> 
> For the out of memory problem discussed in this e-mail, I think the problem is
> not because of lack of memory in DomU. The problem is related to the
> inflate/gunzip the code. The code is using an heap (see perform_gunzip) where
> it allocates memory from.
> 
> I am assuming the kernels for Dom0 and DomU are exactly the same but they are
> coming from different address. Am I correct? If so, I am a bit unsure this
> worked the first time and not the second time. This probably want some
> debugging to understand the problem. Denis, Stefano, can one of you look at
> it?

I couldn't find exactly the root cause yet, but I can reproduce the
issue even with Dom0 only (no domUs, no dom0less):

(XEN) *** LOADING DOMAIN 0 ***
(XEN) DEBUG kernel_probe 445
(XEN) Loading d0 kernel from boot module @ 0000000047000000
(XEN) Loading ramdisk from boot module @ 0000000042000000
(XEN) DEBUG kernel_decompress 268
(XEN) DEBUG kernel_decompress 272
(XEN) DEBUG kernel_decompress 279
(XEN) DEBUG kernel_decompress 284
(XEN) DEBUG kernel_decompress 291 kernel_order_out=52 output_size=0
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...


The issue seems to be that output_size, returned by output_length(input,
size) is 0. Then, kernel_order_out is set to 52 which is too large. As a
consequence kernel_decompress returns with -ENOMEM.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] How to boot domU and dom0 from a device tree

[Julien Grall]

Hi Stefano,

On 6/14/19 9:53 PM, Stefano Stabellini wrote:
> On Wed, 12 Jun 2019, Julien Grall wrote:
>> (Moving from xen-users to xen-devel).
>>
>> On 11/06/2019 23:18, Stefano Stabellini wrote:
>>> I managed to reproduced the issue, and I know how to get past it.  Try
>>> using the raw kernel Image (arch/arm64/boot/Image) instead of Image.gz
>>> for dom0 and domU. That fixed it for me.
>>>
>>> Julien, I didn't manage to figure out what the issue is exactly, but it
>>> looks like Image.gz loading is broken at the moment.
>>
>> Do you mean Image.gz is broken from DomU? Because per the log provided by
>> Denis, this is working perfectly for Dom0 as we don't create domain in
>> parallel.
>>
>> By reading the code I can already spot the reason of the first issue reported
>> by Denis. For reminder, this is when Dom0 and DomU are using the same module
>> address for the gzip Image.
>>
>> This is because when probing the kernel for Dom0, the module will get
>> uncompressed and the module start/end will be updated to point to the
>> uncompress version. Because of that, the probe for DomU kernel will not be
>> able to find the module (the start addressed changed).
>>
>> In this case, I think we only want to uncompress the module one time to avoid
>> wasting memory. The solution I have in mind requires some rework in Xen, I
>> would actually start by probing the information for all the domains, then
>> uncompress the kernels modules, and then finish to build the domain.
>>
>> For the out of memory problem discussed in this e-mail, I think the problem is
>> not because of lack of memory in DomU. The problem is related to the
>> inflate/gunzip the code. The code is using an heap (see perform_gunzip) where
>> it allocates memory from.
>>
>> I am assuming the kernels for Dom0 and DomU are exactly the same but they are
>> coming from different address. Am I correct? If so, I am a bit unsure this
>> worked the first time and not the second time. This probably want some
>> debugging to understand the problem. Denis, Stefano, can one of you look at
>> it?
> 
> I couldn't find exactly the root cause yet, but I can reproduce the
> issue even with Dom0 only (no domUs, no dom0less):

Looking at Denis's report, the error does not seem to be the same:

(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Out of memory
(XEN) ****************************************


But I think they may be related (see below).


> ee.
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) DEBUG kernel_probe 445
> (XEN) Loading d0 kernel from boot module @ 0000000047000000
> (XEN) Loading ramdisk from boot module @ 0000000042000000
> (XEN) DEBUG kernel_decompress 268
> (XEN) DEBUG kernel_decompress 272
> (XEN) DEBUG kernel_decompress 279
> (XEN) DEBUG kernel_decompress 284
> (XEN) DEBUG kernel_decompress 291 kernel_order_out=52 output_size=0
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Could not set up DOM0 guest OS
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> 
> 
> The issue seems to be that output_size, returned by output_length(input,
> size) is 0. Then, kernel_order_out is set to 52 which is too large. As a
> consequence kernel_decompress returns with -ENOMEM.

I have just tried to use compressed kernel and can't reproduce your 
error. However, I think the two problems ("out of memory" and your one) 
are because the module size does not exactly match the size of the 
compressed image.

The uncompressed size is part of the footer (the last 4-bytes). As we 
only have the module size in hand, we assume it is equal to the 
compressed size. If not, then we will return whatever is in the last 
4-bytes of the module.

This means the module size should exactly match the compressed image 
size. AFAICT, gzip format doesn't provide a field for the compressed 
size, so we can't do better in Xen.

In other word, the Device-Tree multiboot nodes should be created with 
the exact size of the compressed image.

Regardless that, I still think we have some issues when using the same 
compressed kernel for Dom0 and DomU (see in my previous e-mail).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] How to boot domU and dom0 from a device tree

[Stefano Stabellini]

On Sat, 15 Jun 2019, Julien Grall wrote:
> Hi Stefano,
> 
> On 6/14/19 9:53 PM, Stefano Stabellini wrote:
> > On Wed, 12 Jun 2019, Julien Grall wrote:
> > > (Moving from xen-users to xen-devel).
> > > 
> > > On 11/06/2019 23:18, Stefano Stabellini wrote:
> > > > I managed to reproduced the issue, and I know how to get past it.  Try
> > > > using the raw kernel Image (arch/arm64/boot/Image) instead of Image.gz
> > > > for dom0 and domU. That fixed it for me.
> > > > 
> > > > Julien, I didn't manage to figure out what the issue is exactly, but it
> > > > looks like Image.gz loading is broken at the moment.
> > > 
> > > Do you mean Image.gz is broken from DomU? Because per the log provided by
> > > Denis, this is working perfectly for Dom0 as we don't create domain in
> > > parallel.
> > > 
> > > By reading the code I can already spot the reason of the first issue
> > > reported
> > > by Denis. For reminder, this is when Dom0 and DomU are using the same
> > > module
> > > address for the gzip Image.
> > > 
> > > This is because when probing the kernel for Dom0, the module will get
> > > uncompressed and the module start/end will be updated to point to the
> > > uncompress version. Because of that, the probe for DomU kernel will not be
> > > able to find the module (the start addressed changed).
> > > 
> > > In this case, I think we only want to uncompress the module one time to
> > > avoid
> > > wasting memory. The solution I have in mind requires some rework in Xen, I
> > > would actually start by probing the information for all the domains, then
> > > uncompress the kernels modules, and then finish to build the domain.
> > > 
> > > For the out of memory problem discussed in this e-mail, I think the
> > > problem is
> > > not because of lack of memory in DomU. The problem is related to the
> > > inflate/gunzip the code. The code is using an heap (see perform_gunzip)
> > > where
> > > it allocates memory from.
> > > 
> > > I am assuming the kernels for Dom0 and DomU are exactly the same but they
> > > are
> > > coming from different address. Am I correct? If so, I am a bit unsure this
> > > worked the first time and not the second time. This probably want some
> > > debugging to understand the problem. Denis, Stefano, can one of you look
> > > at
> > > it?
> > 
> > I couldn't find exactly the root cause yet, but I can reproduce the
> > issue even with Dom0 only (no domUs, no dom0less):
> 
> Looking at Denis's report, the error does not seem to be the same:
> 
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Out of memory
> (XEN) ****************************************
> 
> 
> But I think they may be related (see below).
> 
> 
> > ee.
> > (XEN) *** LOADING DOMAIN 0 ***
> > (XEN) DEBUG kernel_probe 445
> > (XEN) Loading d0 kernel from boot module @ 0000000047000000
> > (XEN) Loading ramdisk from boot module @ 0000000042000000
> > (XEN) DEBUG kernel_decompress 268
> > (XEN) DEBUG kernel_decompress 272
> > (XEN) DEBUG kernel_decompress 279
> > (XEN) DEBUG kernel_decompress 284
> > (XEN) DEBUG kernel_decompress 291 kernel_order_out=52 output_size=0
> > (XEN)
> > (XEN) ****************************************
> > (XEN) Panic on CPU 0:
> > (XEN) Could not set up DOM0 guest OS
> > (XEN) ****************************************
> > (XEN)
> > (XEN) Reboot in five seconds...
> > 
> > 
> > The issue seems to be that output_size, returned by output_length(input,
> > size) is 0. Then, kernel_order_out is set to 52 which is too large. As a
> > consequence kernel_decompress returns with -ENOMEM.
> 
> I have just tried to use compressed kernel and can't reproduce your error.
> However, I think the two problems ("out of memory" and your one) are because
> the module size does not exactly match the size of the compressed image.
> 
> The uncompressed size is part of the footer (the last 4-bytes). As we only
> have the module size in hand, we assume it is equal to the compressed size. If
> not, then we will return whatever is in the last 4-bytes of the module.
> 
> This means the module size should exactly match the compressed image size.
> AFAICT, gzip format doesn't provide a field for the compressed size, so we
> can't do better in Xen.
> 
> In other word, the Device-Tree multiboot nodes should be created with the
> exact size of the compressed image.

Yes, you are right! That was the cause of the issue I was seeing.
Definitely something to watch out for!


> Regardless that, I still think we have some issues when using the same
> compressed kernel for Dom0 and DomU (see in my previous e-mail).

You wrote in the previous email:

> By reading the code I can already spot the reason of the first issue reported by Denis. For reminder, this is when Dom0 and DomU are using
> the same module address for the gzip Image.

By "module address", do you mean they use the same loading address in
u-boot? Because Denis was loading the Image.gz kernel twice at different
addresses for dom0 and domU: at 0x47000000 and at 0x43000000.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] How to boot domU and dom0 from a device tree

[Julien Grall]

On 17/06/2019 19:12, Stefano Stabellini wrote:
> On Sat, 15 Jun 2019, Julien Grall wrote:
>> On 6/14/19 9:53 PM, Stefano Stabellini wrote:
>>> On Wed, 12 Jun 2019, Julien Grall wrote:
>> Regardless that, I still think we have some issues when using the same
>> compressed kernel for Dom0 and DomU (see in my previous e-mail).
> 
> You wrote in the previous email:
> 
>> By reading the code I can already spot the reason of the first issue reported by Denis. For reminder, this is when Dom0 and DomU are using
>> the same module address for the gzip Image.
> 
> By "module address", do you mean they use the same loading address in
> u-boot? Because Denis was loading the Image.gz kernel twice at different
> addresses for dom0 and domU: at 0x47000000 and at 0x43000000.

Look at the very e-mail of this thread. He had dom0 and domU using the 
same address for the kernel.

Anyway, regardless what Denis reported, there are clearly an issue if 
Dom0 and Dom1 kernel are using the exact same loading address and the 
kernel is compressed. This would be fairly easy to reproduce.

Cheers,

-- 
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel