Re: [qemu-upstream-4.11-testing test] 136184: regressions - FAIL

[Julien Grall <julien.grall@arm.com>]

Hi,

Answering to myself.

On 5/17/19 8:00 PM, Julien Grall wrote:
> Hi,
> 
> On 5/17/19 6:23 PM, Anthony PERARD wrote:
>> On Thu, May 16, 2019 at 10:38:54PM +0100, Julien Grall wrote:
>>> Hi Anthony,
>>>
>>> Thank you for CCing me.
>>>
>>> On 5/16/19 11:37 AM, Anthony PERARD wrote:
>>>> On Wed, May 15, 2019 at 07:48:17PM +0000, osstest service owner wrote:
>>>>> flight 136184 qemu-upstream-4.11-testing real [real]
>>>>> http://logs.test-lab.xenproject.org/osstest/logs/136184/
>>>>>
>>>>> Regressions :-(
>>>>>
>>>>> Tests which did not succeed and are blocking,
>>>>> including tests which could not be run:
>>>>>    build-arm64-pvops               <job status>                 
>>>>> broken  in 134594
>>>>>    build-arm64                     <job status>                 
>>>>> broken  in 134594
>>>>>    build-arm64-xsm                 <job status>                 
>>>>> broken  in 134594
>>>>>    build-arm64-xsm            4 host-install(4) broken in 134594 
>>>>> REGR. vs. 125575
>>>>>    build-arm64-pvops          4 host-install(4) broken in 134594 
>>>>> REGR. vs. 125575
>>>>>    build-arm64                4 host-install(4) broken in 134594 
>>>>> REGR. vs. 125575
>>>>>    test-arm64-arm64-libvirt-xsm  7 xen-boot                 fail 
>>>>> REGR. vs. 125575
>>>>>    test-arm64-arm64-xl           7 xen-boot                 fail 
>>>>> REGR. vs. 125575
>>>>>    test-arm64-arm64-xl-xsm       7 xen-boot                 fail 
>>>>> REGR. vs. 125575
>>>>>    test-arm64-arm64-xl-credit2   7 xen-boot                 fail 
>>>>> REGR. vs. 125575
>>>>>
>>>>
>>>> Ian, Julien,
>>>>
>>>> I can't figure out why Xen consistently fails to boot on rochester* in
>>>> the qemu-upstream-4.11-testing flights. The xen-4.11-testing seems to
>>>> pass.
>>>>
>>>> At boot, the boot loader seems to load blobs, but when it's time to Xen
>>>> to shine, there are no output from Xen on the serial.
>>>
>>> The serial console is initializing fairly late in the process. Any 
>>> useful
>>> message (such as memory setup or even part of the interrupts) will be 
>>> hide
>>> out. For getting them, you need earlyprintk. Unfortunately they can't be
>>> configured at runtime today :(.
>>
>> I think I managed to run the job with earlyprintk on rochester, but
>> Xen have booted:
>> http://logs.test-lab.xenproject.org/osstest/logs/136451/
> 
> Yes this is with earlyprintk. That's going to be fun to reproduce if 
> earlyprintk modifies the behavior.
> 
> I think we can interpret as earlyprintk add enough latency to make 
> everything working.
> 
> There are two possible issues I can think of:
>      1) The boot code does not follow the Arm Arm, so it may be possible 
> the board is doing something different compare to the other regarding 
> the memory. IIRC, this is the first hardware we have with core not 
> directly designed by Arm.
>      2) We are missing some errata in Xen. Linux contains 6 errata for 
> that platform. Looking at them, I don't think they matter for boot time.
> 
> 1) is currently been looked at (see MM-PART* patches on the ML). 2) 
> should probably be addressed at some point, but I may not be able to 
> send them as Arm employee (we tend to avoid sending patch showing 
> brokenness in partner silicon).

Ian kindly started a couple of jobs over the week-end to confirm whether 
it can be reproduced on laxton* (Seattle board).

The same error cannot be reproduced on laxton*. Looking at the test 
history, it looks like qemu-upstream-4.12-testing flight has run 
successfully a few times on rochester*. So we may have fixed the error 
in Xen 4.12.

Potential candidates would be:
    - 00c96d7742 "xen/arm: mm: Set-up page permission for Xen mappings 
earlier on"
    - f60658c6ae "xen/arm: Stop relocating Xen"

Ian, is it something the bisector could automatically look at?
If not, I will need to find some time and borrow the board to bisect the 
issues.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel