User Interface issues with kernelci.org

User Interface issues with kernelci.org

[Suram Suram]

[-- Attachment #1.1: Type: text/plain, Size: 1314 bytes --]

Hi,

When investigating some i.MX Upstream kernel boot failures, our engineers have observed some issues with the User Interface.

Following are the details of the issue.  Requesting your help in improving the user interface.


i.MX Local Upstream daily test reported some upstream kernel boot failures.

I checked with kernelci has the same issue. However, the kernelci report seems not to be that clear.



>From the log below, you can see MX6Q Sabresd board boot hanged and result is “FAIL”

https://storage.kernelci.org/next/master/next-20200911/arm/imx_v6_v7_defconfig/gcc-8/lab-nxp/baseline-imx6q-sabresd.html

Boot log: imx6q-sabresd

Warnings: 0

Kernel Warnings: 0

Errors: 0

Boot result: FAIL

Kernel Errors: 0



But from the the kernelci main page below, it didn’t show those failures.

I guess they were categorized as ‘Unknown’ issues. However, even those failures were marked as “Unknown” issues, another problem is that users has no way to

Quickly filter out those ‘Unknown’ issues because the ‘Unknown’ button is grey.

Can you help check those two issues and report to kernelci if it can be improved?



https://kernelci.org/soc/imx/job/next/kernel/next-20200911/plan/baseline/

[cid:791025c9-deba-4e4f-a394-15ba7ee6abf6]



Regards

Aisheng


[-- Attachment #1.2: Type: text/html, Size: 7458 bytes --]

[-- Attachment #2: pastedImagebase640.png --]
[-- Type: image/png, Size: 206120 bytes --]

Re: User Interface issues with kernelci.org

[Guillaume Tucker]

On 18/09/2020 10:26, Suram Suram wrote:
> Hi,
> 
> When investigating some i.MX Upstream kernel boot failures, our engineers have observed some issues with the User Interface.
> 
> Following are the details of the issue.  Requesting your help in improving the user interface.
> 
> 
> i.MX Local Upstream daily test reported some upstream kernel boot failures.
> 
> I checked with kernelci has the same issue. However, the kernelci report seems not to be that clear.
> 
>  
> 
> From the log below, you can see MX6Q Sabresd board boot hanged and result is “FAIL”
> 
> https://storage.kernelci.org/next/master/next-20200911/arm/imx_v6_v7_defconfig/gcc-8/lab-nxp/baseline-imx6q-sabresd.html
> 
> Boot log: imx6q-sabresd
> 
> Warnings: 0
> 
> Kernel Warnings: 0
> 
> Errors: 0
> 
> Boot result: FAIL
> 
> Kernel Errors: 0
> 
>  
> 
> But from the the kernelci main page below, it didn’t show those failures.
> 
> I guess they were categorized as ‘Unknown’ issues. However, even those failures were marked as “Unknown” issues, another problem is that users has no way to
> 
> Quickly filter out those ‘Unknown’ issues because the ‘Unknown’ button is grey.
> 
> Can you help check those two issues and report to kernelci if it can be improved?
> 
>  
> 
> https://kernelci.org/soc/imx/job/next/kernel/next-20200911/plan/baseline/

Yes, this is a known issue with the web dashboard.  Currently, it
shows the following status:

* pass: test case passed
* regression: test case used to pass and started failing
* unknown: test case has always failed, or was skipped

So the "unknown" status is confusing, as you've found. For some
reason, this was not spotted as an issue when first deploying the
updated user interface to show test results.  It's now being
worked on to have 4 status:

* pass: test case passed
* fail: test case has always failed
* regression: test case used to pass and started failing
* unknown: test case was skipped or not run for some reason

This is the pull request implementing the changes described
above:

  https://github.com/kernelci/kernelci-frontend/pull/125

The work-in-progress version of this can be seen on:

  https://staging.kernelci.org

The pie charts haven't been updated yet, and there may be a few
small corner cases too.  This should all get resolved this week
and deployed in production next week, but please feel free to let
us know if it doesn't fix the issue you've found.

Best wishes,
Guillaume

Re: User Interface issues with kernelci.org

[Aisheng Dong]

> From: Guillaume Tucker <guillaume.tucker@collabora.com>
> Sent: Wednesday, September 23, 2020 2:21 PM
> On 18/09/2020 10:26, Suram Suram wrote:
> > Hi,
> >
> > When investigating some i.MX Upstream kernel boot failures, our engineers
> have observed some issues with the User Interface.
> >
> > Following are the details of the issue.  Requesting your help in improving the
> user interface.
> >
> >
> > i.MX Local Upstream daily test reported some upstream kernel boot failures.
> >
> > I checked with kernelci has the same issue. However, the kernelci report seems
> not to be that clear.
> >
> >
> >
> > From the log below, you can see MX6Q Sabresd board boot hanged and result
> is “FAIL”
> >
> > Boot log: imx6q-sabresd
> >
> > Warnings: 0
> >
> > Kernel Warnings: 0
> >
> > Errors: 0
> >
> > Boot result: FAIL
> >
> > Kernel Errors: 0
> >
> >
> >
> > But from the the kernelci main page below, it didn’t show those failures.
> >
> > I guess they were categorized as ‘Unknown’ issues. However, even those
> > failures were marked as “Unknown” issues, another problem is that
> > users has no way to
> >
> > Quickly filter out those ‘Unknown’ issues because the ‘Unknown’ button is
> grey.
> >
> > Can you help check those two issues and report to kernelci if it can be
> improved?
> >
> >
> 
> Yes, this is a known issue with the web dashboard.  Currently, it shows the
> following status:
> 
> * pass: test case passed
> * regression: test case used to pass and started failing
> * unknown: test case has always failed, or was skipped
> 
> So the "unknown" status is confusing, as you've found. For some reason, this
> was not spotted as an issue when first deploying the updated user interface to
> show test results.  It's now being worked on to have 4 status:
> 
> * pass: test case passed
> * fail: test case has always failed
> * regression: test case used to pass and started failing
> * unknown: test case was skipped or not run for some reason
> 
> This is the pull request implementing the changes described
> above:
...
> 
> 
> 
> The work-in-progress version of this can be seen on:
...
> 
> The pie charts haven't been updated yet, and there may be a few small corner
> cases too.  This should all get resolved this week and deployed in production
> next week, but please feel free to let us know if it doesn't fix the issue you've
> found.

Good to know the progress.
Thanks a lot for the feedback.

Regards
Aisheng

> 
> Best wishes,
> Guillaume

Re: User Interface issues with kernelci.org

[Guillaume Tucker]

On 23/09/2020 08:14, Aisheng Dong wrote:
>> From: Guillaume Tucker <guillaume.tucker@collabora.com>
>> Sent: Wednesday, September 23, 2020 2:21 PM
>> On 18/09/2020 10:26, Suram Suram wrote:
>>> Hi,
>>>
>>> When investigating some i.MX Upstream kernel boot failures, our engineers
>> have observed some issues with the User Interface.
>>>
>>> Following are the details of the issue.  Requesting your help in improving the
>> user interface.
>>>
>>>
>>> i.MX Local Upstream daily test reported some upstream kernel boot failures.
>>>
>>> I checked with kernelci has the same issue. However, the kernelci report seems
>> not to be that clear.
>>>
>>>
>>>
>>> From the log below, you can see MX6Q Sabresd board boot hanged and result
>> is “FAIL”
>>>
>>> Boot log: imx6q-sabresd
>>>
>>> Warnings: 0
>>>
>>> Kernel Warnings: 0
>>>
>>> Errors: 0
>>>
>>> Boot result: FAIL
>>>
>>> Kernel Errors: 0
>>>
>>>
>>>
>>> But from the the kernelci main page below, it didn’t show those failures.
>>>
>>> I guess they were categorized as ‘Unknown’ issues. However, even those
>>> failures were marked as “Unknown” issues, another problem is that
>>> users has no way to
>>>
>>> Quickly filter out those ‘Unknown’ issues because the ‘Unknown’ button is
>> grey.
>>>
>>> Can you help check those two issues and report to kernelci if it can be
>> improved?
>>>
>>>
>>
>> Yes, this is a known issue with the web dashboard.  Currently, it shows the
>> following status:
>>
>> * pass: test case passed
>> * regression: test case used to pass and started failing
>> * unknown: test case has always failed, or was skipped
>>
>> So the "unknown" status is confusing, as you've found. For some reason, this
>> was not spotted as an issue when first deploying the updated user interface to
>> show test results.  It's now being worked on to have 4 status:
>>
>> * pass: test case passed
>> * fail: test case has always failed
>> * regression: test case used to pass and started failing
>> * unknown: test case was skipped or not run for some reason
>>
>> This is the pull request implementing the changes described
>> above:
> ...
>>
>>
>>
>> The work-in-progress version of this can be seen on:
> ...
>>
>> The pie charts haven't been updated yet, and there may be a few small corner
>> cases too.  This should all get resolved this week and deployed in production
>> next week, but please feel free to let us know if it doesn't fix the issue you've
>> found.
> 
> Good to know the progress.
> Thanks a lot for the feedback.

This has now been fixed, for example see the latest results for
linux-next:

  https://kernelci.org/test/job/next/branch/master/kernel/next-20201023/plan/baseline/

There were a few failures in lab-nxp, and they're listed
as "failures".  Test cases that used to pass and started failing
are listed as "regressions", and tests that were not run or have
no clear pass/fail status are listed as "unknown".

Meanwhile, there are some known performance issues with the SoC
views.  It was improved in yesterday's update but it appears that
some views are still slow to load.  We'll be working on fixing
that very shortly, sorry for the inconvenience.

Best wishes,
Guillaume