Contribute to Kernel-CI with a new Lab

Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

Hi KernelCI,

A few months back I contacted you about adding a new lab to KernelCI infrastructure. It took us longer than I wished, but now we finally have been able to connect our farm to an internal KernelCI deployment (using docker).

As I explained before, our farm holds 4 boards with different SoCs from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained, Sama5d3_xplained and Sama5d4_xplained. All of then with mainline support.

Our setup, does not uses Lava and it relies on Labgrid to perform the tests. We use "kci_data" tool to publish test results on KernelCI.

We would like to work with you to be able to publish these results. Will it be possible to get a token for the staging database? I'm sure that there are things that need to be polished on our json files.


Best regards,

Santi


Santiago Esteban Zorita
Principal Engineer
---------------------------------------------------------------------
Microchip Technology Inc.

Torre Aragonia.

Juan Pablo II 35, 7ª planta.
50.009 Zaragoza, Spain


Desk: (+34) 976 010 300

santiago.esteban@microchip.com<mailto:santiago.valer@microchip.com>
www.microchip.com<http://www.microchip.com/>

Re: Contribute to Kernel-CI with a new Lab

[Jan Luebbe]

Hi Santi,

On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
> Hi KernelCI,
> 
> A few months back I contacted you about adding a new lab to KernelCI
> infrastructure. It took us longer than I wished, but now we finally
> have been able to connect our farm to an internal KernelCI deployment
> (using docker).
> 
> As I explained before, our farm holds 4 boards with different SoCs
> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
> support.
> 
> Our setup, does not uses Lava and it relies on Labgrid to perform the
> tests. We use "kci_data" tool to publish test results on KernelCI.

That's very interesting. :) Do you have published the interface code
somewhere? I've been trying to find some time to do that myself, but it
seems you've beaten me to it. ;)

Regards
Jan

> We would like to work with you to be able to publish these results.
> Will it be possible to get a token for the staging database? I'm sure
> that there are things that need to be polished on our json files.

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

[-- Attachment #1: Type: text/plain, Size: 2081 bytes --]

On 12/11/20 9:49, Jan Lübbe wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> Hi Santi,
>
> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>> Hi KernelCI,
>>
>> A few months back I contacted you about adding a new lab to KernelCI
>> infrastructure. It took us longer than I wished, but now we finally
>> have been able to connect our farm to an internal KernelCI deployment
>> (using docker).
>>
>> As I explained before, our farm holds 4 boards with different SoCs
>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>> support.
>>
>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>> tests. We use "kci_data" tool to publish test results on KernelCI.
> That's very interesting. :) Do you have published the interface code
> somewhere? I've been trying to find some time to do that myself, but it
> seems you've beaten me to it. ;)
>
> Regards
> Jan

Hi Jan,

No, I haven't publish it, till now. I've never though it would be useful 
to anybody but me  ;)

I have a python script that is tailored (too much) to our systems. It 
performs some actions that depend on our infrastructure and how I've 
implemented the labgrid tests (for example, I grab the results from a 
pytest  json report). At the end, it creates a "tmeta_<board>.json" file 
(equivalent to the "bmeta.json") that is later published with "kci_data" 
tool.

It could be used as an inspiration to make a more generic 
"kci_labgrid_test" tool if there are interest, but, all kernelci 
important stuff, still needs to be validated ;)

I have attached it to this email the script and (more important) an 
example of the output it produces.

BR,

Santi

>
>> We would like to work with you to be able to publish these results.
>> Will it be possible to get a token for the staging database? I'm sure
>> that there are things that need to be polished on our json files.
>


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mpuci-test-labgrid.py --]
[-- Type: text/x-python; name="mpuci-test-labgrid.py", Size: 7626 bytes --]

#!/usr/bin/python3

import sys
import os
import argparse
import hashlib
import subprocess
import time
import json
import socket

def labgrid_test(coord, board, image, test, test_json, bootrr):
    cmd = "labgrid-client -x {} reserve board={} --shell --wait".format(coord,board)    
    result = subprocess.check_output(cmd, stderr=subprocess.STDOUT,shell=True)
    token = result.decode("utf-8").strip().split("=")[-1] 
    
    cmd = "labgrid-client -x {} -p +{} lock".format(coord, token)
    result = subprocess.check_output(cmd, stderr=subprocess.STDOUT,shell=True)
    lab = result.decode("utf-8").strip().split()[-1]
    print("Labgrid resources locked ")
    
    #get lab name 
    lab = "remote-{}.yaml".format(lab)
    print("Execute test in lab {}".format(lab))
    
    currpwd = os.getcwd()

    if bootrr is not  None:
        bootrr = " --bootrr {} ".format(bootrr)
        

    os.chdir("labgrid-test/release/{}".format(board))
    os.system("pwd")
    os.system("rm console_main_ttyUSB0")
    start_test =time.time()
    try:
        cmd = "pytest -v -s --json-report --lg-coordinator {} --lg-log --lg-env {} {} --file {} {}".format(coord, lab, test, image, bootrr)
        print(cmd)
        sys.stdout.flush()
        ret= os.system(cmd)
    except:
        cmd = "labgrid-client -x {} -p +{} release".format(coord, token)
        print(cmd)
        ret =os.system(cmd)
        return 1, []


    if ret == 0:
        status = 'PASS'
    else:
        status = 'FAIL'
    end_test = time.time()
   
    if os.path.isfile("console_main_ttyUSB0"):
        logfile = open("console_main_ttyUSB0", "r", encoding='utf8',errors='ignore')
        test_json["log"] = logfile.read()
    else:
        print ("Can not find pytest console log")
    
    #Add baseline test results from json report
    testlist= ["test_login", "test_dmesg_alert", "test_dmesg_crit", "test_dmesg_err", "test_bootrr"]
    try:
        if os.path.isfile(".report.json"):
            with open(".report.json", "r") as report:
                data = json.load(report)
                for test in data['tests']:
                    if  "metadata" in test:
                        if  'result' in test['metadata']:
                            result = test["metadata"]['result']
                            result["job"] = test_json["job"]
                            result["time"] = test['call']['duration']
                            test_json["test_cases"].append(result) 
                        else:
                            result = { "name" : test["nodeid"] }
                            result = { "status" : "fail" }
                            test_json["test_cases"].append(result) 
                    else:                         
                        if errtest in testlist:
                            name= errtest.split('_')[-1]
                            result = { "name" : name, 'status' : 'fail' }
                        else:
                            continue
                                
                        test_json["test_cases"].append(result)                         
    except:
        print("Error parsing results from labgrid!")        
        ret=1, test_json
                               


    
    cmd = "labgrid-client -x {} -p +{} release".format(coord, token)
    os.system(cmd)
    os.chdir(currpwd) #restore path
    return ret,test_json
        

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description = 'Linux build test tool.')
    parser.add_argument('-p','--path',
                        required = False,
                        help ='Path to kernel to test', 
                        default ="../linux-at91/_install_/fit/")

    parser.add_argument('-k','--kernel',
                        required = False,
                        help ='Kernel image to test.', 
                        default ="sama5d2_xplained.itb")

    parser.add_argument('-b','--board',
                        required = True,
                        help ='Farm board to test with.', 
                        default ="sama5d2_xplained")
    
    parser.add_argument('-c','--coord',
                        required = False,
                        help ='Labgrid cooordinator"', 
                        default = "ws://mpuci-dev.mchp-main.com:20408/ws" )

    parser.add_argument('-t','--test',
                        required = False,
                        help ='Pytest file to execute"', 
                        default = "test_baseline.py" )

    parser.add_argument('-l','--lab',
                        required = False,
                        help ='KCI test lab', 
                        default = "lab-test" )
    
    parser.add_argument('--bootrr',
                        required = False,
                        help ='bootrr script to test', 
                        default = "" )

    #parse arguments
    args = parser.parse_args()


    #Retrieve build data
    sp = args.path.split("/")
    sp.pop(-1)
    buildpath = "/".join(sp) 
    sp.append("tmeta_{}.json".format(args.board))
    outfile = "/".join(sp)
    
    with open(buildpath + "/bmeta.json", 'r') as json_file:
        data = json.load(json_file)

    #Deploy to nfs/tftp shared folder
    image = args.path + args.kernel
    if os.path.isfile(image):
        imagepath = "/opt/tftpd/kernelci/{}/{}".format(socket.gethostname(),data["git_describe_v"])
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        imagepath = "{}/{}".format(imagepath, data["build_environment"])
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        imagepath = "{}/{}".format(imagepath, args.board)
        os.system("ssh labgrid@mpuci-dev.mchp-main.com mkdir -p {}".format(imagepath))

        print("Copy image {} to tftpd server {} -->  {}".format(image, "mpuci-dev", imagepath ))
        ret = os.system("scp {} labgrid@mpuci-dev.mchp-main.com:{}/".format(image,imagepath))
    else:
        print("Kernel Image file {} does not exist!".format(image))
        exit(1)

    if ret != 0:
        print("Error copying file to tftp server.")      
        exit(ret)

    #create test group dict.
    test_group = {
        "arch"  : data["arch"],
        "job"   : data["job"],
        "kernel": data["git_describe"],
        "defconfig": data["defconfig"],
        "defconfig_full": data["defconfig_full"],
        "build_environment": data["build_environment"],
        "git_branch": data["git_branch"],
        "git_commit": data["git_commit"],
        "lab_name": args.lab,
        "endian": "little",
        "mach"  : "at91",
        "board_instance" : "at91-"+args.board,
        "device_type" : "at91-"+args.board,
        "time": 0,
        "name": "baseline",
        "kernel_image": args.kernel,
        "file_server_resource": data["file_server_resource"],
        "test_cases" : []
    }

    start = time.time()
    #Run test
    kernel = imagepath +"/"+ args.kernel.split('/')[-1]
    result, test_group =labgrid_test(args.coord, args.board, kernel , args.test, test_group, args.bootrr )
    end = time.time()
    test_group["time"]= end - start
    
    if result == 0:        
        print("Write Json test result file") 
        with open(outfile, 'w') as wr_file:
            json.dump(test_group,wr_file, indent=4)
    else:
        print("Failed to execute test")
        exit(result)



[-- Attachment #3: tmeta_sama5d2_xplained.json --]
[-- Type: application/json, Size: 23587 bytes --]

Re: Contribute to Kernel-CI with a new Lab

[Guillaume Tucker]

On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
> On 12/11/20 9:49, Jan Lübbe wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> Hi Santi,
>>
>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>> Hi KernelCI,
>>>
>>> A few months back I contacted you about adding a new lab to KernelCI
>>> infrastructure. It took us longer than I wished, but now we finally
>>> have been able to connect our farm to an internal KernelCI deployment
>>> (using docker).
>>>
>>> As I explained before, our farm holds 4 boards with different SoCs
>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>> support.
>>>
>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>> That's very interesting. :) Do you have published the interface code
>> somewhere? I've been trying to find some time to do that myself, but it
>> seems you've beaten me to it. ;)
>>
>> Regards
>> Jan
> 
> Hi Jan,
> 
> No, I haven't publish it, till now. I've never though it would be useful 
> to anybody but me  ;)
> 
> I have a python script that is tailored (too much) to our systems. It 
> performs some actions that depend on our infrastructure and how I've 
> implemented the labgrid tests (for example, I grab the results from a 
> pytest  json report). At the end, it creates a "tmeta_<board>.json" file 
> (equivalent to the "bmeta.json") that is later published with "kci_data" 
> tool.
> 
> It could be used as an inspiration to make a more generic 
> "kci_labgrid_test" tool if there are interest, but, all kernelci 
> important stuff, still needs to be validated ;)

There is already a plugin mechanism behind kci_test to have
specific implementations for generating the test job description
data and submitting it to a remote lab.  At the moment, only LAVA
has an implementation but it sounds like Labgrid would be a neat
addition.  The main challenge I guess, is that there isn't any
built-in API for Labgrid to receive job descriptions and run
them.

One way to address this issue is to provide notifications with
jobs to run, and a lab using Labgrid would subscribe to it.  As
we're planning to refactor kernelci-backend and also come up with
a more generic pipeline mechanis than Jenkins, we should be able
to have this kind of notification system in place anyway in order
to corrdinate the different components of the pipeline in a
modular way.  It would be great to use some Labgrid labs as part
of this exercise.  If you're interested, we can discuss that when
we have some clear ideas about the main part of work.

> I have attached it to this email the script and (more important) an 
> example of the output it produces.

Thanks!  It's great to see that it doesn't really take much code
to get some KernelCI jobs to run in Labgrid.

> BR,
> 
> Santi
> 
>>
>>> We would like to work with you to be able to publish these results.
>>> Will it be possible to get a token for the staging database? I'm sure
>>> that there are things that need to be polished on our json files.

Absolutely, I'll send you a staging API token privately.  Also,
are you on IRC?  That would help with discussing things and
addressing any issues.

Thanks,
Guillaume

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 25/11/20 15:09, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>> On 12/11/20 9:49, Jan Lübbe wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> Hi Santi,
>>>
>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>> Hi KernelCI,
>>>>
>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>> infrastructure. It took us longer than I wished, but now we finally
>>>> have been able to connect our farm to an internal KernelCI deployment
>>>> (using docker).
>>>>
>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>> support.
>>>>
>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>> That's very interesting. :) Do you have published the interface code
>>> somewhere? I've been trying to find some time to do that myself, but it
>>> seems you've beaten me to it. ;)
>>>
>>> Regards
>>> Jan
>> Hi Jan,
>>
>> No, I haven't publish it, till now. I've never though it would be useful
>> to anybody but me  ;)
>>
>> I have a python script that is tailored (too much) to our systems. It
>> performs some actions that depend on our infrastructure and how I've
>> implemented the labgrid tests (for example, I grab the results from a
>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>> tool.
>>
>> It could be used as an inspiration to make a more generic
>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>> important stuff, still needs to be validated ;)
> There is already a plugin mechanism behind kci_test to have
> specific implementations for generating the test job description
> data and submitting it to a remote lab.  At the moment, only LAVA
> has an implementation but it sounds like Labgrid would be a neat
> addition.  The main challenge I guess, is that there isn't any
> built-in API for Labgrid to receive job descriptions and run
> them.

Hi Guillaume,

Labgrid does not perform any task related to build. It focuses on 
testing in real hardware. The build stage is performed somewhere else. 
It is a different approach than KernelCI and LAVA. In our case, we 
trigger the builds in Jenkins and also Jenkins executes the Labgrid 
pytest scripts. It would be great to be able to push KCI jobs into a 
generic Jenkins server. But, for me, the most difficult part would be 
convince IT department to allow a communication from a external source 
into the corporate network. I already know the answer....

> One way to address this issue is to provide notifications with
> jobs to run, and a lab using Labgrid would subscribe to it.  As
> we're planning to refactor kernelci-backend and also come up with
> a more generic pipeline mechanis than Jenkins, we should be able
> to have this kind of notification system in place anyway in order
> to corrdinate the different components of the pipeline in a
> modular way.  It would be great to use some Labgrid labs as part
> of this exercise.  If you're interested, we can discuss that when
> we have some clear ideas about the main part of work.
A publish/subscribe approach will work perfectly to overcome the 
corporate IT hurdles. Of course, I 'm interested. Please let me know how 
can I be part of this.
>> I have attached it to this email the script and (more important) an
>> example of the output it produces.
> Thanks!  It's great to see that it doesn't really take much code
> to get some KernelCI jobs to run in Labgrid.
Labgrid only cares about  3 things: image to test, board in the farm to 
use and test to pass. Most of the code is to create a json output test 
result compatible with the "kci_data" tool.
>> BR,
>>
>> Santi
>>
>>>> We would like to work with you to be able to publish these results.
>>>> Will it be possible to get a token for the staging database? I'm sure
>>>> that there are things that need to be polished on our json files.
> Absolutely, I'll send you a staging API token privately.  Also,
> are you on IRC?  That would help with discussing things and
> addressing any issues.

Great! I'm not ussually in IRC.... I'll be more available there (as 
"sesteban"). I'm waiting the token!

> Thanks,
> Guillaume

Best regards,

Santi

Re: Contribute to Kernel-CI with a new Lab

[Guillaume Tucker]

On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
> On 25/11/20 15:09, Guillaume Tucker wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> Hi Santi,
>>>>
>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>> Hi KernelCI,
>>>>>
>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>> (using docker).
>>>>>
>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>> support.
>>>>>
>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>> That's very interesting. :) Do you have published the interface code
>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>> seems you've beaten me to it. ;)
>>>>
>>>> Regards
>>>> Jan
>>> Hi Jan,
>>>
>>> No, I haven't publish it, till now. I've never though it would be useful
>>> to anybody but me  ;)
>>>
>>> I have a python script that is tailored (too much) to our systems. It
>>> performs some actions that depend on our infrastructure and how I've
>>> implemented the labgrid tests (for example, I grab the results from a
>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>> tool.
>>>
>>> It could be used as an inspiration to make a more generic
>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>> important stuff, still needs to be validated ;)
>> There is already a plugin mechanism behind kci_test to have
>> specific implementations for generating the test job description
>> data and submitting it to a remote lab.  At the moment, only LAVA
>> has an implementation but it sounds like Labgrid would be a neat
>> addition.  The main challenge I guess, is that there isn't any
>> built-in API for Labgrid to receive job descriptions and run
>> them.
> 
> Hi Guillaume,
> 
> Labgrid does not perform any task related to build. It focuses on 
> testing in real hardware. The build stage is performed somewhere else. 
> It is a different approach than KernelCI and LAVA. In our case, we 
> trigger the builds in Jenkins and also Jenkins executes the Labgrid 
> pytest scripts. It would be great to be able to push KCI jobs into a 
> generic Jenkins server. But, for me, the most difficult part would be 
> convince IT department to allow a communication from a external source 
> into the corporate network. I already know the answer....

Absolutely, that's the same as with LAVA.  The idea is that the
same builds from kernelci.org can be used in any lab, they're
just plain kernel builds.  What we could work on is to make it
easier for labs like yours and in particular when using Labgrid
to receive some data when there's a new test to run.  That would
include the URLs to get the kernel image, modules, dtb,
user-space, commands to run the tests etc...

>> One way to address this issue is to provide notifications with
>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>> we're planning to refactor kernelci-backend and also come up with
>> a more generic pipeline mechanis than Jenkins, we should be able
>> to have this kind of notification system in place anyway in order
>> to corrdinate the different components of the pipeline in a
>> modular way.  It would be great to use some Labgrid labs as part
>> of this exercise.  If you're interested, we can discuss that when
>> we have some clear ideas about the main part of work.
> A publish/subscribe approach will work perfectly to overcome the 
> corporate IT hurdles. Of course, I 'm interested. Please let me know how 
> can I be part of this.

Great.  There is a roadmap item on GitHub about this:

  https://github.com/kernelci/kernelci-core/issues/315

We've only just started, with a very basic kci_data
implementation.  This should be broken down into a series of
tasks such as: cleaning up the kernelci-backend API for receiving
results, providing a subscription mechanism and having a
real-world reference system to test it (e.g. a Labgrid lab).  We
probably should schedule a meeting to go through that in January.

>>> I have attached it to this email the script and (more important) an
>>> example of the output it produces.
>> Thanks!  It's great to see that it doesn't really take much code
>> to get some KernelCI jobs to run in Labgrid.
> Labgrid only cares about  3 things: image to test, board in the farm to 
> use and test to pass. Most of the code is to create a json output test 
> result compatible with the "kci_data" tool.

Exactly.

>>> BR,
>>>
>>> Santi
>>>
>>>>> We would like to work with you to be able to publish these results.
>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>> that there are things that need to be polished on our json files.
>> Absolutely, I'll send you a staging API token privately.  Also,
>> are you on IRC?  That would help with discussing things and
>> addressing any issues.
> 
> Great! I'm not ussually in IRC.... I'll be more available there (as 
> "sesteban"). I'm waiting the token!

Sent!  Awaiting some data now :)

Thanks,
Guillaume

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 26/11/20 10:17, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>
>>>>> Hi Santi,
>>>>>
>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>> Hi KernelCI,
>>>>>>
>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>> (using docker).
>>>>>>
>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>> support.
>>>>>>
>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>> That's very interesting. :) Do you have published the interface code
>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>> seems you've beaten me to it. ;)
>>>>>
>>>>> Regards
>>>>> Jan
>>>> Hi Jan,
>>>>
>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>> to anybody but me  ;)
>>>>
>>>> I have a python script that is tailored (too much) to our systems. It
>>>> performs some actions that depend on our infrastructure and how I've
>>>> implemented the labgrid tests (for example, I grab the results from a
>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>> tool.
>>>>
>>>> It could be used as an inspiration to make a more generic
>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>> important stuff, still needs to be validated ;)
>>> There is already a plugin mechanism behind kci_test to have
>>> specific implementations for generating the test job description
>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>> has an implementation but it sounds like Labgrid would be a neat
>>> addition.  The main challenge I guess, is that there isn't any
>>> built-in API for Labgrid to receive job descriptions and run
>>> them.
>> Hi Guillaume,
>>
>> Labgrid does not perform any task related to build. It focuses on
>> testing in real hardware. The build stage is performed somewhere else.
>> It is a different approach than KernelCI and LAVA. In our case, we
>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>> pytest scripts. It would be great to be able to push KCI jobs into a
>> generic Jenkins server. But, for me, the most difficult part would be
>> convince IT department to allow a communication from a external source
>> into the corporate network. I already know the answer....
> Absolutely, that's the same as with LAVA.  The idea is that the
> same builds from kernelci.org can be used in any lab, they're
> just plain kernel builds.  What we could work on is to make it
> easier for labs like yours and in particular when using Labgrid
> to receive some data when there's a new test to run.  That would
> include the URLs to get the kernel image, modules, dtb,
> user-space, commands to run the tests etc...
>
>>> One way to address this issue is to provide notifications with
>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>> we're planning to refactor kernelci-backend and also come up with
>>> a more generic pipeline mechanis than Jenkins, we should be able
>>> to have this kind of notification system in place anyway in order
>>> to corrdinate the different components of the pipeline in a
>>> modular way.  It would be great to use some Labgrid labs as part
>>> of this exercise.  If you're interested, we can discuss that when
>>> we have some clear ideas about the main part of work.
>> A publish/subscribe approach will work perfectly to overcome the
>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>> can I be part of this.
> Great.  There is a roadmap item on GitHub about this:
>
>    https://github.com/kernelci/kernelci-core/issues/315
>
> We've only just started, with a very basic kci_data
> implementation.  This should be broken down into a series of
> tasks such as: cleaning up the kernelci-backend API for receiving
> results, providing a subscription mechanism and having a
> real-world reference system to test it (e.g. a Labgrid lab).  We
> probably should schedule a meeting to go through that in January.
>
>>>> I have attached it to this email the script and (more important) an
>>>> example of the output it produces.
>>> Thanks!  It's great to see that it doesn't really take much code
>>> to get some KernelCI jobs to run in Labgrid.
>> Labgrid only cares about  3 things: image to test, board in the farm to
>> use and test to pass. Most of the code is to create a json output test
>> result compatible with the "kci_data" tool.
> Exactly.
>
>>>> BR,
>>>>
>>>> Santi
>>>>
>>>>>> We would like to work with you to be able to publish these results.
>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>> that there are things that need to be polished on our json files.
>>> Absolutely, I'll send you a staging API token privately.  Also,
>>> are you on IRC?  That would help with discussing things and
>>> addressing any issues.
>> Great! I'm not ussually in IRC.... I'll be more available there (as
>> "sesteban"). I'm waiting the token!
> Sent!  Awaiting some data now :)
>
> Thanks,
> Guillaume

I get to it right now!!


Santi

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 26/11/20 10:17, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>
>>>>> Hi Santi,
>>>>>
>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>> Hi KernelCI,
>>>>>>
>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>> (using docker).
>>>>>>
>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>> support.
>>>>>>
>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>> That's very interesting. :) Do you have published the interface code
>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>> seems you've beaten me to it. ;)
>>>>>
>>>>> Regards
>>>>> Jan
>>>> Hi Jan,
>>>>
>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>> to anybody but me  ;)
>>>>
>>>> I have a python script that is tailored (too much) to our systems. It
>>>> performs some actions that depend on our infrastructure and how I've
>>>> implemented the labgrid tests (for example, I grab the results from a
>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>> tool.
>>>>
>>>> It could be used as an inspiration to make a more generic
>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>> important stuff, still needs to be validated ;)
>>> There is already a plugin mechanism behind kci_test to have
>>> specific implementations for generating the test job description
>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>> has an implementation but it sounds like Labgrid would be a neat
>>> addition.  The main challenge I guess, is that there isn't any
>>> built-in API for Labgrid to receive job descriptions and run
>>> them.
>> Hi Guillaume,
>>
>> Labgrid does not perform any task related to build. It focuses on
>> testing in real hardware. The build stage is performed somewhere else.
>> It is a different approach than KernelCI and LAVA. In our case, we
>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>> pytest scripts. It would be great to be able to push KCI jobs into a
>> generic Jenkins server. But, for me, the most difficult part would be
>> convince IT department to allow a communication from a external source
>> into the corporate network. I already know the answer....
> Absolutely, that's the same as with LAVA.  The idea is that the
> same builds from kernelci.org can be used in any lab, they're
> just plain kernel builds.  What we could work on is to make it
> easier for labs like yours and in particular when using Labgrid
> to receive some data when there's a new test to run.  That would
> include the URLs to get the kernel image, modules, dtb,
> user-space, commands to run the tests etc...
>
>>> One way to address this issue is to provide notifications with
>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>> we're planning to refactor kernelci-backend and also come up with
>>> a more generic pipeline mechanis than Jenkins, we should be able
>>> to have this kind of notification system in place anyway in order
>>> to corrdinate the different components of the pipeline in a
>>> modular way.  It would be great to use some Labgrid labs as part
>>> of this exercise.  If you're interested, we can discuss that when
>>> we have some clear ideas about the main part of work.
>> A publish/subscribe approach will work perfectly to overcome the
>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>> can I be part of this.
> Great.  There is a roadmap item on GitHub about this:
>
>    https://github.com/kernelci/kernelci-core/issues/315
>
> We've only just started, with a very basic kci_data
> implementation.  This should be broken down into a series of
> tasks such as: cleaning up the kernelci-backend API for receiving
> results, providing a subscription mechanism and having a
> real-world reference system to test it (e.g. a Labgrid lab).  We
> probably should schedule a meeting to go through that in January.
>
>>>> I have attached it to this email the script and (more important) an
>>>> example of the output it produces.
>>> Thanks!  It's great to see that it doesn't really take much code
>>> to get some KernelCI jobs to run in Labgrid.
>> Labgrid only cares about  3 things: image to test, board in the farm to
>> use and test to pass. Most of the code is to create a json output test
>> result compatible with the "kci_data" tool.
> Exactly.
>
>>>> BR,
>>>>
>>>> Santi
>>>>
>>>>>> We would like to work with you to be able to publish these results.
>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>> that there are things that need to be polished on our json files.
>>> Absolutely, I'll send you a staging API token privately.  Also,
>>> are you on IRC?  That would help with discussing things and
>>> addressing any issues.
>> Great! I'm not ussually in IRC.... I'll be more available there (as
>> "sesteban"). I'm waiting the token!
> Sent!  Awaiting some data now :)

Hi Guillaume,

I've this credential/token error when trying to push a kernel build:

         requests.exceptions.HTTPError: 403 Client Error: Operation not 
permitted: provided token is not         authorized for url: 
https://api.staging.kernelci.org/upload
         Error: can not push kernel: ../builds/linux-mainline, 
https://api.staging.kernelci.org/test, "my-token"

Something misconfigured in the database?

>
> Thanks,
> Guillaume

Thanks,

Santi

Re: Contribute to Kernel-CI with a new Lab

[Guillaume Tucker]

On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
> On 26/11/20 10:17, Guillaume Tucker wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>
>>>>>> Hi Santi,
>>>>>>
>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>> Hi KernelCI,
>>>>>>>
>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>> (using docker).
>>>>>>>
>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>> support.
>>>>>>>
>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>> seems you've beaten me to it. ;)
>>>>>>
>>>>>> Regards
>>>>>> Jan
>>>>> Hi Jan,
>>>>>
>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>> to anybody but me  ;)
>>>>>
>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>> performs some actions that depend on our infrastructure and how I've
>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>> tool.
>>>>>
>>>>> It could be used as an inspiration to make a more generic
>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>> important stuff, still needs to be validated ;)
>>>> There is already a plugin mechanism behind kci_test to have
>>>> specific implementations for generating the test job description
>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>> has an implementation but it sounds like Labgrid would be a neat
>>>> addition.  The main challenge I guess, is that there isn't any
>>>> built-in API for Labgrid to receive job descriptions and run
>>>> them.
>>> Hi Guillaume,
>>>
>>> Labgrid does not perform any task related to build. It focuses on
>>> testing in real hardware. The build stage is performed somewhere else.
>>> It is a different approach than KernelCI and LAVA. In our case, we
>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>> generic Jenkins server. But, for me, the most difficult part would be
>>> convince IT department to allow a communication from a external source
>>> into the corporate network. I already know the answer....
>> Absolutely, that's the same as with LAVA.  The idea is that the
>> same builds from kernelci.org can be used in any lab, they're
>> just plain kernel builds.  What we could work on is to make it
>> easier for labs like yours and in particular when using Labgrid
>> to receive some data when there's a new test to run.  That would
>> include the URLs to get the kernel image, modules, dtb,
>> user-space, commands to run the tests etc...
>>
>>>> One way to address this issue is to provide notifications with
>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>> we're planning to refactor kernelci-backend and also come up with
>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>> to have this kind of notification system in place anyway in order
>>>> to corrdinate the different components of the pipeline in a
>>>> modular way.  It would be great to use some Labgrid labs as part
>>>> of this exercise.  If you're interested, we can discuss that when
>>>> we have some clear ideas about the main part of work.
>>> A publish/subscribe approach will work perfectly to overcome the
>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>> can I be part of this.
>> Great.  There is a roadmap item on GitHub about this:
>>
>>    https://github.com/kernelci/kernelci-core/issues/315
>>
>> We've only just started, with a very basic kci_data
>> implementation.  This should be broken down into a series of
>> tasks such as: cleaning up the kernelci-backend API for receiving
>> results, providing a subscription mechanism and having a
>> real-world reference system to test it (e.g. a Labgrid lab).  We
>> probably should schedule a meeting to go through that in January.
>>
>>>>> I have attached it to this email the script and (more important) an
>>>>> example of the output it produces.
>>>> Thanks!  It's great to see that it doesn't really take much code
>>>> to get some KernelCI jobs to run in Labgrid.
>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>> use and test to pass. Most of the code is to create a json output test
>>> result compatible with the "kci_data" tool.
>> Exactly.
>>
>>>>> BR,
>>>>>
>>>>> Santi
>>>>>
>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>> that there are things that need to be polished on our json files.
>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>> are you on IRC?  That would help with discussing things and
>>>> addressing any issues.
>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>> "sesteban"). I'm waiting the token!
>> Sent!  Awaiting some data now :)
> 
> Hi Guillaume,
> 
> I've this credential/token error when trying to push a kernel build:
> 
>          requests.exceptions.HTTPError: 403 Client Error: Operation not 
> permitted: provided token is not         authorized for url: 
> https://api.staging.kernelci.org/upload
>          Error: can not push kernel: ../builds/linux-mainline, 
> https://api.staging.kernelci.org/test, "my-token"
> 
> Something misconfigured in the database?

Yes, it looks like an oversight in the part of the code that adds
lab tokens as it doesn't enable the permission to send results
via the generic test API...  I've fixed that directly in the
database and tried your token with kci_data, it works now.

Also I've created this issue on GitHub to fix it properly:

  https://github.com/kernelci/kernelci-backend/issues/268


Let's hope this works for you, and please let us know how it goes
with the Labgrid integration.

Thanks,
Guillaume

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 3/12/20 15:01, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>
>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>
>>>>>>> Hi Santi,
>>>>>>>
>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>> Hi KernelCI,
>>>>>>>>
>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>> (using docker).
>>>>>>>>
>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>> support.
>>>>>>>>
>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>> seems you've beaten me to it. ;)
>>>>>>>
>>>>>>> Regards
>>>>>>> Jan
>>>>>> Hi Jan,
>>>>>>
>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>> to anybody but me  ;)
>>>>>>
>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>> tool.
>>>>>>
>>>>>> It could be used as an inspiration to make a more generic
>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>> important stuff, still needs to be validated ;)
>>>>> There is already a plugin mechanism behind kci_test to have
>>>>> specific implementations for generating the test job description
>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>> them.
>>>> Hi Guillaume,
>>>>
>>>> Labgrid does not perform any task related to build. It focuses on
>>>> testing in real hardware. The build stage is performed somewhere else.
>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>> convince IT department to allow a communication from a external source
>>>> into the corporate network. I already know the answer....
>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>> same builds from kernelci.org can be used in any lab, they're
>>> just plain kernel builds.  What we could work on is to make it
>>> easier for labs like yours and in particular when using Labgrid
>>> to receive some data when there's a new test to run.  That would
>>> include the URLs to get the kernel image, modules, dtb,
>>> user-space, commands to run the tests etc...
>>>
>>>>> One way to address this issue is to provide notifications with
>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>> to have this kind of notification system in place anyway in order
>>>>> to corrdinate the different components of the pipeline in a
>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>> we have some clear ideas about the main part of work.
>>>> A publish/subscribe approach will work perfectly to overcome the
>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>> can I be part of this.
>>> Great.  There is a roadmap item on GitHub about this:
>>>
>>>     https://github.com/kernelci/kernelci-core/issues/315
>>>
>>> We've only just started, with a very basic kci_data
>>> implementation.  This should be broken down into a series of
>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>> results, providing a subscription mechanism and having a
>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>> probably should schedule a meeting to go through that in January.
>>>
>>>>>> I have attached it to this email the script and (more important) an
>>>>>> example of the output it produces.
>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>> to get some KernelCI jobs to run in Labgrid.
>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>> use and test to pass. Most of the code is to create a json output test
>>>> result compatible with the "kci_data" tool.
>>> Exactly.
>>>
>>>>>> BR,
>>>>>>
>>>>>> Santi
>>>>>>
>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>> that there are things that need to be polished on our json files.
>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>> are you on IRC?  That would help with discussing things and
>>>>> addressing any issues.
>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>> "sesteban"). I'm waiting the token!
>>> Sent!  Awaiting some data now :)
>> Hi Guillaume,
>>
>> I've this credential/token error when trying to push a kernel build:
>>
>>           requests.exceptions.HTTPError: 403 Client Error: Operation not
>> permitted: provided token is not         authorized for url:
>> https://api.staging.kernelci.org/upload
>>           Error: can not push kernel: ../builds/linux-mainline,
>> https://api.staging.kernelci.org/test, "my-token"
>>
>> Something misconfigured in the database?
> Yes, it looks like an oversight in the part of the code that adds
> lab tokens as it doesn't enable the permission to send results
> via the generic test API...  I've fixed that directly in the
> database and tried your token with kci_data, it works now.
>
> Also I've created this issue on GitHub to fix it properly:
>
>    https://github.com/kernelci/kernelci-backend/issues/268
>
>
> Let's hope this works for you, and please let us know how it goes
> with the Labgrid integration.

Hi,

I think I had the same problem with my local instance of KernelCI. I had 
to manually update the properties for the lab. At the time, I thought 
that it was not a bug and that somehow LAVA could do it right.

Unfortunatelly, the problem persists.

Traceback (most recent call last):
   File "./kci_build", line 353, in <module>
     status = opts.command(configs, opts)
   File "./kci_build", line 316, in __call__
     args.install_path)
   File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890, 
in push_kernel
     upload_files(api, token, upload_path, artifacts)
   File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line 
44, in upload_files
     resp.raise_for_status()requests.exceptions.HTTPError: 403 Client 
Error: Operation not permitted: provided token is not authorized for 
url: https://api.staging.kernelci.org/upload
         Error: can not push kernel: ../builds/linux-mainline, 
https://api.staging.kernelci.org/test, "my token"

If you tested the token locally and it works, it might be a problem 
related to the IP address allowed to push data in the lab...

> Thanks,
> Guillaume

Thanks to you!

Re: Contribute to Kernel-CI with a new Lab

[Guillaume Tucker]

On 04/12/2020 11:46, Santiago.Esteban@microchip.com wrote:
> On 3/12/20 15:01, Guillaume Tucker wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>
>>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>
>>>>>>>> Hi Santi,
>>>>>>>>
>>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>> Hi KernelCI,
>>>>>>>>>
>>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>>> (using docker).
>>>>>>>>>
>>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>>> support.
>>>>>>>>>
>>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>>> seems you've beaten me to it. ;)
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Jan
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>>> to anybody but me  ;)
>>>>>>>
>>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>>> tool.
>>>>>>>
>>>>>>> It could be used as an inspiration to make a more generic
>>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>>> important stuff, still needs to be validated ;)
>>>>>> There is already a plugin mechanism behind kci_test to have
>>>>>> specific implementations for generating the test job description
>>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>>> them.
>>>>> Hi Guillaume,
>>>>>
>>>>> Labgrid does not perform any task related to build. It focuses on
>>>>> testing in real hardware. The build stage is performed somewhere else.
>>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>>> convince IT department to allow a communication from a external source
>>>>> into the corporate network. I already know the answer....
>>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>>> same builds from kernelci.org can be used in any lab, they're
>>>> just plain kernel builds.  What we could work on is to make it
>>>> easier for labs like yours and in particular when using Labgrid
>>>> to receive some data when there's a new test to run.  That would
>>>> include the URLs to get the kernel image, modules, dtb,
>>>> user-space, commands to run the tests etc...
>>>>
>>>>>> One way to address this issue is to provide notifications with
>>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>>> to have this kind of notification system in place anyway in order
>>>>>> to corrdinate the different components of the pipeline in a
>>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>>> we have some clear ideas about the main part of work.
>>>>> A publish/subscribe approach will work perfectly to overcome the
>>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>>> can I be part of this.
>>>> Great.  There is a roadmap item on GitHub about this:
>>>>
>>>>     https://github.com/kernelci/kernelci-core/issues/315
>>>>
>>>> We've only just started, with a very basic kci_data
>>>> implementation.  This should be broken down into a series of
>>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>>> results, providing a subscription mechanism and having a
>>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>>> probably should schedule a meeting to go through that in January.
>>>>
>>>>>>> I have attached it to this email the script and (more important) an
>>>>>>> example of the output it produces.
>>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>>> to get some KernelCI jobs to run in Labgrid.
>>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>>> use and test to pass. Most of the code is to create a json output test
>>>>> result compatible with the "kci_data" tool.
>>>> Exactly.
>>>>
>>>>>>> BR,
>>>>>>>
>>>>>>> Santi
>>>>>>>
>>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>>> that there are things that need to be polished on our json files.
>>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>>> are you on IRC?  That would help with discussing things and
>>>>>> addressing any issues.
>>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>>> "sesteban"). I'm waiting the token!
>>>> Sent!  Awaiting some data now :)
>>> Hi Guillaume,
>>>
>>> I've this credential/token error when trying to push a kernel build:
>>>
>>>           requests.exceptions.HTTPError: 403 Client Error: Operation not
>>> permitted: provided token is not         authorized for url:
>>> https://api.staging.kernelci.org/upload
>>>           Error: can not push kernel: ../builds/linux-mainline,
>>> https://api.staging.kernelci.org/test, "my-token"
>>>
>>> Something misconfigured in the database?
>> Yes, it looks like an oversight in the part of the code that adds
>> lab tokens as it doesn't enable the permission to send results
>> via the generic test API...  I've fixed that directly in the
>> database and tried your token with kci_data, it works now.
>>
>> Also I've created this issue on GitHub to fix it properly:
>>
>>    https://github.com/kernelci/kernelci-backend/issues/268
>>
>>
>> Let's hope this works for you, and please let us know how it goes
>> with the Labgrid integration.
> 
> Hi,
> 
> I think I had the same problem with my local instance of KernelCI. I had 
> to manually update the properties for the lab. At the time, I thought 
> that it was not a bug and that somehow LAVA could do it right.
> 
> Unfortunatelly, the problem persists.
> 
> Traceback (most recent call last):
>    File "./kci_build", line 353, in <module>
>      status = opts.command(configs, opts)
>    File "./kci_build", line 316, in __call__
>      args.install_path)
>    File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890, 
> in push_kernel
>      upload_files(api, token, upload_path, artifacts)
>    File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line 
> 44, in upload_files
>      resp.raise_for_status()requests.exceptions.HTTPError: 403 Client 
> Error: Operation not permitted: provided token is not authorized for 
> url: https://api.staging.kernelci.org/upload
>          Error: can not push kernel: ../builds/linux-mainline, 
> https://api.staging.kernelci.org/test, "my token"
> 
> If you tested the token locally and it works, it might be a problem 
> related to the IP address allowed to push data in the lab...

The token works, but it's a "lab" token with permissions to
submit test results.  It's not for submitting kernel builds,
which is what you're trying to do here.

Can you try using builds from storage.kernelci.org?  They come
with a bmeta.json file to get the kernel revision and all related
meta-data you need when submitting the test results for a given
kernel build.

If you want to test some kernel revisions not currently built on
kernelci.org you can open a GitHub issue to request it:

  https://github.com/kernelci/kernelci-core/issues/new?assignees=&labels=&template=new-kernel-branch.md&title=Add+branch+BRANCH+from+TREE

Thanks,
Guillaume

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

[-- Attachment #1: Type: text/plain, Size: 11433 bytes --]

On 7/12/20 10:08, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 04/12/2020 11:46, Santiago.Esteban@microchip.com wrote:
>> On 3/12/20 15:01, Guillaume Tucker wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>>>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>
>>>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>
>>>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>
>>>>>>>>> Hi Santi,
>>>>>>>>>
>>>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>> Hi KernelCI,
>>>>>>>>>>
>>>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>>>> (using docker).
>>>>>>>>>>
>>>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>>>> support.
>>>>>>>>>>
>>>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>>>> seems you've beaten me to it. ;)
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Jan
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>>>> to anybody but me  ;)
>>>>>>>>
>>>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>>>> tool.
>>>>>>>>
>>>>>>>> It could be used as an inspiration to make a more generic
>>>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>>>> important stuff, still needs to be validated ;)
>>>>>>> There is already a plugin mechanism behind kci_test to have
>>>>>>> specific implementations for generating the test job description
>>>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>>>> them.
>>>>>> Hi Guillaume,
>>>>>>
>>>>>> Labgrid does not perform any task related to build. It focuses on
>>>>>> testing in real hardware. The build stage is performed somewhere else.
>>>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>>>> convince IT department to allow a communication from a external source
>>>>>> into the corporate network. I already know the answer....
>>>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>>>> same builds from kernelci.org can be used in any lab, they're
>>>>> just plain kernel builds.  What we could work on is to make it
>>>>> easier for labs like yours and in particular when using Labgrid
>>>>> to receive some data when there's a new test to run.  That would
>>>>> include the URLs to get the kernel image, modules, dtb,
>>>>> user-space, commands to run the tests etc...
>>>>>
>>>>>>> One way to address this issue is to provide notifications with
>>>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>>>> to have this kind of notification system in place anyway in order
>>>>>>> to corrdinate the different components of the pipeline in a
>>>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>>>> we have some clear ideas about the main part of work.
>>>>>> A publish/subscribe approach will work perfectly to overcome the
>>>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>>>> can I be part of this.
>>>>> Great.  There is a roadmap item on GitHub about this:
>>>>>
>>>>>      https://github.com/kernelci/kernelci-core/issues/315
>>>>>
>>>>> We've only just started, with a very basic kci_data
>>>>> implementation.  This should be broken down into a series of
>>>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>>>> results, providing a subscription mechanism and having a
>>>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>>>> probably should schedule a meeting to go through that in January.
>>>>>
>>>>>>>> I have attached it to this email the script and (more important) an
>>>>>>>> example of the output it produces.
>>>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>>>> to get some KernelCI jobs to run in Labgrid.
>>>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>>>> use and test to pass. Most of the code is to create a json output test
>>>>>> result compatible with the "kci_data" tool.
>>>>> Exactly.
>>>>>
>>>>>>>> BR,
>>>>>>>>
>>>>>>>> Santi
>>>>>>>>
>>>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>>>> that there are things that need to be polished on our json files.
>>>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>>>> are you on IRC?  That would help with discussing things and
>>>>>>> addressing any issues.
>>>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>>>> "sesteban"). I'm waiting the token!
>>>>> Sent!  Awaiting some data now :)
>>>> Hi Guillaume,
>>>>
>>>> I've this credential/token error when trying to push a kernel build:
>>>>
>>>>            requests.exceptions.HTTPError: 403 Client Error: Operation not
>>>> permitted: provided token is not         authorized for url:
>>>> https://api.staging.kernelci.org/upload
>>>>            Error: can not push kernel: ../builds/linux-mainline,
>>>> https://api.staging.kernelci.org/test, "my-token"
>>>>
>>>> Something misconfigured in the database?
>>> Yes, it looks like an oversight in the part of the code that adds
>>> lab tokens as it doesn't enable the permission to send results
>>> via the generic test API...  I've fixed that directly in the
>>> database and tried your token with kci_data, it works now.
>>>
>>> Also I've created this issue on GitHub to fix it properly:
>>>
>>>     https://github.com/kernelci/kernelci-backend/issues/268
>>>
>>>
>>> Let's hope this works for you, and please let us know how it goes
>>> with the Labgrid integration.
>> Hi,
>>
>> I think I had the same problem with my local instance of KernelCI. I had
>> to manually update the properties for the lab. At the time, I thought
>> that it was not a bug and that somehow LAVA could do it right.
>>
>> Unfortunatelly, the problem persists.
>>
>> Traceback (most recent call last):
>>     File "./kci_build", line 353, in <module>
>>       status = opts.command(configs, opts)
>>     File "./kci_build", line 316, in __call__
>>       args.install_path)
>>     File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890,
>> in push_kernel
>>       upload_files(api, token, upload_path, artifacts)
>>     File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line
>> 44, in upload_files
>>       resp.raise_for_status()requests.exceptions.HTTPError: 403 Client
>> Error: Operation not permitted: provided token is not authorized for
>> url: https://api.staging.kernelci.org/upload
>>           Error: can not push kernel: ../builds/linux-mainline,
>> https://api.staging.kernelci.org/test, "my token"
>>
>> If you tested the token locally and it works, it might be a problem
>> related to the IP address allowed to push data in the lab...
> The token works, but it's a "lab" token with permissions to
> submit test results.  It's not for submitting kernel builds,
> which is what you're trying to do here.
>
> Can you try using builds from storage.kernelci.org?  They come
> with a bmeta.json file to get the kernel revision and all related
> meta-data you need when submitting the test results for a given
> kernel build.
>
> If you want to test some kernel revisions not currently built on
> kernelci.org you can open a GitHub issue to request it:
>
>    https://github.com/kernelci/kernelci-core/issues/new?assignees=&labels=&template=new-kernel-branch.md&title=Add+branch+BRANCH+from+TREE
>
> Thanks,
> Guillaume

Hi Guillaume,

I have finally push some results in the staging database. I'm sorry that 
took me so long, but we just had a bank holiday here in Spain :)

https://staging.kernelci.org/soc/at91/job/mainline/kernel/v5.10-rc7-10-ga68a0262abda/plan/baseline/

At first glance looks ok, but if you click on the "Full results" 
(https://staging.kernelci.org/test/plan/id/5fd21e6c81bee0f1bf417ce4/) 
the page is broken... Well,  the results received from the database are 
not complete. I checked the javascript in view-tests-plan-id.2020.10.js 
and I found that  there are missing fields like "build_id". It seems 
that not all the build information are available to all  "defconfigs" at 
the staging database. The /test/plan/id page works fine in my local 
kernelci deployment.

I chose the build manually, because the token that you provide cannot 
access the https://api.staging.kernelci.org/build information.

While the issue #315 is solved, could we find a easy way to publish 
results?I could poll the backend to check for new builds, do the tests, 
and finally publish them.

As how to solve #315 for good, I would suggest using Autobahn 
(https://autobahn.readthedocs.io/en/latest/). Labgrid is build around it 
and works like a charm. I'm no expert, maybe Pengutronix people can 
share their experience and recommend this path or another.


Best regards,

Santi

p.s I have attached the json file that I have pushed into the staging 
database using kci_data, in case you need to look at it.


[-- Attachment #2: tmeta_sam9x60ek.json --]
[-- Type: application/json, Size: 29105 bytes --]

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 11/12/20 19:40, Santiago Esteban - M44029 wrote:
> On 7/12/20 10:08, Guillaume Tucker wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> On 04/12/2020 11:46, Santiago.Esteban@microchip.com wrote:
>>> On 3/12/20 15:01, Guillaume Tucker wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>>>>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>
>>>>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>
>>>>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>>
>>>>>>>>>> Hi Santi,
>>>>>>>>>>
>>>>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>>> Hi KernelCI,
>>>>>>>>>>>
>>>>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>>>>> (using docker).
>>>>>>>>>>>
>>>>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>>>>> support.
>>>>>>>>>>>
>>>>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>>>>> seems you've beaten me to it. ;)
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Jan
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>>>>> to anybody but me  ;)
>>>>>>>>>
>>>>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>>>>> tool.
>>>>>>>>>
>>>>>>>>> It could be used as an inspiration to make a more generic
>>>>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>>>>> important stuff, still needs to be validated ;)
>>>>>>>> There is already a plugin mechanism behind kci_test to have
>>>>>>>> specific implementations for generating the test job description
>>>>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>>>>> them.
>>>>>>> Hi Guillaume,
>>>>>>>
>>>>>>> Labgrid does not perform any task related to build. It focuses on
>>>>>>> testing in real hardware. The build stage is performed somewhere else.
>>>>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>>>>> convince IT department to allow a communication from a external source
>>>>>>> into the corporate network. I already know the answer....
>>>>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>>>>> same builds from kernelci.org can be used in any lab, they're
>>>>>> just plain kernel builds.  What we could work on is to make it
>>>>>> easier for labs like yours and in particular when using Labgrid
>>>>>> to receive some data when there's a new test to run.  That would
>>>>>> include the URLs to get the kernel image, modules, dtb,
>>>>>> user-space, commands to run the tests etc...
>>>>>>
>>>>>>>> One way to address this issue is to provide notifications with
>>>>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>>>>> to have this kind of notification system in place anyway in order
>>>>>>>> to corrdinate the different components of the pipeline in a
>>>>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>>>>> we have some clear ideas about the main part of work.
>>>>>>> A publish/subscribe approach will work perfectly to overcome the
>>>>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>>>>> can I be part of this.
>>>>>> Great.  There is a roadmap item on GitHub about this:
>>>>>>
>>>>>>       https://github.com/kernelci/kernelci-core/issues/315
>>>>>>
>>>>>> We've only just started, with a very basic kci_data
>>>>>> implementation.  This should be broken down into a series of
>>>>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>>>>> results, providing a subscription mechanism and having a
>>>>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>>>>> probably should schedule a meeting to go through that in January.
>>>>>>
>>>>>>>>> I have attached it to this email the script and (more important) an
>>>>>>>>> example of the output it produces.
>>>>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>>>>> to get some KernelCI jobs to run in Labgrid.
>>>>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>>>>> use and test to pass. Most of the code is to create a json output test
>>>>>>> result compatible with the "kci_data" tool.
>>>>>> Exactly.
>>>>>>
>>>>>>>>> BR,
>>>>>>>>>
>>>>>>>>> Santi
>>>>>>>>>
>>>>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>>>>> that there are things that need to be polished on our json files.
>>>>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>>>>> are you on IRC?  That would help with discussing things and
>>>>>>>> addressing any issues.
>>>>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>>>>> "sesteban"). I'm waiting the token!
>>>>>> Sent!  Awaiting some data now :)
>>>>> Hi Guillaume,
>>>>>
>>>>> I've this credential/token error when trying to push a kernel build:
>>>>>
>>>>>             requests.exceptions.HTTPError: 403 Client Error: Operation not
>>>>> permitted: provided token is not         authorized for url:
>>>>> https://api.staging.kernelci.org/upload
>>>>>             Error: can not push kernel: ../builds/linux-mainline,
>>>>> https://api.staging.kernelci.org/test, "my-token"
>>>>>
>>>>> Something misconfigured in the database?
>>>> Yes, it looks like an oversight in the part of the code that adds
>>>> lab tokens as it doesn't enable the permission to send results
>>>> via the generic test API...  I've fixed that directly in the
>>>> database and tried your token with kci_data, it works now.
>>>>
>>>> Also I've created this issue on GitHub to fix it properly:
>>>>
>>>>      https://github.com/kernelci/kernelci-backend/issues/268
>>>>
>>>>
>>>> Let's hope this works for you, and please let us know how it goes
>>>> with the Labgrid integration.
>>> Hi,
>>>
>>> I think I had the same problem with my local instance of KernelCI. I had
>>> to manually update the properties for the lab. At the time, I thought
>>> that it was not a bug and that somehow LAVA could do it right.
>>>
>>> Unfortunatelly, the problem persists.
>>>
>>> Traceback (most recent call last):
>>>      File "./kci_build", line 353, in <module>
>>>        status = opts.command(configs, opts)
>>>      File "./kci_build", line 316, in __call__
>>>        args.install_path)
>>>      File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890,
>>> in push_kernel
>>>        upload_files(api, token, upload_path, artifacts)
>>>      File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line
>>> 44, in upload_files
>>>        resp.raise_for_status()requests.exceptions.HTTPError: 403 Client
>>> Error: Operation not permitted: provided token is not authorized for
>>> url: https://api.staging.kernelci.org/upload
>>>            Error: can not push kernel: ../builds/linux-mainline,
>>> https://api.staging.kernelci.org/test, "my token"
>>>
>>> If you tested the token locally and it works, it might be a problem
>>> related to the IP address allowed to push data in the lab...
>> The token works, but it's a "lab" token with permissions to
>> submit test results.  It's not for submitting kernel builds,
>> which is what you're trying to do here.
>>
>> Can you try using builds from storage.kernelci.org?  They come
>> with a bmeta.json file to get the kernel revision and all related
>> meta-data you need when submitting the test results for a given
>> kernel build.
>>
>> If you want to test some kernel revisions not currently built on
>> kernelci.org you can open a GitHub issue to request it:
>>
>>     https://github.com/kernelci/kernelci-core/issues/new?assignees=&labels=&template=new-kernel-branch.md&title=Add+branch+BRANCH+from+TREE
>>
>> Thanks,
>> Guillaume
> Hi Guillaume,
>
> I have finally push some results in the staging database. I'm sorry that
> took me so long, but we just had a bank holiday here in Spain :)
>
> https://staging.kernelci.org/soc/at91/job/mainline/kernel/v5.10-rc7-10-ga68a0262abda/plan/baseline/
>
> At first glance looks ok, but if you click on the "Full results"
> (https://staging.kernelci.org/test/plan/id/5fd21e6c81bee0f1bf417ce4/)
> the page is broken... Well,  the results received from the database are
> not complete. I checked the javascript in view-tests-plan-id.2020.10.js
> and I found that  there are missing fields like "build_id". It seems
> that not all the build information are available to all  "defconfigs" at
> the staging database. The /test/plan/id page works fine in my local
> kernelci deployment.
>
> I chose the build manually, because the token that you provide cannot
> access the https://api.staging.kernelci.org/build information.
>
> While the issue #315 is solved, could we find a easy way to publish
> results?I could poll the backend to check for new builds, do the tests,
> and finally publish them.
>
> As how to solve #315 for good, I would suggest using Autobahn
> (https://autobahn.readthedocs.io/en/latest/). Labgrid is build around it
> and works like a charm. I'm no expert, maybe Pengutronix people can
> share their experience and recommend this path or another.
>
>
> Best regards,
>
> Santi
>
> p.s I have attached the json file that I have pushed into the staging
> database using kci_data, in case you need to look at it.

Hi Guillaume,

About pushing data into staging... Could you add some kernel builds with 
"sama5_defconfig" and "at91_dt_defconfig" configurations for 
mainline/master(or any other tree/branch )? I have not found those 
configurations in the staging database, I would like to test this 
thoroughly.

In a (kind of) different topic, what is the status of Generic Lab 
Support #315? Are there any plans to implement this soon? I could 
dedicate time to push forward this task, but definitively, I would need 
some steering to avoid wasting both my time and your time.

Best regards,


Santi Esteban

Re: Contribute to Kernel-CI with a new Lab

[Guillaume Tucker]

On 27/01/2021 12:11, Santiago.Esteban@microchip.com wrote:
> On 11/12/20 19:40, Santiago Esteban - M44029 wrote:
>> On 7/12/20 10:08, Guillaume Tucker wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On 04/12/2020 11:46, Santiago.Esteban@microchip.com wrote:
>>>> On 3/12/20 15:01, Guillaume Tucker wrote:
>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>
>>>>> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>>>>>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>
>>>>>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>>>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>
>>>>>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>>>
>>>>>>>>>>> Hi Santi,
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>>>> Hi KernelCI,
>>>>>>>>>>>>
>>>>>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>>>>>> (using docker).
>>>>>>>>>>>>
>>>>>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>>>>>> support.
>>>>>>>>>>>>
>>>>>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>>>>>> seems you've beaten me to it. ;)
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Jan
>>>>>>>>>> Hi Jan,
>>>>>>>>>>
>>>>>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>>>>>> to anybody but me  ;)
>>>>>>>>>>
>>>>>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>>>>>> tool.
>>>>>>>>>>
>>>>>>>>>> It could be used as an inspiration to make a more generic
>>>>>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>>>>>> important stuff, still needs to be validated ;)
>>>>>>>>> There is already a plugin mechanism behind kci_test to have
>>>>>>>>> specific implementations for generating the test job description
>>>>>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>>>>>> them.
>>>>>>>> Hi Guillaume,
>>>>>>>>
>>>>>>>> Labgrid does not perform any task related to build. It focuses on
>>>>>>>> testing in real hardware. The build stage is performed somewhere else.
>>>>>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>>>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>>>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>>>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>>>>>> convince IT department to allow a communication from a external source
>>>>>>>> into the corporate network. I already know the answer....
>>>>>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>>>>>> same builds from kernelci.org can be used in any lab, they're
>>>>>>> just plain kernel builds.  What we could work on is to make it
>>>>>>> easier for labs like yours and in particular when using Labgrid
>>>>>>> to receive some data when there's a new test to run.  That would
>>>>>>> include the URLs to get the kernel image, modules, dtb,
>>>>>>> user-space, commands to run the tests etc...
>>>>>>>
>>>>>>>>> One way to address this issue is to provide notifications with
>>>>>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>>>>>> to have this kind of notification system in place anyway in order
>>>>>>>>> to corrdinate the different components of the pipeline in a
>>>>>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>>>>>> we have some clear ideas about the main part of work.
>>>>>>>> A publish/subscribe approach will work perfectly to overcome the
>>>>>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>>>>>> can I be part of this.
>>>>>>> Great.  There is a roadmap item on GitHub about this:
>>>>>>>
>>>>>>>       https://github.com/kernelci/kernelci-core/issues/315
>>>>>>>
>>>>>>> We've only just started, with a very basic kci_data
>>>>>>> implementation.  This should be broken down into a series of
>>>>>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>>>>>> results, providing a subscription mechanism and having a
>>>>>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>>>>>> probably should schedule a meeting to go through that in January.
>>>>>>>
>>>>>>>>>> I have attached it to this email the script and (more important) an
>>>>>>>>>> example of the output it produces.
>>>>>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>>>>>> to get some KernelCI jobs to run in Labgrid.
>>>>>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>>>>>> use and test to pass. Most of the code is to create a json output test
>>>>>>>> result compatible with the "kci_data" tool.
>>>>>>> Exactly.
>>>>>>>
>>>>>>>>>> BR,
>>>>>>>>>>
>>>>>>>>>> Santi
>>>>>>>>>>
>>>>>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>>>>>> that there are things that need to be polished on our json files.
>>>>>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>>>>>> are you on IRC?  That would help with discussing things and
>>>>>>>>> addressing any issues.
>>>>>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>>>>>> "sesteban"). I'm waiting the token!
>>>>>>> Sent!  Awaiting some data now :)
>>>>>> Hi Guillaume,
>>>>>>
>>>>>> I've this credential/token error when trying to push a kernel build:
>>>>>>
>>>>>>             requests.exceptions.HTTPError: 403 Client Error: Operation not
>>>>>> permitted: provided token is not         authorized for url:
>>>>>> https://api.staging.kernelci.org/upload
>>>>>>             Error: can not push kernel: ../builds/linux-mainline,
>>>>>> https://api.staging.kernelci.org/test, "my-token"
>>>>>>
>>>>>> Something misconfigured in the database?
>>>>> Yes, it looks like an oversight in the part of the code that adds
>>>>> lab tokens as it doesn't enable the permission to send results
>>>>> via the generic test API...  I've fixed that directly in the
>>>>> database and tried your token with kci_data, it works now.
>>>>>
>>>>> Also I've created this issue on GitHub to fix it properly:
>>>>>
>>>>>      https://github.com/kernelci/kernelci-backend/issues/268
>>>>>
>>>>>
>>>>> Let's hope this works for you, and please let us know how it goes
>>>>> with the Labgrid integration.
>>>> Hi,
>>>>
>>>> I think I had the same problem with my local instance of KernelCI. I had
>>>> to manually update the properties for the lab. At the time, I thought
>>>> that it was not a bug and that somehow LAVA could do it right.
>>>>
>>>> Unfortunatelly, the problem persists.
>>>>
>>>> Traceback (most recent call last):
>>>>      File "./kci_build", line 353, in <module>
>>>>        status = opts.command(configs, opts)
>>>>      File "./kci_build", line 316, in __call__
>>>>        args.install_path)
>>>>      File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890,
>>>> in push_kernel
>>>>        upload_files(api, token, upload_path, artifacts)
>>>>      File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line
>>>> 44, in upload_files
>>>>        resp.raise_for_status()requests.exceptions.HTTPError: 403 Client
>>>> Error: Operation not permitted: provided token is not authorized for
>>>> url: https://api.staging.kernelci.org/upload
>>>>            Error: can not push kernel: ../builds/linux-mainline,
>>>> https://api.staging.kernelci.org/test, "my token"
>>>>
>>>> If you tested the token locally and it works, it might be a problem
>>>> related to the IP address allowed to push data in the lab...
>>> The token works, but it's a "lab" token with permissions to
>>> submit test results.  It's not for submitting kernel builds,
>>> which is what you're trying to do here.
>>>
>>> Can you try using builds from storage.kernelci.org?  They come
>>> with a bmeta.json file to get the kernel revision and all related
>>> meta-data you need when submitting the test results for a given
>>> kernel build.
>>>
>>> If you want to test some kernel revisions not currently built on
>>> kernelci.org you can open a GitHub issue to request it:
>>>
>>>     https://github.com/kernelci/kernelci-core/issues/new?assignees=&labels=&template=new-kernel-branch.md&title=Add+branch+BRANCH+from+TREE
>>>
>>> Thanks,
>>> Guillaume
>> Hi Guillaume,
>>
>> I have finally push some results in the staging database. I'm sorry that
>> took me so long, but we just had a bank holiday here in Spain :)
>>
>> https://staging.kernelci.org/soc/at91/job/mainline/kernel/v5.10-rc7-10-ga68a0262abda/plan/baseline/
>>
>> At first glance looks ok, but if you click on the "Full results"
>> (https://staging.kernelci.org/test/plan/id/5fd21e6c81bee0f1bf417ce4/)
>> the page is broken... Well,  the results received from the database are
>> not complete. I checked the javascript in view-tests-plan-id.2020.10.js
>> and I found that  there are missing fields like "build_id". It seems
>> that not all the build information are available to all  "defconfigs" at
>> the staging database. The /test/plan/id page works fine in my local
>> kernelci deployment.
>>
>> I chose the build manually, because the token that you provide cannot
>> access the https://api.staging.kernelci.org/build information.
>>
>> While the issue #315 is solved, could we find a easy way to publish
>> results?I could poll the backend to check for new builds, do the tests,
>> and finally publish them.

Yes that is what other non-LAVA labs already do.  It's strange
your token doesn't let you read the build data, I'll check the
permissions associated with it.  What is the query you're using?

>> As how to solve #315 for good, I would suggest using Autobahn
>> (https://autobahn.readthedocs.io/en/latest/). Labgrid is build around it
>> and works like a charm. I'm no expert, maybe Pengutronix people can
>> share their experience and recommend this path or another.

OK thanks for sharing this.  I've started to look at CloudEvents:

  https://cloudevents.io/

I guess it would be good to list all the existing potential
solutions and compare to choose what works best for the KernelCI
use-case.

>> Best regards,
>>
>> Santi
>>
>> p.s I have attached the json file that I have pushed into the staging
>> database using kci_data, in case you need to look at it.
> 
> Hi Guillaume,

Sorry for the delay in replying, we've been going through a lot
of things with KernelCI lately to grow our team of maintainers.
Hopefully communication should become smoother now :)

> About pushing data into staging... Could you add some kernel builds with 
> "sama5_defconfig" and "at91_dt_defconfig" configurations for 
> mainline/master(or any other tree/branch )? I have not found those 
> configurations in the staging database, I would like to test this 
> thoroughly.

We can have them built once on staging, but we're not going to
enable them all the time because staging builds have to remain
limited to the bare minimum.  You can however get those builds
from production:

* sama5_defconfig https://kernelci.org/build/id/60331b14c9b222ae1aaddd1a/
* at91_dt_defconfig https://kernelci.org/build/id/60331a4b0b58f1585faddcb6/

You can download the build artifacts and the meta-data from there
and should be able to run some tests with these builds.  Would
that work for you?

> In a (kind of) different topic, what is the status of Generic Lab 
> Support #315? Are there any plans to implement this soon? I could 
> dedicate time to push forward this task, but definitively, I would need 
> some steering to avoid wasting both my time and your time.

As mentioned in my earlier email about KCIDB, we've been trying
to fix the blocking issues with submitting data from
kernelci-bakend to KCIDB.  This should get resolved in a week or
two hopefully, then kernelci-backend will need to go through some
redesign.  With the code and experience we have, I think it will
be a rather straightforward development "sprint" but it might
take a couple of months of maturing to get that working really
well and enable it in production.  We should have a dedicated
meeting to discuss that in greater detail in the coming weeks.

Best wishes,
Guillaume

Re: Contribute to Kernel-CI with a new Lab

[Santiago.Esteban via info]

On 22/2/21 11:44, Guillaume Tucker wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>
> On 27/01/2021 12:11, Santiago.Esteban@microchip.com wrote:
>> On 11/12/20 19:40, Santiago Esteban - M44029 wrote:
>>> On 7/12/20 10:08, Guillaume Tucker wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> On 04/12/2020 11:46, Santiago.Esteban@microchip.com wrote:
>>>>> On 3/12/20 15:01, Guillaume Tucker wrote:
>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>
>>>>>> On 26/11/2020 10:57, Santiago.Esteban@microchip.com wrote:
>>>>>>> On 26/11/20 10:17, Guillaume Tucker wrote:
>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>
>>>>>>>> On 26/11/2020 07:30, Santiago.Esteban@microchip.com wrote:
>>>>>>>>> On 25/11/20 15:09, Guillaume Tucker wrote:
>>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>>
>>>>>>>>>> On 12/11/2020 11:46, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>>> On 12/11/20 9:49, Jan Lübbe wrote:
>>>>>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Santi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 2020-11-11 at 18:05 +0000, Santiago.Esteban via info via groups.io wrote:
>>>>>>>>>>>>> Hi KernelCI,
>>>>>>>>>>>>>
>>>>>>>>>>>>> A few months back I contacted you about adding a new lab to KernelCI
>>>>>>>>>>>>> infrastructure. It took us longer than I wished, but now we finally
>>>>>>>>>>>>> have been able to connect our farm to an internal KernelCI deployment
>>>>>>>>>>>>> (using docker).
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I explained before, our farm holds 4 boards with different SoCs
>>>>>>>>>>>>> from Microchip (and Atmel): Sam9x60ek, Sama5d2_xplained,
>>>>>>>>>>>>> Sama5d3_xplained and Sama5d4_xplained. All of then with mainline
>>>>>>>>>>>>> support.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Our setup, does not uses Lava and it relies on Labgrid to perform the
>>>>>>>>>>>>> tests. We use "kci_data" tool to publish test results on KernelCI.
>>>>>>>>>>>> That's very interesting. :) Do you have published the interface code
>>>>>>>>>>>> somewhere? I've been trying to find some time to do that myself, but it
>>>>>>>>>>>> seems you've beaten me to it. ;)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Jan
>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>
>>>>>>>>>>> No, I haven't publish it, till now. I've never though it would be useful
>>>>>>>>>>> to anybody but me  ;)
>>>>>>>>>>>
>>>>>>>>>>> I have a python script that is tailored (too much) to our systems. It
>>>>>>>>>>> performs some actions that depend on our infrastructure and how I've
>>>>>>>>>>> implemented the labgrid tests (for example, I grab the results from a
>>>>>>>>>>> pytest  json report). At the end, it creates a "tmeta_<board>.json" file
>>>>>>>>>>> (equivalent to the "bmeta.json") that is later published with "kci_data"
>>>>>>>>>>> tool.
>>>>>>>>>>>
>>>>>>>>>>> It could be used as an inspiration to make a more generic
>>>>>>>>>>> "kci_labgrid_test" tool if there are interest, but, all kernelci
>>>>>>>>>>> important stuff, still needs to be validated ;)
>>>>>>>>>> There is already a plugin mechanism behind kci_test to have
>>>>>>>>>> specific implementations for generating the test job description
>>>>>>>>>> data and submitting it to a remote lab.  At the moment, only LAVA
>>>>>>>>>> has an implementation but it sounds like Labgrid would be a neat
>>>>>>>>>> addition.  The main challenge I guess, is that there isn't any
>>>>>>>>>> built-in API for Labgrid to receive job descriptions and run
>>>>>>>>>> them.
>>>>>>>>> Hi Guillaume,
>>>>>>>>>
>>>>>>>>> Labgrid does not perform any task related to build. It focuses on
>>>>>>>>> testing in real hardware. The build stage is performed somewhere else.
>>>>>>>>> It is a different approach than KernelCI and LAVA. In our case, we
>>>>>>>>> trigger the builds in Jenkins and also Jenkins executes the Labgrid
>>>>>>>>> pytest scripts. It would be great to be able to push KCI jobs into a
>>>>>>>>> generic Jenkins server. But, for me, the most difficult part would be
>>>>>>>>> convince IT department to allow a communication from a external source
>>>>>>>>> into the corporate network. I already know the answer....
>>>>>>>> Absolutely, that's the same as with LAVA.  The idea is that the
>>>>>>>> same builds from kernelci.org can be used in any lab, they're
>>>>>>>> just plain kernel builds.  What we could work on is to make it
>>>>>>>> easier for labs like yours and in particular when using Labgrid
>>>>>>>> to receive some data when there's a new test to run.  That would
>>>>>>>> include the URLs to get the kernel image, modules, dtb,
>>>>>>>> user-space, commands to run the tests etc...
>>>>>>>>
>>>>>>>>>> One way to address this issue is to provide notifications with
>>>>>>>>>> jobs to run, and a lab using Labgrid would subscribe to it.  As
>>>>>>>>>> we're planning to refactor kernelci-backend and also come up with
>>>>>>>>>> a more generic pipeline mechanis than Jenkins, we should be able
>>>>>>>>>> to have this kind of notification system in place anyway in order
>>>>>>>>>> to corrdinate the different components of the pipeline in a
>>>>>>>>>> modular way.  It would be great to use some Labgrid labs as part
>>>>>>>>>> of this exercise.  If you're interested, we can discuss that when
>>>>>>>>>> we have some clear ideas about the main part of work.
>>>>>>>>> A publish/subscribe approach will work perfectly to overcome the
>>>>>>>>> corporate IT hurdles. Of course, I 'm interested. Please let me know how
>>>>>>>>> can I be part of this.
>>>>>>>> Great.  There is a roadmap item on GitHub about this:
>>>>>>>>
>>>>>>>>        https://github.com/kernelci/kernelci-core/issues/315
>>>>>>>>
>>>>>>>> We've only just started, with a very basic kci_data
>>>>>>>> implementation.  This should be broken down into a series of
>>>>>>>> tasks such as: cleaning up the kernelci-backend API for receiving
>>>>>>>> results, providing a subscription mechanism and having a
>>>>>>>> real-world reference system to test it (e.g. a Labgrid lab).  We
>>>>>>>> probably should schedule a meeting to go through that in January.
>>>>>>>>
>>>>>>>>>>> I have attached it to this email the script and (more important) an
>>>>>>>>>>> example of the output it produces.
>>>>>>>>>> Thanks!  It's great to see that it doesn't really take much code
>>>>>>>>>> to get some KernelCI jobs to run in Labgrid.
>>>>>>>>> Labgrid only cares about  3 things: image to test, board in the farm to
>>>>>>>>> use and test to pass. Most of the code is to create a json output test
>>>>>>>>> result compatible with the "kci_data" tool.
>>>>>>>> Exactly.
>>>>>>>>
>>>>>>>>>>> BR,
>>>>>>>>>>>
>>>>>>>>>>> Santi
>>>>>>>>>>>
>>>>>>>>>>>>> We would like to work with you to be able to publish these results.
>>>>>>>>>>>>> Will it be possible to get a token for the staging database? I'm sure
>>>>>>>>>>>>> that there are things that need to be polished on our json files.
>>>>>>>>>> Absolutely, I'll send you a staging API token privately.  Also,
>>>>>>>>>> are you on IRC?  That would help with discussing things and
>>>>>>>>>> addressing any issues.
>>>>>>>>> Great! I'm not ussually in IRC.... I'll be more available there (as
>>>>>>>>> "sesteban"). I'm waiting the token!
>>>>>>>> Sent!  Awaiting some data now :)
>>>>>>> Hi Guillaume,
>>>>>>>
>>>>>>> I've this credential/token error when trying to push a kernel build:
>>>>>>>
>>>>>>>              requests.exceptions.HTTPError: 403 Client Error: Operation not
>>>>>>> permitted: provided token is not         authorized for url:
>>>>>>> https://api.staging.kernelci.org/upload
>>>>>>>              Error: can not push kernel: ../builds/linux-mainline,
>>>>>>> https://api.staging.kernelci.org/test, "my-token"
>>>>>>>
>>>>>>> Something misconfigured in the database?
>>>>>> Yes, it looks like an oversight in the part of the code that adds
>>>>>> lab tokens as it doesn't enable the permission to send results
>>>>>> via the generic test API...  I've fixed that directly in the
>>>>>> database and tried your token with kci_data, it works now.
>>>>>>
>>>>>> Also I've created this issue on GitHub to fix it properly:
>>>>>>
>>>>>>       https://github.com/kernelci/kernelci-backend/issues/268
>>>>>>
>>>>>>
>>>>>> Let's hope this works for you, and please let us know how it goes
>>>>>> with the Labgrid integration.
>>>>> Hi,
>>>>>
>>>>> I think I had the same problem with my local instance of KernelCI. I had
>>>>> to manually update the properties for the lab. At the time, I thought
>>>>> that it was not a bug and that somehow LAVA could do it right.
>>>>>
>>>>> Unfortunatelly, the problem persists.
>>>>>
>>>>> Traceback (most recent call last):
>>>>>       File "./kci_build", line 353, in <module>
>>>>>         status = opts.command(configs, opts)
>>>>>       File "./kci_build", line 316, in __call__
>>>>>         args.install_path)
>>>>>       File "/opt/mpuci-kernelci/kernelci-core/kernelci/build.py", line 890,
>>>>> in push_kernel
>>>>>         upload_files(api, token, upload_path, artifacts)
>>>>>       File "/opt/mpuci-kernelci/kernelci-core/kernelci/storage.py", line
>>>>> 44, in upload_files
>>>>>         resp.raise_for_status()requests.exceptions.HTTPError: 403 Client
>>>>> Error: Operation not permitted: provided token is not authorized for
>>>>> url: https://api.staging.kernelci.org/upload
>>>>>             Error: can not push kernel: ../builds/linux-mainline,
>>>>> https://api.staging.kernelci.org/test, "my token"
>>>>>
>>>>> If you tested the token locally and it works, it might be a problem
>>>>> related to the IP address allowed to push data in the lab...
>>>> The token works, but it's a "lab" token with permissions to
>>>> submit test results.  It's not for submitting kernel builds,
>>>> which is what you're trying to do here.
>>>>
>>>> Can you try using builds from storage.kernelci.org?  They come
>>>> with a bmeta.json file to get the kernel revision and all related
>>>> meta-data you need when submitting the test results for a given
>>>> kernel build.
>>>>
>>>> If you want to test some kernel revisions not currently built on
>>>> kernelci.org you can open a GitHub issue to request it:
>>>>
>>>>      https://github.com/kernelci/kernelci-core/issues/new?assignees=&labels=&template=new-kernel-branch.md&title=Add+branch+BRANCH+from+TREE
>>>>
>>>> Thanks,
>>>> Guillaume
>>> Hi Guillaume,
>>>
>>> I have finally push some results in the staging database. I'm sorry that
>>> took me so long, but we just had a bank holiday here in Spain :)
>>>
>>> https://staging.kernelci.org/soc/at91/job/mainline/kernel/v5.10-rc7-10-ga68a0262abda/plan/baseline/
>>>
>>> At first glance looks ok, but if you click on the "Full results"
>>> (https://staging.kernelci.org/test/plan/id/5fd21e6c81bee0f1bf417ce4/)
>>> the page is broken... Well,  the results received from the database are
>>> not complete. I checked the javascript in view-tests-plan-id.2020.10.js
>>> and I found that  there are missing fields like "build_id". It seems
>>> that not all the build information are available to all  "defconfigs" at
>>> the staging database. The /test/plan/id page works fine in my local
>>> kernelci deployment.
>>>
>>> I chose the build manually, because the token that you provide cannot
>>> access the https://api.staging.kernelci.org/build information.
>>>
>>> While the issue #315 is solved, could we find a easy way to publish
>>> results?I could poll the backend to check for new builds, do the tests,
>>> and finally publish them.
> Yes that is what other non-LAVA labs already do.  It's strange
> your token doesn't let you read the build data, I'll check the
> permissions associated with it.  What is the query you're using?
>
>>> As how to solve #315 for good, I would suggest using Autobahn
>>> (https://autobahn.readthedocs.io/en/latest/). Labgrid is build around it
>>> and works like a charm. I'm no expert, maybe Pengutronix people can
>>> share their experience and recommend this path or another.
> OK thanks for sharing this.  I've started to look at CloudEvents:
>
>    https://cloudevents.io/
>
> I guess it would be good to list all the existing potential
> solutions and compare to choose what works best for the KernelCI
> use-case.
I'll look at it.
>
>>> Best regards,
>>>
>>> Santi
>>>
>>> p.s I have attached the json file that I have pushed into the staging
>>> database using kci_data, in case you need to look at it.
>> Hi Guillaume,
> Sorry for the delay in replying, we've been going through a lot
> of things with KernelCI lately to grow our team of maintainers.
> Hopefully communication should become smoother now :)
>
>> About pushing data into staging... Could you add some kernel builds with
>> "sama5_defconfig" and "at91_dt_defconfig" configurations for
>> mainline/master(or any other tree/branch )? I have not found those
>> configurations in the staging database, I would like to test this
>> thoroughly.
> We can have them built once on staging, but we're not going to
> enable them all the time because staging builds have to remain
> limited to the bare minimum.  You can however get those builds
> from production:
>
> * sama5_defconfig https://kernelci.org/build/id/60331b14c9b222ae1aaddd1a/
> * at91_dt_defconfig https://kernelci.org/build/id/60331a4b0b58f1585faddcb6/
>
> You can download the build artifacts and the meta-data from there
> and should be able to run some tests with these builds.  Would
> that work for you?

I'll check with those builds. Thanks.

>> In a (kind of) different topic, what is the status of Generic Lab
>> Support #315? Are there any plans to implement this soon? I could
>> dedicate time to push forward this task, but definitively, I would need
>> some steering to avoid wasting both my time and your time.
> As mentioned in my earlier email about KCIDB, we've been trying
> to fix the blocking issues with submitting data from
> kernelci-bakend to KCIDB.  This should get resolved in a week or
> two hopefully, then kernelci-backend will need to go through some
> redesign.  With the code and experience we have, I think it will
> be a rather straightforward development "sprint" but it might
> take a couple of months of maturing to get that working really
> well and enable it in production.  We should have a dedicated
> meeting to discuss that in greater detail in the coming weeks.
>
> Best wishes,
> Guillaume

I'll be patience ;)

Thank you very much,

Santi