Re: [kernelci-members] Working with the KernelCI project

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

+ kernelci maillist and a colleague

On 7/9/20 8:59 PM, Dmitry Vyukov wrote:
 > On Thu, Jul 9, 2020 at 7:53 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> I can take a look at a specific syzbot issue data, try to craft a mock
 >> submission to KCIDB (in JSON), and walk through the potential processing logic
 >> for you. Would that be useful?
 >
 > Yes, it would be very useful...

Alright, first a few kcidb basics. You can submit all your data by just piping
JSON to the "kcidb-submit" tool (providing some options for authentication and
destination), or you could use the Python 3 API.

The expected JSON data is a dictionary of arrays, along with the schema
version identifier. Each of the arrays in the dictionary could be missing or
empty, but otherwise they contain various report objects. The database will
accept them in any (reasonable) amount, order, or combination, although
submission order can affect when and how notifications are sent out.

Every object has an ID which can be used to refer to it and link objects
together, but all the IDs are generated by the submitter, who is making sure
they're unique. For most of the objects (except revisions at this moment) you
can just use your CI system's ID for it and prefix it with your CI system's
name. This way you don't have to maintain a mapping between your system's IDs
and our IDs when you report results gradually. If you don't have that, you can
just generate them, for example hash some key fields or, as the last resort,
use UUIDs. Revisions are just using commit hashes at the moment.

Every object has a special property called "misc" which can contain arbitrary
data. You could use that to submit data we don't have a schema for yet. The
users/developers will be able to see it, and we can use that as samples for
implementing future support.

See example use of "misc" for a KernelCI test and its execution environment
(yes, it could be formatted better):
https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec

I mocked up a submission of two syzkaller runs from
https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326
with the initial support for issues we can implement. Note that there are
more properties defined by the schema you can add to the submissions, but
they're not required (same as some of the included).

Let's say you found a new commit or release to test with syzkaller. You can
submit that information right away:

     {
         "revisions": [
             {
                 "id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
                 "origin": "syzbot",
                 "discovery_time": "2020-06-25T03:21:48+00:00",
                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
                 "git_repository_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
                 "git_repository_branch": "master",
                 "valid": true
             },
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

When you've built it, you can tell us about it:

     {
         "builds": [
             {
                 "id": "syzkaller:d195fe572fb15312",
                 "revision_id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
                 "architecture": "x86_64",
                 "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
                 "start_time": "2020-06-25T03:21:48+00:00",
                 "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=d195fe572fb15312",
                 "valid": true
             },
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

If it failed to build you can send us the same data, but with "valid" set to
"false", and maybe attach the log. You don't have to, though.

Next, after you made it crash with syzkaller, you can send us your test
result along with links to some output files:

     {
         "tests": [
             {
                 "build_id": "syzkaller:d195fe572fb15312",
                 "id": "syzkaller:d195fe572fb15312",
                 "output_files": [
                     {
                         "name": "log.txt",
                         "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=137c5419100000"
                     },
                     {
                         "name": "report.txt",
                         "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=1543f44b100000"
                     },
                     {
                         "name": "repro.syz.txt",
                         "url": "https://syzkaller.appspot.com/text?tag=ReproSyz&x=12941c03100000"
                     },
                     {
                         "name": "repro.c",
                         "url": "https://syzkaller.appspot.com/text?tag=ReproC&x=12002cf9100000"
                     }
                 ],
                 "path": "syzkaller",
                 "start_time": "2020-06-25T03:21:48+00:00",
                 "status": "FAIL",
                 "waived": false
             },
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

We're putting the reproducers into output files here, but later we can add
dedicated fields, to allow inclusion into notification e-mails, for example.

At this point, some subscriptions watching for any new test failures might
trigger a notification (e-mail report) to some subscribers
(developers/maintainers).

Everything described above is currently supported, now onto what I think we
can do to support the issues (for the start).

Once you've created the issue for the crash you detected above, you can send
it to us:

     {
         "issues": [
             {
                 "id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
                 "origin": "syzkaller",
                 "subject": "WARNING in idr_alloc",
                 "url": "https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326"
             }
         ],
         "version": {
             "major": 4,
             "minor": 0
         }
     }

You can also send a record of this issue occurring (an "incident") in the test
run you reported above:

     {
         "incidents": [
             {
                 "test_id": "syzkaller:d195fe572fb15312",
                 "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
                 "origin": "syzkaller"
             },
         ],
         "version": {
             "major": 4,
             "minor": 0
         }
     }

All of the above could be sent in one go as well. For example you could send
another record of the same crash occurring in another revision like this:

     {
         "revisions": [
             {
                 "id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
                 "origin": "syzbot",
                 "discovery_time": "2020-07-12T15:56:32+00:00",
                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
                 "git_repository_commit_hash": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
                 "git_repository_branch": "master",
                 "valid": true
             }
         ],
         "builds": [
             {
                 "id": "syzkaller:66ad203c2bb6d8b",
                 "revision_id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
                 "architecture": "x86_64",
                 "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
                 "start_time": "2020-07-12T15:56:32+00:00",
                 "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=66ad203c2bb6d8b",
                 "valid": true
             }
         ],
         "tests": [
             {
                 "build_id": "syzkaller:66ad203c2bb6d8b",
                 "id": "syzkaller:66ad203c2bb6d8b",
                 "output_files": [
                     {
                         "name": "log.txt",
                         "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=12dcca00900000"
                     },
                     {
                         "name": "report.txt",
                         "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=136ebf47100000"
                     }
                 ],
                 "path": "syzkaller",
                 "start_time": "2020-07-12T15:56:03+00:00",
                 "status": "FAIL",
                 "waived": false
             }
         ],
         "incidents": [
             {
                 "test_id": "syzkaller:66ad203c2bb6d8b",
                 "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
                 "origin": "syzkaller"
             }
         ],
         "version": {
             "major": 4,
             "minor": 0
         }
     }

This time the notification system will notice that this failure is linked to
an already-reported issue and won't issue a notification e-mail.

CI systems which don't track issues would just not report any "issues" or
"incidents". Any tests arriving without an issue incident in the same JSON
document, or in the database already, would each be considered incidents of an
(unknown) new issue, and reported appropriately.

This could be the first step in implementing issue support. After that we can
work on marking duplicate bugs, perhaps, or explore what we could do with
marking bugs obsolete. However, we probably don't want to replace a bug
tracker, but only use issue information in order to produce appropriate
result notifications and otherwise redirect to a real bug tracker for issue
discussion, investigation, etc.

You mention extracting an issue reproducer as a separate event. How is that
done, who would be interested in such an event and how would they be alerted?

Meanwhile, I can help you set up a simple bridge from syzbot to KCIDB sending
the data we can already support (the above minus "issues" and "incidents",
plus maybe some extra data), so that we can see how your data can (or can't)
fit, what kind of load we get, what we can do with notifications and UI, what
we need to change to accommodate it better, etc.

We could set test status to "DONE" in all your reports (i.e. no "PASS" or
"FAIL" verdict), so that we don't generate unnecessary notifications until we
can handle issues.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

On 7/17/20 3:22 PM, Nikolai Kondrashov wrote:
 > On 7/9/20 8:59 PM, Dmitry Vyukov wrote:
 >  > On Thu, Jul 9, 2020 at 7:53 PM Nikolai Kondrashov
 >  > <Nikolai.Kondrashov@redhat.com> wrote:
 >  >> I can take a look at a specific syzbot issue data, try to craft a mock
 >  >> submission to KCIDB (in JSON), and walk through the potential processing logic
 >  >> for you. Would that be useful?
 >  >
 >  > Yes, it would be very useful...
 >  >
 > This could be the first step in implementing issue support. After that we can
 > work on marking duplicate bugs, perhaps, or explore what we could do with
 > marking bugs obsolete. However, we probably don't want to replace a bug
 > tracker, but only use issue information in order to produce appropriate
 > result notifications and otherwise redirect to a real bug tracker for issue
 > discussion, investigation, etc.
 >
 > You mention extracting an issue reproducer as a separate event. How is that
 > done, who would be interested in such an event and how would they be alerted?
 >
 > Meanwhile, I can help you set up a simple bridge from syzbot to KCIDB sending
 > the data we can already support (the above minus "issues" and "incidents",
 > plus maybe some extra data), so that we can see how your data can (or can't)
 > fit, what kind of load we get, what we can do with notifications and UI, what
 > we need to change to accommodate it better, etc.
 >
 > We could set test status to "DONE" in all your reports (i.e. no "PASS" or
 > "FAIL" verdict), so that we don't generate unnecessary notifications until we
 > can handle issues.

Did you have time to take a look at my mock-up? It probably looks like too
much text, but please don't hesitate to reach me with any questions or
suggestions you have as you look through it, if you haven't already.

Please note that you don't have to start with sending all this data. In fact,
only a few structural fields are required and we can start with the absolute
minimum. The mock-up is showing what you *could* already send, but you don't
have to go that far.

I'd like to schedule a hacking session at Plumbers where we go and try to get
some data out of participants' systems and into KCIDB. Would you be interested
to attend, perhaps?

Nick

On 7/17/20 3:22 PM, Nikolai Kondrashov wrote:
 > Hi Dmitry,
 >
 > + kernelci maillist and a colleague
 >
 > On 7/9/20 8:59 PM, Dmitry Vyukov wrote:
 >  > On Thu, Jul 9, 2020 at 7:53 PM Nikolai Kondrashov
 >  > <Nikolai.Kondrashov@redhat.com> wrote:
 >  >> I can take a look at a specific syzbot issue data, try to craft a mock
 >  >> submission to KCIDB (in JSON), and walk through the potential processing logic
 >  >> for you. Would that be useful?
 >  >
 >  > Yes, it would be very useful...
 >
 > Alright, first a few kcidb basics. You can submit all your data by just piping
 > JSON to the "kcidb-submit" tool (providing some options for authentication and
 > destination), or you could use the Python 3 API.
 >
 > The expected JSON data is a dictionary of arrays, along with the schema
 > version identifier. Each of the arrays in the dictionary could be missing or
 > empty, but otherwise they contain various report objects. The database will
 > accept them in any (reasonable) amount, order, or combination, although
 > submission order can affect when and how notifications are sent out.
 >
 > Every object has an ID which can be used to refer to it and link objects
 > together, but all the IDs are generated by the submitter, who is making sure
 > they're unique. For most of the objects (except revisions at this moment) you
 > can just use your CI system's ID for it and prefix it with your CI system's
 > name. This way you don't have to maintain a mapping between your system's IDs
 > and our IDs when you report results gradually. If you don't have that, you can
 > just generate them, for example hash some key fields or, as the last resort,
 > use UUIDs. Revisions are just using commit hashes at the moment.
 >
 > Every object has a special property called "misc" which can contain arbitrary
 > data. You could use that to submit data we don't have a schema for yet. The
 > users/developers will be able to see it, and we can use that as samples for
 > implementing future support.
 >
 > See example use of "misc" for a KernelCI test and its execution environment
 > (yes, it could be formatted better):
 > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
 >
 > I mocked up a submission of two syzkaller runs from
 > https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326
 > with the initial support for issues we can implement. Note that there are
 > more properties defined by the schema you can add to the submissions, but
 > they're not required (same as some of the included).
 >
 > Let's say you found a new commit or release to test with syzkaller. You can
 > submit that information right away:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
 >                  "origin": "syzbot",
 >                  "discovery_time": "2020-06-25T03:21:48+00:00",
 >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
 >                  "git_repository_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
 >                  "git_repository_branch": "master",
 >                  "valid": true
 >              },
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > When you've built it, you can tell us about it:
 >
 >      {
 >          "builds": [
 >              {
 >                  "id": "syzkaller:d195fe572fb15312",
 >                  "revision_id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
 >                  "architecture": "x86_64",
 >                  "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
 >                  "start_time": "2020-06-25T03:21:48+00:00",
 >                  "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=d195fe572fb15312",
 >                  "valid": true
 >              },
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > If it failed to build you can send us the same data, but with "valid" set to
 > "false", and maybe attach the log. You don't have to, though.
 >
 > Next, after you made it crash with syzkaller, you can send us your test
 > result along with links to some output files:
 >
 >      {
 >          "tests": [
 >              {
 >                  "build_id": "syzkaller:d195fe572fb15312",
 >                  "id": "syzkaller:d195fe572fb15312",
 >                  "output_files": [
 >                      {
 >                          "name": "log.txt",
 >                          "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=137c5419100000"
 >                      },
 >                      {
 >                          "name": "report.txt",
 >                          "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=1543f44b100000"
 >                      },
 >                      {
 >                          "name": "repro.syz.txt",
 >                          "url": "https://syzkaller.appspot.com/text?tag=ReproSyz&x=12941c03100000"
 >                      },
 >                      {
 >                          "name": "repro.c",
 >                          "url": "https://syzkaller.appspot.com/text?tag=ReproC&x=12002cf9100000"
 >                      }
 >                  ],
 >                  "path": "syzkaller",
 >                  "start_time": "2020-06-25T03:21:48+00:00",
 >                  "status": "FAIL",
 >                  "waived": false
 >              },
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > We're putting the reproducers into output files here, but later we can add
 > dedicated fields, to allow inclusion into notification e-mails, for example.
 >
 > At this point, some subscriptions watching for any new test failures might
 > trigger a notification (e-mail report) to some subscribers
 > (developers/maintainers).
 >
 > Everything described above is currently supported, now onto what I think we
 > can do to support the issues (for the start).
 >
 > Once you've created the issue for the crash you detected above, you can send
 > it to us:
 >
 >      {
 >          "issues": [
 >              {
 >                  "id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
 >                  "origin": "syzkaller",
 >                  "subject": "WARNING in idr_alloc",
 >                  "url": "https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326"
 >              }
 >          ],
 >          "version": {
 >              "major": 4,
 >              "minor": 0
 >          }
 >      }
 >
 > You can also send a record of this issue occurring (an "incident") in the test
 > run you reported above:
 >
 >      {
 >          "incidents": [
 >              {
 >                  "test_id": "syzkaller:d195fe572fb15312",
 >                  "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
 >                  "origin": "syzkaller"
 >              },
 >          ],
 >          "version": {
 >              "major": 4,
 >              "minor": 0
 >          }
 >      }
 >
 > All of the above could be sent in one go as well. For example you could send
 > another record of the same crash occurring in another revision like this:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
 >                  "origin": "syzbot",
 >                  "discovery_time": "2020-07-12T15:56:32+00:00",
 >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
 >                  "git_repository_commit_hash": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
 >                  "git_repository_branch": "master",
 >                  "valid": true
 >              }
 >          ],
 >          "builds": [
 >              {
 >                  "id": "syzkaller:66ad203c2bb6d8b",
 >                  "revision_id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
 >                  "architecture": "x86_64",
 >                  "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
 >                  "start_time": "2020-07-12T15:56:32+00:00",
 >                  "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=66ad203c2bb6d8b",
 >                  "valid": true
 >              }
 >          ],
 >          "tests": [
 >              {
 >                  "build_id": "syzkaller:66ad203c2bb6d8b",
 >                  "id": "syzkaller:66ad203c2bb6d8b",
 >                  "output_files": [
 >                      {
 >                          "name": "log.txt",
 >                          "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=12dcca00900000"
 >                      },
 >                      {
 >                          "name": "report.txt",
 >                          "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=136ebf47100000"
 >                      }
 >                  ],
 >                  "path": "syzkaller",
 >                  "start_time": "2020-07-12T15:56:03+00:00",
 >                  "status": "FAIL",
 >                  "waived": false
 >              }
 >          ],
 >          "incidents": [
 >              {
 >                  "test_id": "syzkaller:66ad203c2bb6d8b",
 >                  "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
 >                  "origin": "syzkaller"
 >              }
 >          ],
 >          "version": {
 >              "major": 4,
 >              "minor": 0
 >          }
 >      }
 >
 > This time the notification system will notice that this failure is linked to
 > an already-reported issue and won't issue a notification e-mail.
 >
 > CI systems which don't track issues would just not report any "issues" or
 > "incidents". Any tests arriving without an issue incident in the same JSON
 > document, or in the database already, would each be considered incidents of an
 > (unknown) new issue, and reported appropriately.
 >
 > This could be the first step in implementing issue support. After that we can
 > work on marking duplicate bugs, perhaps, or explore what we could do with
 > marking bugs obsolete. However, we probably don't want to replace a bug
 > tracker, but only use issue information in order to produce appropriate
 > result notifications and otherwise redirect to a real bug tracker for issue
 > discussion, investigation, etc.
 >
 > You mention extracting an issue reproducer as a separate event. How is that
 > done, who would be interested in such an event and how would they be alerted?
 >
 > Meanwhile, I can help you set up a simple bridge from syzbot to KCIDB sending
 > the data we can already support (the above minus "issues" and "incidents",
 > plus maybe some extra data), so that we can see how your data can (or can't)
 > fit, what kind of load we get, what we can do with notifications and UI, what
 > we need to change to accommodate it better, etc.
 >
 > We could set test status to "DONE" in all your reports (i.e. no "PASS" or
 > "FAIL" verdict), so that we don't generate unnecessary notifications until we
 > can handle issues.
 >
 > Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Dmitry,
>
> On 7/17/20 3:22 PM, Nikolai Kondrashov wrote:
>  > On 7/9/20 8:59 PM, Dmitry Vyukov wrote:
>  >  > On Thu, Jul 9, 2020 at 7:53 PM Nikolai Kondrashov
>  >  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >  >> I can take a look at a specific syzbot issue data, try to craft a mock
>  >  >> submission to KCIDB (in JSON), and walk through the potential processing logic
>  >  >> for you. Would that be useful?
>  >  >
>  >  > Yes, it would be very useful...
>  >  >
>  > This could be the first step in implementing issue support. After that we can
>  > work on marking duplicate bugs, perhaps, or explore what we could do with
>  > marking bugs obsolete. However, we probably don't want to replace a bug
>  > tracker, but only use issue information in order to produce appropriate
>  > result notifications and otherwise redirect to a real bug tracker for issue
>  > discussion, investigation, etc.
>  >
>  > You mention extracting an issue reproducer as a separate event. How is that
>  > done, who would be interested in such an event and how would they be alerted?
>  >
>  > Meanwhile, I can help you set up a simple bridge from syzbot to KCIDB sending
>  > the data we can already support (the above minus "issues" and "incidents",
>  > plus maybe some extra data), so that we can see how your data can (or can't)
>  > fit, what kind of load we get, what we can do with notifications and UI, what
>  > we need to change to accommodate it better, etc.
>  >
>  > We could set test status to "DONE" in all your reports (i.e. no "PASS" or
>  > "FAIL" verdict), so that we don't generate unnecessary notifications until we
>  > can handle issues.
>
> Did you have time to take a look at my mock-up? It probably looks like too
> much text, but please don't hesitate to reach me with any questions or
> suggestions you have as you look through it, if you haven't already.
>
> Please note that you don't have to start with sending all this data. In fact,
> only a few structural fields are required and we can start with the absolute
> minimum. The mock-up is showing what you *could* already send, but you don't
> have to go that far.
>
> I'd like to schedule a hacking session at Plumbers where we go and try to get
> some data out of participants' systems and into KCIDB. Would you be interested
> to attend, perhaps?

Hi Nick,

I've seen your reply and seen that you provided an example of how to
fill in the data. I was starred in my inbox, but I did not have time
to look at the details yet (we have somewhat crazy intern season). I
still want to do this.

I am not sure how a remote virtual hacking session is different from
just doing this normally and some email exchange :)





> On 7/17/20 3:22 PM, Nikolai Kondrashov wrote:
>  > Hi Dmitry,
>  >
>  > + kernelci maillist and a colleague
>  >
>  > On 7/9/20 8:59 PM, Dmitry Vyukov wrote:
>  >  > On Thu, Jul 9, 2020 at 7:53 PM Nikolai Kondrashov
>  >  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >  >> I can take a look at a specific syzbot issue data, try to craft a mock
>  >  >> submission to KCIDB (in JSON), and walk through the potential processing logic
>  >  >> for you. Would that be useful?
>  >  >
>  >  > Yes, it would be very useful...
>  >
>  > Alright, first a few kcidb basics. You can submit all your data by just piping
>  > JSON to the "kcidb-submit" tool (providing some options for authentication and
>  > destination), or you could use the Python 3 API.
>  >
>  > The expected JSON data is a dictionary of arrays, along with the schema
>  > version identifier. Each of the arrays in the dictionary could be missing or
>  > empty, but otherwise they contain various report objects. The database will
>  > accept them in any (reasonable) amount, order, or combination, although
>  > submission order can affect when and how notifications are sent out.
>  >
>  > Every object has an ID which can be used to refer to it and link objects
>  > together, but all the IDs are generated by the submitter, who is making sure
>  > they're unique. For most of the objects (except revisions at this moment) you
>  > can just use your CI system's ID for it and prefix it with your CI system's
>  > name. This way you don't have to maintain a mapping between your system's IDs
>  > and our IDs when you report results gradually. If you don't have that, you can
>  > just generate them, for example hash some key fields or, as the last resort,
>  > use UUIDs. Revisions are just using commit hashes at the moment.
>  >
>  > Every object has a special property called "misc" which can contain arbitrary
>  > data. You could use that to submit data we don't have a schema for yet. The
>  > users/developers will be able to see it, and we can use that as samples for
>  > implementing future support.
>  >
>  > See example use of "misc" for a KernelCI test and its execution environment
>  > (yes, it could be formatted better):
>  > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
>  >
>  > I mocked up a submission of two syzkaller runs from
>  > https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326
>  > with the initial support for issues we can implement. Note that there are
>  > more properties defined by the schema you can add to the submissions, but
>  > they're not required (same as some of the included).
>  >
>  > Let's say you found a new commit or release to test with syzkaller. You can
>  > submit that information right away:
>  >
>  >      {
>  >          "revisions": [
>  >              {
>  >                  "id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
>  >                  "origin": "syzbot",
>  >                  "discovery_time": "2020-06-25T03:21:48+00:00",
>  >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
>  >                  "git_repository_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
>  >                  "git_repository_branch": "master",
>  >                  "valid": true
>  >              },
>  >          ],
>  >          "version": {
>  >              "major": 3,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > When you've built it, you can tell us about it:
>  >
>  >      {
>  >          "builds": [
>  >              {
>  >                  "id": "syzkaller:d195fe572fb15312",
>  >                  "revision_id": "7ae77150d94d3b535c7b85e6b3647113095e79bf",
>  >                  "architecture": "x86_64",
>  >                  "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
>  >                  "start_time": "2020-06-25T03:21:48+00:00",
>  >                  "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=d195fe572fb15312",
>  >                  "valid": true
>  >              },
>  >          ],
>  >          "version": {
>  >              "major": 3,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > If it failed to build you can send us the same data, but with "valid" set to
>  > "false", and maybe attach the log. You don't have to, though.
>  >
>  > Next, after you made it crash with syzkaller, you can send us your test
>  > result along with links to some output files:
>  >
>  >      {
>  >          "tests": [
>  >              {
>  >                  "build_id": "syzkaller:d195fe572fb15312",
>  >                  "id": "syzkaller:d195fe572fb15312",
>  >                  "output_files": [
>  >                      {
>  >                          "name": "log.txt",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=137c5419100000"
>  >                      },
>  >                      {
>  >                          "name": "report.txt",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=1543f44b100000"
>  >                      },
>  >                      {
>  >                          "name": "repro.syz.txt",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=ReproSyz&x=12941c03100000"
>  >                      },
>  >                      {
>  >                          "name": "repro.c",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=ReproC&x=12002cf9100000"
>  >                      }
>  >                  ],
>  >                  "path": "syzkaller",
>  >                  "start_time": "2020-06-25T03:21:48+00:00",
>  >                  "status": "FAIL",
>  >                  "waived": false
>  >              },
>  >          ],
>  >          "version": {
>  >              "major": 3,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > We're putting the reproducers into output files here, but later we can add
>  > dedicated fields, to allow inclusion into notification e-mails, for example.
>  >
>  > At this point, some subscriptions watching for any new test failures might
>  > trigger a notification (e-mail report) to some subscribers
>  > (developers/maintainers).
>  >
>  > Everything described above is currently supported, now onto what I think we
>  > can do to support the issues (for the start).
>  >
>  > Once you've created the issue for the crash you detected above, you can send
>  > it to us:
>  >
>  >      {
>  >          "issues": [
>  >              {
>  >                  "id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
>  >                  "origin": "syzkaller",
>  >                  "subject": "WARNING in idr_alloc",
>  >                  "url": "https://syzkaller.appspot.com/bug?id=ca2299cf11b3e3d3d0f44ac479410a14eecbd326"
>  >              }
>  >          ],
>  >          "version": {
>  >              "major": 4,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > You can also send a record of this issue occurring (an "incident") in the test
>  > run you reported above:
>  >
>  >      {
>  >          "incidents": [
>  >              {
>  >                  "test_id": "syzkaller:d195fe572fb15312",
>  >                  "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
>  >                  "origin": "syzkaller"
>  >              },
>  >          ],
>  >          "version": {
>  >              "major": 4,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > All of the above could be sent in one go as well. For example you could send
>  > another record of the same crash occurring in another revision like this:
>  >
>  >      {
>  >          "revisions": [
>  >              {
>  >                  "id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
>  >                  "origin": "syzbot",
>  >                  "discovery_time": "2020-07-12T15:56:32+00:00",
>  >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
>  >                  "git_repository_commit_hash": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
>  >                  "git_repository_branch": "master",
>  >                  "valid": true
>  >              }
>  >          ],
>  >          "builds": [
>  >              {
>  >                  "id": "syzkaller:66ad203c2bb6d8b",
>  >                  "revision_id": "0aea6d5c5be33ce94c16f9ab2f64de1f481f424b",
>  >                  "architecture": "x86_64",
>  >                  "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
>  >                  "start_time": "2020-07-12T15:56:32+00:00",
>  >                  "config_url": "https://syzkaller.appspot.com/text?tag=KernelConfig&x=66ad203c2bb6d8b",
>  >                  "valid": true
>  >              }
>  >          ],
>  >          "tests": [
>  >              {
>  >                  "build_id": "syzkaller:66ad203c2bb6d8b",
>  >                  "id": "syzkaller:66ad203c2bb6d8b",
>  >                  "output_files": [
>  >                      {
>  >                          "name": "log.txt",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=CrashLog&x=12dcca00900000"
>  >                      },
>  >                      {
>  >                          "name": "report.txt",
>  >                          "url": "https://syzkaller.appspot.com/text?tag=CrashReport&x=136ebf47100000"
>  >                      }
>  >                  ],
>  >                  "path": "syzkaller",
>  >                  "start_time": "2020-07-12T15:56:03+00:00",
>  >                  "status": "FAIL",
>  >                  "waived": false
>  >              }
>  >          ],
>  >          "incidents": [
>  >              {
>  >                  "test_id": "syzkaller:66ad203c2bb6d8b",
>  >                  "issue_id": "syzkaller:ca2299cf11b3e3d3d0f44ac479410a14eecbd326",
>  >                  "origin": "syzkaller"
>  >              }
>  >          ],
>  >          "version": {
>  >              "major": 4,
>  >              "minor": 0
>  >          }
>  >      }
>  >
>  > This time the notification system will notice that this failure is linked to
>  > an already-reported issue and won't issue a notification e-mail.
>  >
>  > CI systems which don't track issues would just not report any "issues" or
>  > "incidents". Any tests arriving without an issue incident in the same JSON
>  > document, or in the database already, would each be considered incidents of an
>  > (unknown) new issue, and reported appropriately.
>  >
>  > This could be the first step in implementing issue support. After that we can
>  > work on marking duplicate bugs, perhaps, or explore what we could do with
>  > marking bugs obsolete. However, we probably don't want to replace a bug
>  > tracker, but only use issue information in order to produce appropriate
>  > result notifications and otherwise redirect to a real bug tracker for issue
>  > discussion, investigation, etc.
>  >
>  > You mention extracting an issue reproducer as a separate event. How is that
>  > done, who would be interested in such an event and how would they be alerted?
>  >
>  > Meanwhile, I can help you set up a simple bridge from syzbot to KCIDB sending
>  > the data we can already support (the above minus "issues" and "incidents",
>  > plus maybe some extra data), so that we can see how your data can (or can't)
>  > fit, what kind of load we get, what we can do with notifications and UI, what
>  > we need to change to accommodate it better, etc.
>  >
>  > We could set test status to "DONE" in all your reports (i.e. no "PASS" or
>  > "FAIL" verdict), so that we don't generate unnecessary notifications until we
>  > can handle issues.
>  >
>  > Nick
>
>
> 
>

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
 > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
 >> Did you have time to take a look at my mock-up? It probably looks like too
 >> much text, but please don't hesitate to reach me with any questions or
 >> suggestions you have as you look through it, if you haven't already.
 >>
 >> Please note that you don't have to start with sending all this data. In fact,
 >> only a few structural fields are required and we can start with the absolute
 >> minimum. The mock-up is showing what you *could* already send, but you don't
 >> have to go that far.
 >>
 >> I'd like to schedule a hacking session at Plumbers where we go and try to get
 >> some data out of participants' systems and into KCIDB. Would you be interested
 >> to attend, perhaps?
 >
 > Hi Nick,
 >
 > I've seen your reply and seen that you provided an example of how to
 > fill in the data. I was starred in my inbox, but I did not have time
 > to look at the details yet (we have somewhat crazy intern season). I
 > still want to do this.

Glad to hear that, and I understand about the time.

 > I am not sure how a remote virtual hacking session is different from
 > just doing this normally and some email exchange :)

I would say the low response latency helps move faster :)

We could have a video call in background and/or share a screen or a tmate
session, as we try to make things work. Even just having someone available in
an IRC chat for quick responses helps, I'd say.

Everyone has their own preferences, of course, I'm just offering to be there
and discuss things and maybe work on something :)

As I just wrote to Philip, I've finally published the schema introduction
article:

     https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

Here's our talk, same MC as yours, but later:

     https://linuxplumbersconf.org/event/7/contributions/730/

and the BoF session next day:

     https://linuxplumbersconf.org/event/7/contributions/731/

Will be glad to meet you there, even though we won't be able to do
much in the 45 minutes we were allocated.

So please also drop by our "hallway chat", whenever you have time:

     https://chat.2020.linuxplumbersconf.org/channel/kcidb

I'll be there every day of the conference, during Europe's working hours.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Fri, Aug 21, 2020 at 12:10 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Dmitry,
>
> On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
>  > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
>  >> Did you have time to take a look at my mock-up? It probably looks like too
>  >> much text, but please don't hesitate to reach me with any questions or
>  >> suggestions you have as you look through it, if you haven't already.
>  >>
>  >> Please note that you don't have to start with sending all this data. In fact,
>  >> only a few structural fields are required and we can start with the absolute
>  >> minimum. The mock-up is showing what you *could* already send, but you don't
>  >> have to go that far.
>  >>
>  >> I'd like to schedule a hacking session at Plumbers where we go and try to get
>  >> some data out of participants' systems and into KCIDB. Would you be interested
>  >> to attend, perhaps?
>  >
>  > Hi Nick,
>  >
>  > I've seen your reply and seen that you provided an example of how to
>  > fill in the data. I was starred in my inbox, but I did not have time
>  > to look at the details yet (we have somewhat crazy intern season). I
>  > still want to do this.
>
> Glad to hear that, and I understand about the time.
>
>  > I am not sure how a remote virtual hacking session is different from
>  > just doing this normally and some email exchange :)
>
> I would say the low response latency helps move faster :)
>
> We could have a video call in background and/or share a screen or a tmate
> session, as we try to make things work. Even just having someone available in
> an IRC chat for quick responses helps, I'd say.
>
> Everyone has their own preferences, of course, I'm just offering to be there
> and discuss things and maybe work on something :)
>
> As I just wrote to Philip, I've finally published the schema introduction
> article:
>
>      https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>
> Here's our talk, same MC as yours, but later:
>
>      https://linuxplumbersconf.org/event/7/contributions/730/
>
> and the BoF session next day:
>
>      https://linuxplumbersconf.org/event/7/contributions/731/
>
> Will be glad to meet you there, even though we won't be able to do
> much in the 45 minutes we were allocated.
>
> So please also drop by our "hallway chat", whenever you have time:
>
>      https://chat.2020.linuxplumbersconf.org/channel/kcidb
>
> I'll be there every day of the conference, during Europe's working hours.


Hi Nikolai,

The examples are helpful. I think this should be doable. The ability
to send everything in a single request is handy, and since we don't
send bulky data (only links), should not cause any issues.

I need an API key, right? How are they distributed? Could you send me
one to my email privately?

However, we don't use any Python and use of kcidb-submit will be
problematic as the app is a Go App Engine app.
I looked what's the actual protocol and got to here:
https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
and this "client" seems to come from google.cloud.pubsub.

Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
it would be much more practical for syzbot to use that directly. But
what exactly is the protocol? I assume it's that JSON, but is it
wrapped into some other layers? What's the topic name to publish? Is
there description/example on this level?

Also, how can I test things? Is there a test instance? Or if I push
something, can I observe it on the KernelCI dashboard or something?
What would be the procedure for deploying it on the syzbot side?

Thanks

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 > The examples are helpful. I think this should be doable. The ability
 > to send everything in a single request is handy, and since we don't
 > send bulky data (only links), should not cause any issues.

Awesome!

 > I need an API key, right? How are they distributed? Could you send me
 > one to my email privately?

Yep, just sent it.

 > However, we don't use any Python and use of kcidb-submit will be
 > problematic as the app is a Go App Engine app.

Do you mean you can't run executables, or that having the Python stack in
there as well is not possible/burdensome? Just curious.

 > I looked what's the actual protocol and got to here:
 > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 > and this "client" seems to come from google.cloud.pubsub.
 >
 > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 > it would be much more practical for syzbot to use that directly. But
 > what exactly is the protocol? I assume it's that JSON, but is it
 > wrapped into some other layers? What's the topic name to publish? Is
 > there description/example on this level?

There's no documentation available for this level, yet, just the code.

However, yes, it's just JSON submitted to a PubSub topic, currently, and the
schema is exactly as output by "kcidb-schema".

Please validate JSON fully before submitting to the PubSub topic
("kcidb-validate" could help you start). We don't have error reporting to
submitters from the server side yet, we'll have to read the logs to find out
what happened, if it breaks :)

I sent you the project/topic names in the message with the credentials.

Oh, and we would definitely appreciate a contribution of the Go submission
module, if you make one :)

 > Also, how can I test things? Is there a test instance? Or if I push
 > something, can I observe it on the KernelCI dashboard or something?

Yes, there's a "playground" instance, details are in the message with the
credentials. You can see the results on the "playground" dashboard, and query
them from the "playground" database with "kcidb-query".

 > What would be the procedure for deploying it on the syzbot side?

Do you mean "deploying the dashboard"?

Excited to have you join the effort!

Nick

On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 > On Fri, Aug 21, 2020 at 12:10 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> Hi Dmitry,
 >>
 >> On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
 >>   > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
 >>   >> Did you have time to take a look at my mock-up? It probably looks like too
 >>   >> much text, but please don't hesitate to reach me with any questions or
 >>   >> suggestions you have as you look through it, if you haven't already.
 >>   >>
 >>   >> Please note that you don't have to start with sending all this data. In fact,
 >>   >> only a few structural fields are required and we can start with the absolute
 >>   >> minimum. The mock-up is showing what you *could* already send, but you don't
 >>   >> have to go that far.
 >>   >>
 >>   >> I'd like to schedule a hacking session at Plumbers where we go and try to get
 >>   >> some data out of participants' systems and into KCIDB. Would you be interested
 >>   >> to attend, perhaps?
 >>   >
 >>   > Hi Nick,
 >>   >
 >>   > I've seen your reply and seen that you provided an example of how to
 >>   > fill in the data. I was starred in my inbox, but I did not have time
 >>   > to look at the details yet (we have somewhat crazy intern season). I
 >>   > still want to do this.
 >>
 >> Glad to hear that, and I understand about the time.
 >>
 >>   > I am not sure how a remote virtual hacking session is different from
 >>   > just doing this normally and some email exchange :)
 >>
 >> I would say the low response latency helps move faster :)
 >>
 >> We could have a video call in background and/or share a screen or a tmate
 >> session, as we try to make things work. Even just having someone available in
 >> an IRC chat for quick responses helps, I'd say.
 >>
 >> Everyone has their own preferences, of course, I'm just offering to be there
 >> and discuss things and maybe work on something :)
 >>
 >> As I just wrote to Philip, I've finally published the schema introduction
 >> article:
 >>
 >>       https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
 >>
 >> Here's our talk, same MC as yours, but later:
 >>
 >>       https://linuxplumbersconf.org/event/7/contributions/730/
 >>
 >> and the BoF session next day:
 >>
 >>       https://linuxplumbersconf.org/event/7/contributions/731/
 >>
 >> Will be glad to meet you there, even though we won't be able to do
 >> much in the 45 minutes we were allocated.
 >>
 >> So please also drop by our "hallway chat", whenever you have time:
 >>
 >>       https://chat.2020.linuxplumbersconf.org/channel/kcidb
 >>
 >> I'll be there every day of the conference, during Europe's working hours.
 >
 >
 > Hi Nikolai,
 >
 > The examples are helpful. I think this should be doable. The ability
 > to send everything in a single request is handy, and since we don't
 > send bulky data (only links), should not cause any issues.
 >
 > I need an API key, right? How are they distributed? Could you send me
 > one to my email privately?
 >
 > However, we don't use any Python and use of kcidb-submit will be
 > problematic as the app is a Go App Engine app.
 > I looked what's the actual protocol and got to here:
 > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 > and this "client" seems to come from google.cloud.pubsub.
 >
 > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 > it would be much more practical for syzbot to use that directly. But
 > what exactly is the protocol? I assume it's that JSON, but is it
 > wrapped into some other layers? What's the topic name to publish? Is
 > there description/example on this level?
 >
 > Also, how can I test things? Is there a test instance? Or if I push
 > something, can I observe it on the KernelCI dashboard or something?
 > What would be the procedure for deploying it on the syzbot side?
 >
 > Thanks
 >

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Dmitry,
>
> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
>  > The examples are helpful. I think this should be doable. The ability
>  > to send everything in a single request is handy, and since we don't
>  > send bulky data (only links), should not cause any issues.
>
> Awesome!
>
>  > I need an API key, right? How are they distributed? Could you send me
>  > one to my email privately?
>
> Yep, just sent it.
>
>  > However, we don't use any Python and use of kcidb-submit will be
>  > problematic as the app is a Go App Engine app.
>
> Do you mean you can't run executables, or that having the Python stack in
> there as well is not possible/burdensome? Just curious.

Appengine is what's called "serverless" today. There are no servers,
filesystem, custom binaries. I just upload a set of source Go files
and Appengine takes care of the rest. So there is no way to deploy a
custom binary written in a different language.


>  > I looked what's the actual protocol and got to here:
>  > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
>  > and this "client" seems to come from google.cloud.pubsub.
>  >
>  > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
>  > it would be much more practical for syzbot to use that directly. But
>  > what exactly is the protocol? I assume it's that JSON, but is it
>  > wrapped into some other layers? What's the topic name to publish? Is
>  > there description/example on this level?
>
> There's no documentation available for this level, yet, just the code.
>
> However, yes, it's just JSON submitted to a PubSub topic, currently, and the
> schema is exactly as output by "kcidb-schema".
>
> Please validate JSON fully before submitting to the PubSub topic
> ("kcidb-validate" could help you start). We don't have error reporting to
> submitters from the server side yet, we'll have to read the logs to find out
> what happened, if it breaks :)
>
> I sent you the project/topic names in the message with the credentials.
>
> Oh, and we would definitely appreciate a contribution of the Go submission
> module, if you make one :)
>
>  > Also, how can I test things? Is there a test instance? Or if I push
>  > something, can I observe it on the KernelCI dashboard or something?
>
> Yes, there's a "playground" instance, details are in the message with the
> credentials. You can see the results on the "playground" dashboard, and query
> them from the "playground" database with "kcidb-query".
>
>  > What would be the procedure for deploying it on the syzbot side?
>
> Do you mean "deploying the dashboard"?

I meant when/how do we deploy this uploading to production. I mean if
I do a prototype and it works with a playground, what's next?
... answering my own question, I guess we can make production syzbot
upload to the playground kcidb first and then see how it works. If it
works well, then we can switch to production kcidb.



I have a question re schema. You mentioned "incidents" above. But I
don't see them in the kcidb-schema output.



> Excited to have you join the effort!
>
> Nick
>
> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
>  > On Fri, Aug 21, 2020 at 12:10 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> Hi Dmitry,
>  >>
>  >> On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
>  >>   > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
>  >>   >> Did you have time to take a look at my mock-up? It probably looks like too
>  >>   >> much text, but please don't hesitate to reach me with any questions or
>  >>   >> suggestions you have as you look through it, if you haven't already.
>  >>   >>
>  >>   >> Please note that you don't have to start with sending all this data. In fact,
>  >>   >> only a few structural fields are required and we can start with the absolute
>  >>   >> minimum. The mock-up is showing what you *could* already send, but you don't
>  >>   >> have to go that far.
>  >>   >>
>  >>   >> I'd like to schedule a hacking session at Plumbers where we go and try to get
>  >>   >> some data out of participants' systems and into KCIDB. Would you be interested
>  >>   >> to attend, perhaps?
>  >>   >
>  >>   > Hi Nick,
>  >>   >
>  >>   > I've seen your reply and seen that you provided an example of how to
>  >>   > fill in the data. I was starred in my inbox, but I did not have time
>  >>   > to look at the details yet (we have somewhat crazy intern season). I
>  >>   > still want to do this.
>  >>
>  >> Glad to hear that, and I understand about the time.
>  >>
>  >>   > I am not sure how a remote virtual hacking session is different from
>  >>   > just doing this normally and some email exchange :)
>  >>
>  >> I would say the low response latency helps move faster :)
>  >>
>  >> We could have a video call in background and/or share a screen or a tmate
>  >> session, as we try to make things work. Even just having someone available in
>  >> an IRC chat for quick responses helps, I'd say.
>  >>
>  >> Everyone has their own preferences, of course, I'm just offering to be there
>  >> and discuss things and maybe work on something :)
>  >>
>  >> As I just wrote to Philip, I've finally published the schema introduction
>  >> article:
>  >>
>  >>       https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>  >>
>  >> Here's our talk, same MC as yours, but later:
>  >>
>  >>       https://linuxplumbersconf.org/event/7/contributions/730/
>  >>
>  >> and the BoF session next day:
>  >>
>  >>       https://linuxplumbersconf.org/event/7/contributions/731/
>  >>
>  >> Will be glad to meet you there, even though we won't be able to do
>  >> much in the 45 minutes we were allocated.
>  >>
>  >> So please also drop by our "hallway chat", whenever you have time:
>  >>
>  >>       https://chat.2020.linuxplumbersconf.org/channel/kcidb
>  >>
>  >> I'll be there every day of the conference, during Europe's working hours.
>  >
>  >
>  > Hi Nikolai,
>  >
>  > The examples are helpful. I think this should be doable. The ability
>  > to send everything in a single request is handy, and since we don't
>  > send bulky data (only links), should not cause any issues.
>  >
>  > I need an API key, right? How are they distributed? Could you send me
>  > one to my email privately?
>  >
>  > However, we don't use any Python and use of kcidb-submit will be
>  > problematic as the app is a Go App Engine app.
>  > I looked what's the actual protocol and got to here:
>  > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
>  > and this "client" seems to come from google.cloud.pubsub.
>  >
>  > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
>  > it would be much more practical for syzbot to use that directly. But
>  > what exactly is the protocol? I assume it's that JSON, but is it
>  > wrapped into some other layers? What's the topic name to publish? Is
>  > there description/example on this level?
>  >
>  > Also, how can I test things? Is there a test instance? Or if I push
>  > something, can I observe it on the KernelCI dashboard or something?
>  > What would be the procedure for deploying it on the syzbot side?
>  >
>  > Thanks
>  >
>

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 5:23 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
> >
> > Hi Dmitry,
> >
> > On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
> >  > The examples are helpful. I think this should be doable. The ability
> >  > to send everything in a single request is handy, and since we don't
> >  > send bulky data (only links), should not cause any issues.
> >
> > Awesome!
> >
> >  > I need an API key, right? How are they distributed? Could you send me
> >  > one to my email privately?
> >
> > Yep, just sent it.
> >
> >  > However, we don't use any Python and use of kcidb-submit will be
> >  > problematic as the app is a Go App Engine app.
> >
> > Do you mean you can't run executables, or that having the Python stack in
> > there as well is not possible/burdensome? Just curious.
>
> Appengine is what's called "serverless" today. There are no servers,
> filesystem, custom binaries. I just upload a set of source Go files
> and Appengine takes care of the rest. So there is no way to deploy a
> custom binary written in a different language.
>
>
> >  > I looked what's the actual protocol and got to here:
> >  > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
> >  > and this "client" seems to come from google.cloud.pubsub.
> >  >
> >  > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
> >  > it would be much more practical for syzbot to use that directly. But
> >  > what exactly is the protocol? I assume it's that JSON, but is it
> >  > wrapped into some other layers? What's the topic name to publish? Is
> >  > there description/example on this level?
> >
> > There's no documentation available for this level, yet, just the code.
> >
> > However, yes, it's just JSON submitted to a PubSub topic, currently, and the
> > schema is exactly as output by "kcidb-schema".
> >
> > Please validate JSON fully before submitting to the PubSub topic
> > ("kcidb-validate" could help you start). We don't have error reporting to
> > submitters from the server side yet, we'll have to read the logs to find out
> > what happened, if it breaks :)
> >
> > I sent you the project/topic names in the message with the credentials.
> >
> > Oh, and we would definitely appreciate a contribution of the Go submission
> > module, if you make one :)
> >
> >  > Also, how can I test things? Is there a test instance? Or if I push
> >  > something, can I observe it on the KernelCI dashboard or something?
> >
> > Yes, there's a "playground" instance, details are in the message with the
> > credentials. You can see the results on the "playground" dashboard, and query
> > them from the "playground" database with "kcidb-query".
> >
> >  > What would be the procedure for deploying it on the syzbot side?
> >
> > Do you mean "deploying the dashboard"?
>
> I meant when/how do we deploy this uploading to production. I mean if
> I do a prototype and it works with a playground, what's next?
> ... answering my own question, I guess we can make production syzbot
> upload to the playground kcidb first and then see how it works. If it
> works well, then we can switch to production kcidb.
>
>
>
> I have a question re schema. You mentioned "incidents" above. But I
> don't see them in the kcidb-schema output.

There are no "issues" either...
Here is what I've got from kcidb-schema (on latest checkout):
https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 6:09 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Mon, Sep 28, 2020 at 5:23 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
> > <Nikolai.Kondrashov@redhat.com> wrote:
> > >
> > > Hi Dmitry,
> > >
> > > On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
> > >  > The examples are helpful. I think this should be doable. The ability
> > >  > to send everything in a single request is handy, and since we don't
> > >  > send bulky data (only links), should not cause any issues.
> > >
> > > Awesome!
> > >
> > >  > I need an API key, right? How are they distributed? Could you send me
> > >  > one to my email privately?
> > >
> > > Yep, just sent it.
> > >
> > >  > However, we don't use any Python and use of kcidb-submit will be
> > >  > problematic as the app is a Go App Engine app.
> > >
> > > Do you mean you can't run executables, or that having the Python stack in
> > > there as well is not possible/burdensome? Just curious.
> >
> > Appengine is what's called "serverless" today. There are no servers,
> > filesystem, custom binaries. I just upload a set of source Go files
> > and Appengine takes care of the rest. So there is no way to deploy a
> > custom binary written in a different language.
> >
> >
> > >  > I looked what's the actual protocol and got to here:
> > >  > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
> > >  > and this "client" seems to come from google.cloud.pubsub.
> > >  >
> > >  > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
> > >  > it would be much more practical for syzbot to use that directly. But
> > >  > what exactly is the protocol? I assume it's that JSON, but is it
> > >  > wrapped into some other layers? What's the topic name to publish? Is
> > >  > there description/example on this level?
> > >
> > > There's no documentation available for this level, yet, just the code.
> > >
> > > However, yes, it's just JSON submitted to a PubSub topic, currently, and the
> > > schema is exactly as output by "kcidb-schema".
> > >
> > > Please validate JSON fully before submitting to the PubSub topic
> > > ("kcidb-validate" could help you start). We don't have error reporting to
> > > submitters from the server side yet, we'll have to read the logs to find out
> > > what happened, if it breaks :)
> > >
> > > I sent you the project/topic names in the message with the credentials.
> > >
> > > Oh, and we would definitely appreciate a contribution of the Go submission
> > > module, if you make one :)
> > >
> > >  > Also, how can I test things? Is there a test instance? Or if I push
> > >  > something, can I observe it on the KernelCI dashboard or something?
> > >
> > > Yes, there's a "playground" instance, details are in the message with the
> > > credentials. You can see the results on the "playground" dashboard, and query
> > > them from the "playground" database with "kcidb-query".
> > >
> > >  > What would be the procedure for deploying it on the syzbot side?
> > >
> > > Do you mean "deploying the dashboard"?
> >
> > I meant when/how do we deploy this uploading to production. I mean if
> > I do a prototype and it works with a playground, what's next?
> > ... answering my own question, I guess we can make production syzbot
> > upload to the playground kcidb first and then see how it works. If it
> > works well, then we can switch to production kcidb.
> >
> >
> >
> > I have a question re schema. You mentioned "incidents" above. But I
> > don't see them in the kcidb-schema output.
>
> There are no "issues" either...
> Here is what I've got from kcidb-schema (on latest checkout):
> https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt

I've submitted the following json to the pubsub endpoint, and it seems
to succeed as I got server message ID back: 1592574115213643
But I can't find this entry on the dashboard, among other places I've looked at:
https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
Wasn't it submitted? Or am I looking in  a wrong place? Or is there some log?

{
  "builds": [
    {
      "architecture": "x86_64",
      "command": "",
      "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
      "config_name": "",
      "config_url":
"https://syzkaller.appspot.com/text?tag=KernelConfig\u0026x=d195fe572fb15312",
      "description": "",
      "duration": 0,
      "id": "syzkaller:d195fe572fb15355",
      "input_files": null,
      "log_url": "",
      "misc": null,
      "origin": "syzbot",
      "output_files": null,
      "revision_id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
      "start_time": "2020-06-25T03:21:48+00:00",
      "valid": true
    }
  ],
  "revisions": [
    {
      "contacts": null,
      "description": "",
      "discovery_time": "2020-06-25T03:21:48+00:00",
      "git_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e7900",
      "git_commit_name": "",
      "git_repository_branch": "master",
      "git_repository_url":
"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
      "id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
      "log_url": "",
      "message_id": "",
      "misc": null,
      "origin": "syzbot",
      "patch_mboxes": null,
      "publishing_time": "",
      "tree_name": "",
      "valid": true
    }
  ],
  "tests": [
    {
      "build_id": "syzkaller:d195fe572fb15355",
      "description": "",
      "duration": 0,
      "environment": null,
      "id": "syzkaller:d195fe572fb15366",
      "misc": null,
      "origin": "syzbot",
      "output_files": [
        {
          "name": "log.txt",
          "url":
"https://syzkaller.appspot.com/text?tag=CrashLog\u0026x=137c5419100000"
        },
        {
          "name": "report.txt",
          "url":
"https://syzkaller.appspot.com/text?tag=CrashReport\u0026x=1543f44b100000"
        },
        {
          "name": "repro.syz.txt",
          "url":
"https://syzkaller.appspot.com/text?tag=ReproSyz\u0026x=12941c03100000"
        },
        {
          "name": "repro.c",
          "url":
"https://syzkaller.appspot.com/text?tag=ReproC\u0026x=12002cf9100000"
        }
      ],
      "path": "syzkaller",
      "start_time": "2020-06-25T03:21:48+00:00",
      "status": "FAIL",
      "waived": false
    }
  ],
  "version": {
    "major": 3,
    "minor": 0
  }
}

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

[-- Attachment #1: Type: text/plain, Size: 9902 bytes --]

On 9/28/20 7:24 PM, Dmitry Vyukov wrote:
 > I've submitted the following json to the pubsub endpoint, and it seems
 > to succeed as I got server message ID back: 1592574115213643

Weeeell, the message ID is only a confirmation that the message reached the
queue, we could still fail after taking it out of the queue :)

The PubSub protocol (or any message queue for that matter) simply doesn't
provide a way to report on how the message was used on the other end.

 > But I can't find this entry on the dashboard, among other places I've looked at:
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
 > Wasn't it submitted? Or am I looking in  a wrong place? Or is there some log?

It was submitted, but it wasn't valid according to the schema.

As I said earlier, we haven't implemented sending server-side errors to
submitters yet, although I have some ideas how that could be done.

So, to minimize the errors on the server side, please make sure to validate
your JSON before submitting. You can take the schema produced by kcidb-schema
and use it with a Go module for JSON validation. Apparently there are at least
three of them: https://json-schema.org/implementations.html#validator-go

For the start, though, you can also use "kcidb-validate". Its output is
somewhat more verbose than it should be, and we're working on that, but if you
feed the JSON you sent to it like this:

     kcidb-validate < syzbot.json

somewhere closer to the end of the output you'll find:

     ValidationError: None is not of type 'array'

     Failed validating 'type' in schema['properties']['revisions']['items']['properties']['patch_mboxes']:

It basically says that it didn't expect to find "null" as the value for the
"patch_mboxes" property.

In general "null"s are not accepted by the schema, to minimize ambiguity and
the number of checks when interpreting the data. If you want to omit any data,
just don't add the corresponding property.

Other errors are:

* "message_id" is specified as an empty string, but it should be an e-mail
   address. Just omit the property if you have nothing to send.
* "publishing_time" is an empty string and not an RFC3339 timestamp. Again,
   just omit it, if you don't have it.
* "log_url" should be a URL, but is an empty string. Same approach.

After the above is fixed, the file will validate. However, there are still
issues of the same sort - empty properties, which would be good to just
remove: "command", "config_name", "description", "git_commit_name",
"tree_name". Perhaps we should make the schema check those as well.

I've fixed all of the issues in the JSON you provided and attached it as
"syzbot_fixed.json".

Hope that helps!
Nick

Note to self: mention the property-omission approach explicitly in the docs
               https://github.com/kernelci/kcidb/issues/123

On 9/28/20 7:24 PM, Dmitry Vyukov wrote:
 > On Mon, Sep 28, 2020 at 6:09 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>
 >> On Mon, Sep 28, 2020 at 5:23 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>>
 >>> On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
 >>> <Nikolai.Kondrashov@redhat.com> wrote:
 >>>>
 >>>> Hi Dmitry,
 >>>>
 >>>> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 >>>>   > The examples are helpful. I think this should be doable. The ability
 >>>>   > to send everything in a single request is handy, and since we don't
 >>>>   > send bulky data (only links), should not cause any issues.
 >>>>
 >>>> Awesome!
 >>>>
 >>>>   > I need an API key, right? How are they distributed? Could you send me
 >>>>   > one to my email privately?
 >>>>
 >>>> Yep, just sent it.
 >>>>
 >>>>   > However, we don't use any Python and use of kcidb-submit will be
 >>>>   > problematic as the app is a Go App Engine app.
 >>>>
 >>>> Do you mean you can't run executables, or that having the Python stack in
 >>>> there as well is not possible/burdensome? Just curious.
 >>>
 >>> Appengine is what's called "serverless" today. There are no servers,
 >>> filesystem, custom binaries. I just upload a set of source Go files
 >>> and Appengine takes care of the rest. So there is no way to deploy a
 >>> custom binary written in a different language.
 >>>
 >>>
 >>>>   > I looked what's the actual protocol and got to here:
 >>>>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 >>>>   > and this "client" seems to come from google.cloud.pubsub.
 >>>>   >
 >>>>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 >>>>   > it would be much more practical for syzbot to use that directly. But
 >>>>   > what exactly is the protocol? I assume it's that JSON, but is it
 >>>>   > wrapped into some other layers? What's the topic name to publish? Is
 >>>>   > there description/example on this level?
 >>>>
 >>>> There's no documentation available for this level, yet, just the code.
 >>>>
 >>>> However, yes, it's just JSON submitted to a PubSub topic, currently, and the
 >>>> schema is exactly as output by "kcidb-schema".
 >>>>
 >>>> Please validate JSON fully before submitting to the PubSub topic
 >>>> ("kcidb-validate" could help you start). We don't have error reporting to
 >>>> submitters from the server side yet, we'll have to read the logs to find out
 >>>> what happened, if it breaks :)
 >>>>
 >>>> I sent you the project/topic names in the message with the credentials.
 >>>>
 >>>> Oh, and we would definitely appreciate a contribution of the Go submission
 >>>> module, if you make one :)
 >>>>
 >>>>   > Also, how can I test things? Is there a test instance? Or if I push
 >>>>   > something, can I observe it on the KernelCI dashboard or something?
 >>>>
 >>>> Yes, there's a "playground" instance, details are in the message with the
 >>>> credentials. You can see the results on the "playground" dashboard, and query
 >>>> them from the "playground" database with "kcidb-query".
 >>>>
 >>>>   > What would be the procedure for deploying it on the syzbot side?
 >>>>
 >>>> Do you mean "deploying the dashboard"?
 >>>
 >>> I meant when/how do we deploy this uploading to production. I mean if
 >>> I do a prototype and it works with a playground, what's next?
 >>> ... answering my own question, I guess we can make production syzbot
 >>> upload to the playground kcidb first and then see how it works. If it
 >>> works well, then we can switch to production kcidb.
 >>>
 >>>
 >>>
 >>> I have a question re schema. You mentioned "incidents" above. But I
 >>> don't see them in the kcidb-schema output.
 >>
 >> There are no "issues" either...
 >> Here is what I've got from kcidb-schema (on latest checkout):
 >> https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt
 >
 > I've submitted the following json to the pubsub endpoint, and it seems
 > to succeed as I got server message ID back: 1592574115213643
 > But I can't find this entry on the dashboard, among other places I've looked at:
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
 > Wasn't it submitted? Or am I looking in  a wrong place? Or is there some log?
 >
 > {
 >    "builds": [
 >      {
 >        "architecture": "x86_64",
 >        "command": "",
 >        "compiler": "gcc (GCC) 9.0.0 20181231 (experimental)",
 >        "config_name": "",
 >        "config_url":
 > "https://syzkaller.appspot.com/text?tag=KernelConfig\u0026x=d195fe572fb15312",
 >        "description": "",
 >        "duration": 0,
 >        "id": "syzkaller:d195fe572fb15355",
 >        "input_files": null,
 >        "log_url": "",
 >        "misc": null,
 >        "origin": "syzbot",
 >        "output_files": null,
 >        "revision_id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
 >        "start_time": "2020-06-25T03:21:48+00:00",
 >        "valid": true
 >      }
 >    ],
 >    "revisions": [
 >      {
 >        "contacts": null,
 >        "description": "",
 >        "discovery_time": "2020-06-25T03:21:48+00:00",
 >        "git_commit_hash": "7ae77150d94d3b535c7b85e6b3647113095e7900",
 >        "git_commit_name": "",
 >        "git_repository_branch": "master",
 >        "git_repository_url":
 > "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
 >        "id": "7ae77150d94d3b535c7b85e6b3647113095e7900",
 >        "log_url": "",
 >        "message_id": "",
 >        "misc": null,
 >        "origin": "syzbot",
 >        "patch_mboxes": null,
 >        "publishing_time": "",
 >        "tree_name": "",
 >        "valid": true
 >      }
 >    ],
 >    "tests": [
 >      {
 >        "build_id": "syzkaller:d195fe572fb15355",
 >        "description": "",
 >        "duration": 0,
 >        "environment": null,
 >        "id": "syzkaller:d195fe572fb15366",
 >        "misc": null,
 >        "origin": "syzbot",
 >        "output_files": [
 >          {
 >            "name": "log.txt",
 >            "url":
 > "https://syzkaller.appspot.com/text?tag=CrashLog\u0026x=137c5419100000"
 >          },
 >          {
 >            "name": "report.txt",
 >            "url":
 > "https://syzkaller.appspot.com/text?tag=CrashReport\u0026x=1543f44b100000"
 >          },
 >          {
 >            "name": "repro.syz.txt",
 >            "url":
 > "https://syzkaller.appspot.com/text?tag=ReproSyz\u0026x=12941c03100000"
 >          },
 >          {
 >            "name": "repro.c",
 >            "url":
 > "https://syzkaller.appspot.com/text?tag=ReproC\u0026x=12002cf9100000"
 >          }
 >        ],
 >        "path": "syzkaller",
 >        "start_time": "2020-06-25T03:21:48+00:00",
 >        "status": "FAIL",
 >        "waived": false
 >      }
 >    ],
 >    "version": {
 >      "major": 3,
 >      "minor": 0
 >    }
 > }
 >


[-- Attachment #2: syzbot_fixed.json --]
[-- Type: application/json, Size: 1784 bytes --]

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 7:16 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 9/28/20 7:24 PM, Dmitry Vyukov wrote:
>  > I've submitted the following json to the pubsub endpoint, and it seems
>  > to succeed as I got server message ID back: 1592574115213643
>
> Weeeell, the message ID is only a confirmation that the message reached the
> queue, we could still fail after taking it out of the queue :)
>
> The PubSub protocol (or any message queue for that matter) simply doesn't
> provide a way to report on how the message was used on the other end.
>
>  > But I can't find this entry on the dashboard, among other places I've looked at:
>  > https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
>  > Wasn't it submitted? Or am I looking in  a wrong place? Or is there some log?
>
> It was submitted, but it wasn't valid according to the schema.
>
> As I said earlier, we haven't implemented sending server-side errors to
> submitters yet, although I have some ideas how that could be done.
>
> So, to minimize the errors on the server side, please make sure to validate
> your JSON before submitting. You can take the schema produced by kcidb-schema
> and use it with a Go module for JSON validation. Apparently there are at least
> three of them: https://json-schema.org/implementations.html#validator-go
>
> For the start, though, you can also use "kcidb-validate". Its output is
> somewhat more verbose than it should be, and we're working on that, but if you
> feed the JSON you sent to it like this:
>
>      kcidb-validate < syzbot.json
>
> somewhere closer to the end of the output you'll find:
>
>      ValidationError: None is not of type 'array'
>
>      Failed validating 'type' in schema['properties']['revisions']['items']['properties']['patch_mboxes']:
>
> It basically says that it didn't expect to find "null" as the value for the
> "patch_mboxes" property.
>
> In general "null"s are not accepted by the schema, to minimize ambiguity and
> the number of checks when interpreting the data. If you want to omit any data,
> just don't add the corresponding property.
>
> Other errors are:
>
> * "message_id" is specified as an empty string, but it should be an e-mail
>    address. Just omit the property if you have nothing to send.
> * "publishing_time" is an empty string and not an RFC3339 timestamp. Again,
>    just omit it, if you don't have it.
> * "log_url" should be a URL, but is an empty string. Same approach.
>
> After the above is fixed, the file will validate. However, there are still
> issues of the same sort - empty properties, which would be good to just
> remove: "command", "config_name", "description", "git_commit_name",
> "tree_name". Perhaps we should make the schema check those as well.
>
> I've fixed all of the issues in the JSON you provided and attached it as
> "syzbot_fixed.json".

I managed to upload some data manually! Woohoo!
https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04

Some questions regarding data conversion:

1. We usually first get just a crash on 1 instance. Then later we may
get a reproducer, or we see it happen on more trees, or we've got
bisection results, or something tested a proposed fix. How can we
"add" more to the same instance of the bug?

2. Maybe a related question: we generally identify bugs and bucket
crashes based on "crash title" (obtained from the kernel oops message
via some non-trivial transformation), e.g. "KASAN: use-after-free in
open_file". If we've got a reproducer, this string is used to link it
to the previously happened crash. I don't see where I can put this
string. test.description?

3. The build.id and test.id: should they just be unique? Or the system
needs to be able to map these IDs back to objects? In particular does
KCIDB need to generate links back to the external system?
E.g. for each bug we have a page with full info, like this one:
https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
I think it would be useful for KCIDB to be able to provide this link.
However, the test.id generation that I am thinking of now will give
unique IDs, but we won't be able to map them back to anything useful.
Is it fine or not?
Currently I also add this link as a resource named "dashboard".  But
that's a custom name that KCIDB won't understand.

4. Should we add the kernel oops message itself inline? Or only via a link?
We have kernel oops messages nicely symbolized, cleaned of noise,
cleaned of unrelated threads, etc as you can see here:
https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d

5. We figure out emails (maintainers+lists) that we think should be
looking at this bug report. Where should we put them?
There is revision.contacts, but this is not related to revision but
rather test (or issue).

6. We have that tagged Reported-by line that we very much want
developers to use in the fixing commit to understand when bugs are
fixed.
If we detach them from info/reports available in KCIDB, then nobody
will use them in commits.
How can we pass it to KCIDB? Perhaps it's a useful feature for all
reports/test failures? If a bug is reported by Fuego and KCIDB
notifications/dashboard will ask to use "Reported-by: fuego", it will
give nice credits. We also found use of the hashes in the tag very
useful for statistics/tracking purposes. It makes it possible to link
failures to fixing commits and understand if a fix (1) exists in
general, (2) present in a particular tree, (3) or even if somebody
just mailed a proposed fix (useful to identify lost fixes that were
never merged, or avoid several people fixing the same bug if you can
show that somebody mailed a fix already).

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,
On 9/29/20 10:52 AM, Dmitry Vyukov wrote:
 > I managed to upload some data manually! Woohoo!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04

Aweeesooomeee :)

 > Some questions regarding data conversion:
 >
 > 1. We usually first get just a crash on 1 instance. Then later we may
 > get a reproducer, or we see it happen on more trees, or we've got
 > bisection results, or something tested a proposed fix. How can we
 > "add" more to the same instance of the bug?

Currently we don't have support for issues/incidents, so we can only accept
test results without the issue context. That will have to do for the start.

Once we have support for issues/incidents I think we would be able to use the
feature I'm working on next - an ability to add previously-missing fields to
an already-submitted object.

So, you would send a test result object, then an issue object with minimal
information, and then an incident object linking the two. As you get to know
more about the issue, you can send it again, omitting the fields you already
sent (or sending them with exact same values), but adding the
previously-missing fields with the new data. I.e. you first send the issue
without the proposed fix, and then send it again with the fix.

This feature is still in the research stage, but I'll be working on it after
the next kcidb release, which is coming in one or two weeks.

Regarding bisection results I was hoping we could represent that data
implicitly. I.e. by having information about which 'revision' comes after
which, and executing a query to find a pair of adjacent revisions one of which
has the bug and another doesn't. In theory that should be possible, but in
practice... well, you know :) If that fails, we can attach that data
explicitly somewhere. Or we can simply start with that to have it faster.

 > 2. Maybe a related question: we generally identify bugs and bucket
 > crashes based on "crash title" (obtained from the kernel oops message
 > via some non-trivial transformation), e.g. "KASAN: use-after-free in
 > open_file". If we've got a reproducer, this string is used to link it
 > to the previously happened crash. I don't see where I can put this
 > string. test.description?

So far my idea was to leave the identification and handling of such strings to
you. We could then represent that link using an incident object linking a test
execution to an issue, and the issue could then have the reproducer
linked/attached in a property.

For now, you can put that information into the "description" (which I plan to
rename into "comment" sometime soon), or perhaps into the "misc" object.

 > 3. The build.id and test.id: should they just be unique?

They need to be unique among all the corresponding objects you're submitting.
That's the only requirement.

 > Or the system
 > needs to be able to map these IDs back to objects? In particular does
 > KCIDB need to generate links back to the external system?
 > E.g. for each bug we have a page with full info, like this one:
 > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
 > I think it would be useful for KCIDB to be able to provide this link.

We're considering that, and I agree it could be useful.
See https://github.com/kernelci/kcidb/issues/93

 > However, the test.id generation that I am thinking of now will give
 > unique IDs, but we won't be able to map them back to anything useful.
 > Is it fine or not?

It's OK, as long as it's unique.

 > Currently I also add this link as a resource named "dashboard".  But
 > that's a custom name that KCIDB won't understand.

We have the "misc" object specifically for things like that: data you'd like
to see represented in an object, but which isn't supported by the schema yet.

You can put there whatever you want (within reasonable size limit), dashboard
users can get it now, and we can have a sample of what we need to support in
the schema.

See https://github.com/kernelci/kcidb/blob/master/SUBMISSION_HOWTO.md#extra-data

 > 4. Should we add the kernel oops message itself inline? Or only via a link?
 > We have kernel oops messages nicely symbolized, cleaned of noise,
 > cleaned of unrelated threads, etc as you can see here:
 > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d

Yes, those are very nice! We're planning to add support for embedding log
snippets and such into reports (0day needs that too), but for now you'll
have to submit that as a link to a file hosted somewhere, via test's
"output_files".

 > 5. We figure out emails (maintainers+lists) that we think should be
 > looking at this bug report. Where should we put them?
 > There is revision.contacts, but this is not related to revision but
 > rather test (or issue).

Yes, there's revision.contacts, which comes from initial CKI needs, and we
could add the same to the issues, when we have them.

However, with KCIDB, I'd like to try a reverse approach: instead of the CI
system deciding who to send reports to, give developers control on what they
receive via a "subscription" system. I.e. let them pick what they'll be
interested in: which CI systems, which trees, architectures, tests, results,
and so on. With that approach such data becomes unnecessary.

There's, of course many questions to answer and a proper implementation to
write before we know how that works, and perhaps we could start with using
that data, but that's the direction I'd like to try.

With many people submitting and the amount of data we're already getting we
need to give the developers the upper hand. I think it should be up to the CI
systems to submit the best data they can, up to KCIDB to provide
discoverability of that data, and up to developers to decide which data they
care about.

 > 6. We have that tagged Reported-by line that we very much want
 > developers to use in the fixing commit to understand when bugs are
 > fixed.
 > If we detach them from info/reports available in KCIDB, then nobody
 > will use them in commits.
 > How can we pass it to KCIDB? Perhaps it's a useful feature for all
 > reports/test failures? If a bug is reported by Fuego and KCIDB
 > notifications/dashboard will ask to use "Reported-by: fuego", it will
 > give nice credits. We also found use of the hashes in the tag very
 > useful for statistics/tracking purposes. It makes it possible to link
 > failures to fixing commits and understand if a fix (1) exists in
 > general, (2) present in a particular tree, (3) or even if somebody
 > just mailed a proposed fix (useful to identify lost fixes that were
 > never merged, or avoid several people fixing the same bug if you can
 > show that somebody mailed a fix already).

Hmm, yes, that could be very useful indeed, thank you.

We would have to use an ID managed by KCIDB, though. This brings me to the
question what we're going to use for identifying issues. Perhaps use the
submitter-supplied ID, similarly to builds and tests right now, under the
assumption that, at least for the start, issues are not going to be reused
across CI systems. If that's so, then you would be able to map those IDs back
to your issues.

Nick

On 9/29/20 10:52 AM, Dmitry Vyukov wrote:
 > On Mon, Sep 28, 2020 at 7:16 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 9/28/20 7:24 PM, Dmitry Vyukov wrote:
 >>   > I've submitted the following json to the pubsub endpoint, and it seems
 >>   > to succeed as I got server message ID back: 1592574115213643
 >>
 >> Weeeell, the message ID is only a confirmation that the message reached the
 >> queue, we could still fail after taking it out of the queue :)
 >>
 >> The PubSub protocol (or any message queue for that matter) simply doesn't
 >> provide a way to report on how the message was used on the other end.
 >>
 >>   > But I can't find this entry on the dashboard, among other places I've looked at:
 >>   > https://staging.kernelci.org:3000/d/home/home?orgId=1&from=now-30m&to=now&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&refresh=30m
 >>   > Wasn't it submitted? Or am I looking in  a wrong place? Or is there some log?
 >>
 >> It was submitted, but it wasn't valid according to the schema.
 >>
 >> As I said earlier, we haven't implemented sending server-side errors to
 >> submitters yet, although I have some ideas how that could be done.
 >>
 >> So, to minimize the errors on the server side, please make sure to validate
 >> your JSON before submitting. You can take the schema produced by kcidb-schema
 >> and use it with a Go module for JSON validation. Apparently there are at least
 >> three of them: https://json-schema.org/implementations.html#validator-go
 >>
 >> For the start, though, you can also use "kcidb-validate". Its output is
 >> somewhat more verbose than it should be, and we're working on that, but if you
 >> feed the JSON you sent to it like this:
 >>
 >>       kcidb-validate < syzbot.json
 >>
 >> somewhere closer to the end of the output you'll find:
 >>
 >>       ValidationError: None is not of type 'array'
 >>
 >>       Failed validating 'type' in schema['properties']['revisions']['items']['properties']['patch_mboxes']:
 >>
 >> It basically says that it didn't expect to find "null" as the value for the
 >> "patch_mboxes" property.
 >>
 >> In general "null"s are not accepted by the schema, to minimize ambiguity and
 >> the number of checks when interpreting the data. If you want to omit any data,
 >> just don't add the corresponding property.
 >>
 >> Other errors are:
 >>
 >> * "message_id" is specified as an empty string, but it should be an e-mail
 >>     address. Just omit the property if you have nothing to send.
 >> * "publishing_time" is an empty string and not an RFC3339 timestamp. Again,
 >>     just omit it, if you don't have it.
 >> * "log_url" should be a URL, but is an empty string. Same approach.
 >>
 >> After the above is fixed, the file will validate. However, there are still
 >> issues of the same sort - empty properties, which would be good to just
 >> remove: "command", "config_name", "description", "git_commit_name",
 >> "tree_name". Perhaps we should make the schema check those as well.
 >>
 >> I've fixed all of the issues in the JSON you provided and attached it as
 >> "syzbot_fixed.json".
 >
 > I managed to upload some data manually! Woohoo!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04
 >
 > Some questions regarding data conversion:
 >
 > 1. We usually first get just a crash on 1 instance. Then later we may
 > get a reproducer, or we see it happen on more trees, or we've got
 > bisection results, or something tested a proposed fix. How can we
 > "add" more to the same instance of the bug?
 >
 > 2. Maybe a related question: we generally identify bugs and bucket
 > crashes based on "crash title" (obtained from the kernel oops message
 > via some non-trivial transformation), e.g. "KASAN: use-after-free in
 > open_file". If we've got a reproducer, this string is used to link it
 > to the previously happened crash. I don't see where I can put this
 > string. test.description?
 >
 > 3. The build.id and test.id: should they just be unique? Or the system
 > needs to be able to map these IDs back to objects? In particular does
 > KCIDB need to generate links back to the external system?
 > E.g. for each bug we have a page with full info, like this one:
 > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
 > I think it would be useful for KCIDB to be able to provide this link.
 > However, the test.id generation that I am thinking of now will give
 > unique IDs, but we won't be able to map them back to anything useful.
 > Is it fine or not?
 > Currently I also add this link as a resource named "dashboard".  But
 > that's a custom name that KCIDB won't understand.
 >
 > 4. Should we add the kernel oops message itself inline? Or only via a link?
 > We have kernel oops messages nicely symbolized, cleaned of noise,
 > cleaned of unrelated threads, etc as you can see here:
 > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
 >
 > 5. We figure out emails (maintainers+lists) that we think should be
 > looking at this bug report. Where should we put them?
 > There is revision.contacts, but this is not related to revision but
 > rather test (or issue).
 >
 > 6. We have that tagged Reported-by line that we very much want
 > developers to use in the fixing commit to understand when bugs are
 > fixed.
 > If we detach them from info/reports available in KCIDB, then nobody
 > will use them in commits.
 > How can we pass it to KCIDB? Perhaps it's a useful feature for all
 > reports/test failures? If a bug is reported by Fuego and KCIDB
 > notifications/dashboard will ask to use "Reported-by: fuego", it will
 > give nice credits. We also found use of the hashes in the tag very
 > useful for statistics/tracking purposes. It makes it possible to link
 > failures to fixing commits and understand if a fix (1) exists in
 > general, (2) present in a particular tree, (3) or even if somebody
 > just mailed a proposed fix (useful to identify lost fixes that were
 > never merged, or avoid several people fixing the same bug if you can
 > show that somebody mailed a fix already).
 >

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Dmitry,
> On 9/29/20 10:52 AM, Dmitry Vyukov wrote:
>  > I managed to upload some data manually! Woohoo!
>  > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04
>
> Aweeesooomeee :)
>
>  > Some questions regarding data conversion:
>  >
>  > 1. We usually first get just a crash on 1 instance. Then later we may
>  > get a reproducer, or we see it happen on more trees, or we've got
>  > bisection results, or something tested a proposed fix. How can we
>  > "add" more to the same instance of the bug?
>
> Currently we don't have support for issues/incidents, so we can only accept
> test results without the issue context. That will have to do for the start.
>
> Once we have support for issues/incidents I think we would be able to use the
> feature I'm working on next - an ability to add previously-missing fields to
> an already-submitted object.
>
> So, you would send a test result object, then an issue object with minimal
> information, and then an incident object linking the two. As you get to know
> more about the issue, you can send it again, omitting the fields you already
> sent (or sending them with exact same values), but adding the
> previously-missing fields with the new data. I.e. you first send the issue
> without the proposed fix, and then send it again with the fix.
>
> This feature is still in the research stage, but I'll be working on it after
> the next kcidb release, which is coming in one or two weeks.
>
> Regarding bisection results I was hoping we could represent that data
> implicitly. I.e. by having information about which 'revision' comes after
> which, and executing a query to find a pair of adjacent revisions one of which
> has the bug and another doesn't. In theory that should be possible, but in
> practice... well, you know :) If that fails, we can attach that data
> explicitly somewhere. Or we can simply start with that to have it faster.

Humm... interesting. I guess it can work. We could mark the issue as
happening on the cause commit and as not happening on the previous
commit.
However, one problem is that we don't keep info about the previous
commit as of now, we only store the root cause commit in the database.
Another is that in git it's possible to have several preceding
commits.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 9/30/20 10:07 AM, Dmitry Vyukov wrote:
 > On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> Regarding bisection results I was hoping we could represent that data
 >> implicitly. I.e. by having information about which 'revision' comes after
 >> which, and executing a query to find a pair of adjacent revisions one of which
 >> has the bug and another doesn't. In theory that should be possible, but in
 >> practice... well, you know :) If that fails, we can attach that data
 >> explicitly somewhere. Or we can simply start with that to have it faster.
 >
 > Humm... interesting. I guess it can work. We could mark the issue as
 > happening on the cause commit and as not happening on the previous
 > commit.
 > However, one problem is that we don't keep info about the previous
 > commit as of now, we only store the root cause commit in the database.

Something has to maintain the information on history connectivity, and so far
the plan was to let the submitting system do it. E.g. refer to a clone of the
repo they're testing. There's definitely more research and design to be done.

However, you should've been testing the previous commit to be able to find the
root cause commit? Shouldn't you?

 > Another is that in git it's possible to have several preceding
 > commits.

Right, I didn't think through that yet. We might need a separate history link
object, then. Do you know if a commit can have more than two parents?

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 10:30 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 9/30/20 10:07 AM, Dmitry Vyukov wrote:
>  > On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >> Regarding bisection results I was hoping we could represent that data
>  >> implicitly. I.e. by having information about which 'revision' comes after
>  >> which, and executing a query to find a pair of adjacent revisions one of which
>  >> has the bug and another doesn't. In theory that should be possible, but in
>  >> practice... well, you know :) If that fails, we can attach that data
>  >> explicitly somewhere. Or we can simply start with that to have it faster.
>  >
>  > Humm... interesting. I guess it can work. We could mark the issue as
>  > happening on the cause commit and as not happening on the previous
>  > commit.
>  > However, one problem is that we don't keep info about the previous
>  > commit as of now, we only store the root cause commit in the database.
>
> Something has to maintain the information on history connectivity, and so far
> the plan was to let the submitting system do it. E.g. refer to a clone of the
> repo they're testing. There's definitely more research and design to be done.
>
> However, you should've been testing the previous commit to be able to find the
> root cause commit? Shouldn't you?

We do. But that happens in a very different part of the system and
that info is not kept in a structured form because it's not useful for
anything in our system.

Well, it actually turned out that it's not necessary to test all of
the parents of a commit during bisection to point at a particular
commit in some interesting git tree topologies.


>  > Another is that in git it's possible to have several preceding
>  > commits.
>
> Right, I didn't think through that yet. We might need a separate history link
> object, then. Do you know if a commit can have more than two parents?

Octopus merge commits can have lots of parents.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 11:43 AM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 10:30 AM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 9/30/20 10:07 AM, Dmitry Vyukov wrote:
 >>   > On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
 >>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   >> Regarding bisection results I was hoping we could represent that data
 >>   >> implicitly. I.e. by having information about which 'revision' comes after
 >>   >> which, and executing a query to find a pair of adjacent revisions one of which
 >>   >> has the bug and another doesn't. In theory that should be possible, but in
 >>   >> practice... well, you know :) If that fails, we can attach that data
 >>   >> explicitly somewhere. Or we can simply start with that to have it faster.
 >>   >
 >>   > Humm... interesting. I guess it can work. We could mark the issue as
 >>   > happening on the cause commit and as not happening on the previous
 >>   > commit.
 >>   > However, one problem is that we don't keep info about the previous
 >>   > commit as of now, we only store the root cause commit in the database.
 >>
 >> Something has to maintain the information on history connectivity, and so far
 >> the plan was to let the submitting system do it. E.g. refer to a clone of the
 >> repo they're testing. There's definitely more research and design to be done.
 >>
 >> However, you should've been testing the previous commit to be able to find the
 >> root cause commit? Shouldn't you?
 >
 > We do. But that happens in a very different part of the system and
 > that info is not kept in a structured form because it's not useful for
 > anything in our system.
 >
 > Well, it actually turned out that it's not necessary to test all of
 > the parents of a commit during bisection to point at a particular
 > commit in some interesting git tree topologies.

Ah, I see I'll need to ask you more questions about bisections once we get to
that :)

 >>   > Another is that in git it's possible to have several preceding
 >>   > commits.
 >>
 >> Right, I didn't think through that yet. We might need a separate history link
 >> object, then. Do you know if a commit can have more than two parents?
 >
 > Octopus merge commits can have lots of parents.

Right, I forgot about those.

Thank you!
Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>  > 2. Maybe a related question: we generally identify bugs and bucket
>  > crashes based on "crash title" (obtained from the kernel oops message
>  > via some non-trivial transformation), e.g. "KASAN: use-after-free in
>  > open_file". If we've got a reproducer, this string is used to link it
>  > to the previously happened crash. I don't see where I can put this
>  > string. test.description?
>
> So far my idea was to leave the identification and handling of such strings to
> you. We could then represent that link using an incident object linking a test
> execution to an issue, and the issue could then have the reproducer
> linked/attached in a property.
>
> For now, you can put that information into the "description" (which I plan to
> rename into "comment" sometime soon), or perhaps into the "misc" object.

Wouldn't it be useful to have at least some information about the
actual kernel oops in kcidb?
It would allow searches and interconnect things reported by different
systems. E.g. are there any known issues in this function? Or
understanding that several tests fail due to the same root cause. Or
doing some flakiness analysis, e.g. episodic failures on random tests
but all have similar kernel oopses. Or interconnecting bugs reported
by different syzbot deployments (yes, there are others, I don't know
if they will want to participate, but with support in syzbot code they
will just need to flip a config flag).

>  > 3. The build.id and test.id: should they just be unique?
>
> They need to be unique among all the corresponding objects you're submitting.
> That's the only requirement.
>
>  > Or the system
>  > needs to be able to map these IDs back to objects? In particular does
>  > KCIDB need to generate links back to the external system?
>  > E.g. for each bug we have a page with full info, like this one:
>  > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
>  > I think it would be useful for KCIDB to be able to provide this link.
>
> We're considering that, and I agree it could be useful.
> See https://github.com/kernelci/kcidb/issues/93
>
>  > However, the test.id generation that I am thinking of now will give
>  > unique IDs, but we won't be able to map them back to anything useful.
>  > Is it fine or not?
>
> It's OK, as long as it's unique.
>
>  > Currently I also add this link as a resource named "dashboard".  But
>  > that's a custom name that KCIDB won't understand.
>
> We have the "misc" object specifically for things like that: data you'd like
> to see represented in an object, but which isn't supported by the schema yet.
>
> You can put there whatever you want (within reasonable size limit), dashboard
> users can get it now, and we can have a sample of what we need to support in
> the schema.
>
> See https://github.com/kernelci/kcidb/blob/master/SUBMISSION_HOWTO.md#extra-data

Got it.
Yes, some support for "link to external dashboard" would be useful.
Perhaps just a canonical name for resource?


>  > 4. Should we add the kernel oops message itself inline? Or only via a link?
>  > We have kernel oops messages nicely symbolized, cleaned of noise,
>  > cleaned of unrelated threads, etc as you can see here:
>  > https://syzkaller.appspot.com/bug?extid=8afc256dce324523609d
>
> Yes, those are very nice! We're planning to add support for embedding log
> snippets and such into reports (0day needs that too), but for now you'll
> have to submit that as a link to a file hosted somewhere, via test's
> "output_files".
>
>  > 5. We figure out emails (maintainers+lists) that we think should be
>  > looking at this bug report. Where should we put them?
>  > There is revision.contacts, but this is not related to revision but
>  > rather test (or issue).
>
> Yes, there's revision.contacts, which comes from initial CKI needs, and we
> could add the same to the issues, when we have them.
>
> However, with KCIDB, I'd like to try a reverse approach: instead of the CI
> system deciding who to send reports to, give developers control on what they
> receive via a "subscription" system. I.e. let them pick what they'll be
> interested in: which CI systems, which trees, architectures, tests, results,
> and so on. With that approach such data becomes unnecessary.
>
> There's, of course many questions to answer and a proper implementation to
> write before we know how that works, and perhaps we could start with using
> that data, but that's the direction I'd like to try.
>
> With many people submitting and the amount of data we're already getting we
> need to give the developers the upper hand. I think it should be up to the CI
> systems to submit the best data they can, up to KCIDB to provide
> discoverability of that data, and up to developers to decide which data they
> care about.

I see. Makes sense.
But then we need to ensure that kcidb has the necessary info for this.
Again, fuzzing is different from unit testing in this regard. We
effectively have 1 test case "syzkaller" and crashes happen on an
"upstream" tree and the commit we point to is not related to the crash
(it just describes the state of the tree). That's no useful
information to subscribe to at all.

The way we do it is as follows. We analyze the oops message to find
what we assume is the guilty source file and then run
get_maintainers.pl on the file. I assume it's very reasonable for
developers to subscribe to parts of the source tree they are
interested in (e.g. net/sctp/*). But for this kcidb will need either
the file name or the oops message (in machine discoverable form).

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 9/30/20 10:21 AM, Dmitry Vyukov wrote:
 > On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   > 2. Maybe a related question: we generally identify bugs and bucket
 >>   > crashes based on "crash title" (obtained from the kernel oops message
 >>   > via some non-trivial transformation), e.g. "KASAN: use-after-free in
 >>   > open_file". If we've got a reproducer, this string is used to link it
 >>   > to the previously happened crash. I don't see where I can put this
 >>   > string. test.description?
 >>
 >> So far my idea was to leave the identification and handling of such strings to
 >> you. We could then represent that link using an incident object linking a test
 >> execution to an issue, and the issue could then have the reproducer
 >> linked/attached in a property.
 >>
 >> For now, you can put that information into the "description" (which I plan to
 >> rename into "comment" sometime soon), or perhaps into the "misc" object.
 >
 > Wouldn't it be useful to have at least some information about the
 > actual kernel oops in kcidb?
 > It would allow searches and interconnect things reported by different
 > systems. E.g. are there any known issues in this function? Or
 > understanding that several tests fail due to the same root cause. Or
 > doing some flakiness analysis, e.g. episodic failures on random tests
 > but all have similar kernel oopses. Or interconnecting bugs reported
 > by different syzbot deployments

Absolutely, that would be useful. We just had to start doing this without
implementing everything at once. We have a plan to allow submitting short,
most-relevant log/file snippets into the database directly (I probably
mentioned that before). E.g. 0day would need that from the start.

For now we only take links to files, but hopefully snippets soon :)

 > (yes, there are others, I don't know if they will want to participate, but
 > with support in syzbot code they will just need to flip a config flag).

Awesooomeeee, this situation is exactly what we're aiming at :D

 >>   > Currently I also add this link as a resource named "dashboard".  But
 >>   > that's a custom name that KCIDB won't understand.
 >>
 >> We have the "misc" object specifically for things like that: data you'd like
 >> to see represented in an object, but which isn't supported by the schema yet.
 >>
 >> You can put there whatever you want (within reasonable size limit), dashboard
 >> users can get it now, and we can have a sample of what we need to support in
 >> the schema.
 >>
 >> See https://github.com/kernelci/kcidb/blob/master/SUBMISSION_HOWTO.md#extra-data
 >
 > Got it.
 > Yes, some support for "link to external dashboard" would be useful.
 > Perhaps just a canonical name for resource?

At the moment resources are intended for files, which could e.g. be downloaded
and re-hosted on KCIDB side, if necessary. Page links shouldn't go in there.

Ping Guillaume in the associated PR, and explain your requirements to nudge
him to take a look at it again, and perhaps we can get this moved forward
sooner :)

That PR needs to move to another repo now, but it's good enough to continue
the discussion.

 >>   > 5. We figure out emails (maintainers+lists) that we think should be
 >>   > looking at this bug report. Where should we put them?
 >>   > There is revision.contacts, but this is not related to revision but
 >>   > rather test (or issue).
 >>
 >> Yes, there's revision.contacts, which comes from initial CKI needs, and we
 >> could add the same to the issues, when we have them.
 >>
 >> However, with KCIDB, I'd like to try a reverse approach: instead of the CI
 >> system deciding who to send reports to, give developers control on what they
 >> receive via a "subscription" system. I.e. let them pick what they'll be
 >> interested in: which CI systems, which trees, architectures, tests, results,
 >> and so on. With that approach such data becomes unnecessary.
 >>
 >> There's, of course many questions to answer and a proper implementation to
 >> write before we know how that works, and perhaps we could start with using
 >> that data, but that's the direction I'd like to try.
 >>
 >> With many people submitting and the amount of data we're already getting we
 >> need to give the developers the upper hand. I think it should be up to the CI
 >> systems to submit the best data they can, up to KCIDB to provide
 >> discoverability of that data, and up to developers to decide which data they
 >> care about.
 >
 > I see. Makes sense.
 > But then we need to ensure that kcidb has the necessary info for this.
 > Again, fuzzing is different from unit testing in this regard. We
 > effectively have 1 test case "syzkaller" and crashes happen on an
 > "upstream" tree and the commit we point to is not related to the crash
 > (it just describes the state of the tree). That's no useful
 > information to subscribe to at all.
 >
 > The way we do it is as follows. We analyze the oops message to find
 > what we assume is the guilty source file and then run
 > get_maintainers.pl on the file. I assume it's very reasonable for
 > developers to subscribe to parts of the source tree they are
 > interested in (e.g. net/sctp/*). But for this kcidb will need either
 > the file name or the oops message (in machine discoverable form).

Yes, the file where the fault happened could definitely be useful. This needs
more thinking, and I think we have too big an implementation gap for that yet.

For the start, we can perhaps let people subscribe to test failures which
happened to revisions with parents which don't have those same failures. Plus
e.g. let them check that the failure has a bug with a reproducer, or such.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>  > 6. We have that tagged Reported-by line that we very much want
>  > developers to use in the fixing commit to understand when bugs are
>  > fixed.
>  > If we detach them from info/reports available in KCIDB, then nobody
>  > will use them in commits.
>  > How can we pass it to KCIDB? Perhaps it's a useful feature for all
>  > reports/test failures? If a bug is reported by Fuego and KCIDB
>  > notifications/dashboard will ask to use "Reported-by: fuego", it will
>  > give nice credits. We also found use of the hashes in the tag very
>  > useful for statistics/tracking purposes. It makes it possible to link
>  > failures to fixing commits and understand if a fix (1) exists in
>  > general, (2) present in a particular tree, (3) or even if somebody
>  > just mailed a proposed fix (useful to identify lost fixes that were
>  > never merged, or avoid several people fixing the same bug if you can
>  > show that somebody mailed a fix already).
>
> Hmm, yes, that could be very useful indeed, thank you.
>
> We would have to use an ID managed by KCIDB, though. This brings me to the
> question what we're going to use for identifying issues. Perhaps use the
> submitter-supplied ID, similarly to builds and tests right now, under the
> assumption that, at least for the start, issues are not going to be reused
> across CI systems. If that's so, then you would be able to map those IDs back
> to your issues.

If we report a failed test/issue with "id": "syzbot:d195fe572fb15368",
then if the commit includes:
Reported-by: syzbot:d195fe572fb15368
I think it should work for us.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 9/30/20 10:31 AM, Dmitry Vyukov wrote:
 > On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   > 6. We have that tagged Reported-by line that we very much want
 >>   > developers to use in the fixing commit to understand when bugs are
 >>   > fixed.
 >>   > If we detach them from info/reports available in KCIDB, then nobody
 >>   > will use them in commits.
 >>   > How can we pass it to KCIDB? Perhaps it's a useful feature for all
 >>   > reports/test failures? If a bug is reported by Fuego and KCIDB
 >>   > notifications/dashboard will ask to use "Reported-by: fuego", it will
 >>   > give nice credits. We also found use of the hashes in the tag very
 >>   > useful for statistics/tracking purposes. It makes it possible to link
 >>   > failures to fixing commits and understand if a fix (1) exists in
 >>   > general, (2) present in a particular tree, (3) or even if somebody
 >>   > just mailed a proposed fix (useful to identify lost fixes that were
 >>   > never merged, or avoid several people fixing the same bug if you can
 >>   > show that somebody mailed a fix already).
 >>
 >> Hmm, yes, that could be very useful indeed, thank you.
 >>
 >> We would have to use an ID managed by KCIDB, though. This brings me to the
 >> question what we're going to use for identifying issues. Perhaps use the
 >> submitter-supplied ID, similarly to builds and tests right now, under the
 >> assumption that, at least for the start, issues are not going to be reused
 >> across CI systems. If that's so, then you would be able to map those IDs back
 >> to your issues.
 >
 > If we report a failed test/issue with "id": "syzbot:d195fe572fb15368",
 > then if the commit includes:
 > Reported-by: syzbot:d195fe572fb15368
 > I think it should work for us.

Not sure about using "Reported-by" for such a value, but, yeah, I was thinking
along these lines too.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Wed, Sep 30, 2020 at 9:32 AM Dmitry Vyukov via groups.io
<dvyukov=google.com@groups.io> wrote:
>
> On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
> >  > 6. We have that tagged Reported-by line that we very much want
> >  > developers to use in the fixing commit to understand when bugs are
> >  > fixed.
> >  > If we detach them from info/reports available in KCIDB, then nobody
> >  > will use them in commits.
> >  > How can we pass it to KCIDB? Perhaps it's a useful feature for all
> >  > reports/test failures? If a bug is reported by Fuego and KCIDB
> >  > notifications/dashboard will ask to use "Reported-by: fuego", it will
> >  > give nice credits. We also found use of the hashes in the tag very
> >  > useful for statistics/tracking purposes. It makes it possible to link
> >  > failures to fixing commits and understand if a fix (1) exists in
> >  > general, (2) present in a particular tree, (3) or even if somebody
> >  > just mailed a proposed fix (useful to identify lost fixes that were
> >  > never merged, or avoid several people fixing the same bug if you can
> >  > show that somebody mailed a fix already).
> >
> > Hmm, yes, that could be very useful indeed, thank you.
> >
> > We would have to use an ID managed by KCIDB, though. This brings me to the
> > question what we're going to use for identifying issues. Perhaps use the
> > submitter-supplied ID, similarly to builds and tests right now, under the
> > assumption that, at least for the start, issues are not going to be reused
> > across CI systems. If that's so, then you would be able to map those IDs back
> > to your issues.
>
> If we report a failed test/issue with "id": "syzbot:d195fe572fb15368",
> then if the commit includes:
> Reported-by: syzbot:d195fe572fb15368
> I think it should work for us.

So we are alive!
https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&from=now-90d&to=now

Here is syzkaller PR (will be merged as soon as CI passes):
https://github.com/google/syzkaller/pull/2150

Please check if the data it uploads look sane.
Also please check for server errors. There may be few produced with
earlier versions of the code, but the latest code uploaded the
majority of data.

You may skim through the syzkaller bits to check how we fill data,
this is the most interesting part:
https://github.com/google/syzkaller/pull/2150/commits/766714e5dcf6ceb6bab69754173f6f8f314d78c6
Build errors and runtime errors are handled slightly differently, we
also add some info into the misc objects (Report-by tag, bisection
result, userspace arch).

Re contributing Go support to Kcidb.
The main thing that is useful is the schema converted to Go types, see
the same link above. It was produced automatically with
github.com/a-h/generate, but then slightly modified by hand.

The rest is effectively 3 lines of code to push it:
client, err := pubsub.NewClient(ctx, project,
option.WithCredentialsJSON(credentials))
topic:  client.Topic(topic),
_, err = c.topic.Publish(c.ctx, &pubsub.Message{Data: data}).Get(c.ctx)

I don't think these 3 lines of code are worth factoring out, but of
course we can point people to the syzkaller code as an example
implementation.
The Go schema may be worth contributing if we decide to detach it from
the original source and keep maintaining manually. A better long term
solution may be to write/find a proper converter that produces working
code.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 9/30/20 7:07 PM, Dmitry Vyukov wrote:
 > So we are alive!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&from=now-90d&to=now

Woo-hoo!

 > Here is syzkaller PR (will be merged as soon as CI passes):
 > https://github.com/google/syzkaller/pull/2150

Awesome, and I see it's merged already :)

 > Please check if the data it uploads look sane.

Alright, there are a few things to improve, which is fine, but you did an
awesome job setting up automatic submission so quickly! None of those things
should prevent you switching to production, and I already added the necessary
permissions to your account.

Just switch to submitting to the "kernelci_new" topic when you're ready.

Here are the things which could be improved:

* To be able to correlate Git repos across submitters, and to make subscribing
   easier, we have to normalize repo URLs. That's why in the schema docs we're
   asking to use https:// repo URLs, and only use git:// in the rare case
   https:// is not available (we only have one so far).

   E.g. you're sending the stable repo URL as:

     git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

   and we'd prefer it sent as:

     https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

   This is also the reason we didn't get e-mail reports for your
   submissions - the PoC stable tree subscription expects https://

   Ah, that, and we also weren't subscribed to stable/linux-stable.git, just
   sashal/linux-stable.git, but it should be fixed soon :)

* Could you move the "dashboard" link out of test's "output_files", and into
   a "misc" field? It won't be presented as a link in Grafana unfortunately,
   but "output_files" is for something which could be useful when downloaded
   (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
   Let's deal with the dashboard link issue in
   https://github.com/kernelci/kcidb/issues/93

* Could you add `"waived": false` field to the tests? This way the system will
   know to take your test results seriously, and Grafana will color the status
   fields nicely and not display that question mark in them.

   We can adjust the subscriptions to ignore syzbot tests and not send failure
   reports until we get the issue/incident system figured out.

* Could you submit a PR adding syzkaller to
   https://github.com/kernelci/kcidb/blob/master/tests.yaml
   so its canonical name is registered?

We can, of course, use more fields filled in, such as build logs and output
files, or environment descriptions, but that can be done later.
Thank you for your awesome contribution!

 > Also please check for server errors. There may be few produced with
 > earlier versions of the code, but the latest code uploaded the
 > majority of data.

Nope, couldn't find anything caused by your submissions :)

 > You may skim through the syzkaller bits to check how we fill data,
 > this is the most interesting part:
 > https://github.com/google/syzkaller/pull/2150/commits/766714e5dcf6ceb6bab69754173f6f8f314d78c6
 >
 > Build errors and runtime errors are handled slightly differently, we
 > also add some info into the misc objects (Report-by tag, bisection
 > result, userspace arch).
 >
 > Re contributing Go support to Kcidb.
 > The main thing that is useful is the schema converted to Go types, see
 > the same link above. It was produced automatically with
 > github.com/a-h/generate, but then slightly modified by hand.

Nice!

 > The rest is effectively 3 lines of code to push it:
 > client, err := pubsub.NewClient(ctx, project,
 > option.WithCredentialsJSON(credentials))
 > topic:  client.Topic(topic),
 > _, err = c.topic.Publish(c.ctx, &pubsub.Message{Data: data}).Get(c.ctx)
 >
 > I don't think these 3 lines of code are worth factoring out, but of
 > course we can point people to the syzkaller code as an example
 > implementation.

We need a module providing an abstract interface to submitting (at least), so
people who use Go don't have to rewrite their code (much) if we e.g. change
our cloud architecture, provider, or even move off-cloud. Something similar to
https://github.com/kernelci/kcidb/blob/master/kcidb/__init__.py#L36

 > The Go schema may be worth contributing if we decide to detach it from
 > the original source and keep maintaining manually. A better long term
 > solution may be to write/find a proper converter that produces working
 > code.

Yes, we need to maintain a single source for the schema and generate the rest
from that. We can switch to publishing the schema as a JSON file instead of
(more readable) Python code generating it, if it helps you to generate your
module. Actually, maybe a YAML file could be a good compromise of readability
vs accessibility.

Nick

On 9/30/20 7:07 PM, Dmitry Vyukov wrote:
 > On Wed, Sep 30, 2020 at 9:32 AM Dmitry Vyukov via groups.io
 > <dvyukov=google.com@groups.io> wrote:
 >>
 >> On Tue, Sep 29, 2020 at 7:13 PM Nikolai Kondrashov
 >> <Nikolai.Kondrashov@redhat.com> wrote:
 >>>   > 6. We have that tagged Reported-by line that we very much want
 >>>   > developers to use in the fixing commit to understand when bugs are
 >>>   > fixed.
 >>>   > If we detach them from info/reports available in KCIDB, then nobody
 >>>   > will use them in commits.
 >>>   > How can we pass it to KCIDB? Perhaps it's a useful feature for all
 >>>   > reports/test failures? If a bug is reported by Fuego and KCIDB
 >>>   > notifications/dashboard will ask to use "Reported-by: fuego", it will
 >>>   > give nice credits. We also found use of the hashes in the tag very
 >>>   > useful for statistics/tracking purposes. It makes it possible to link
 >>>   > failures to fixing commits and understand if a fix (1) exists in
 >>>   > general, (2) present in a particular tree, (3) or even if somebody
 >>>   > just mailed a proposed fix (useful to identify lost fixes that were
 >>>   > never merged, or avoid several people fixing the same bug if you can
 >>>   > show that somebody mailed a fix already).
 >>>
 >>> Hmm, yes, that could be very useful indeed, thank you.
 >>>
 >>> We would have to use an ID managed by KCIDB, though. This brings me to the
 >>> question what we're going to use for identifying issues. Perhaps use the
 >>> submitter-supplied ID, similarly to builds and tests right now, under the
 >>> assumption that, at least for the start, issues are not going to be reused
 >>> across CI systems. If that's so, then you would be able to map those IDs back
 >>> to your issues.
 >>
 >> If we report a failed test/issue with "id": "syzbot:d195fe572fb15368",
 >> then if the commit includes:
 >> Reported-by: syzbot:d195fe572fb15368
 >> I think it should work for us.
 >
 > So we are alive!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&var-origin=syzbot&var-git_repository_url=All&var-dataset=playground_kernelci04&from=now-90d&to=now
 >
 > Here is syzkaller PR (will be merged as soon as CI passes):
 > https://github.com/google/syzkaller/pull/2150
 >
 > Please check if the data it uploads look sane.
 > Also please check for server errors. There may be few produced with
 > earlier versions of the code, but the latest code uploaded the
 > majority of data.
 >
 > You may skim through the syzkaller bits to check how we fill data,
 > this is the most interesting part:
 > https://github.com/google/syzkaller/pull/2150/commits/766714e5dcf6ceb6bab69754173f6f8f314d78c6
 > Build errors and runtime errors are handled slightly differently, we
 > also add some info into the misc objects (Report-by tag, bisection
 > result, userspace arch).
 >
 > Re contributing Go support to Kcidb.
 > The main thing that is useful is the schema converted to Go types, see
 > the same link above. It was produced automatically with
 > github.com/a-h/generate, but then slightly modified by hand.
 >
 > The rest is effectively 3 lines of code to push it:
 > client, err := pubsub.NewClient(ctx, project,
 > option.WithCredentialsJSON(credentials))
 > topic:  client.Topic(topic),
 > _, err = c.topic.Publish(c.ctx, &pubsub.Message{Data: data}).Get(c.ctx)
 >
 > I don't think these 3 lines of code are worth factoring out, but of
 > course we can point people to the syzkaller code as an example
 > implementation.
 > The Go schema may be worth contributing if we decide to detach it from
 > the original source and keep maintaining manually. A better long term
 > solution may be to write/find a proper converter that produces working
 > code.
 >

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
 > Here are the things which could be improved:

Oh, and another thing: could you avoid re-sending the identical
revisions/builds you've already sent? I.e. send them only once, unless you
have fields to add? Each of those costs us a row in the database, which is not
a big deal, but would be good to avoid.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
>  > Here are the things which could be improved:
>
> Oh, and another thing: could you avoid re-sending the identical
> revisions/builds you've already sent? I.e. send them only once, unless you
> have fields to add? Each of those costs us a row in the database, which is not
> a big deal, but would be good to avoid.

Do you see that we send dups? I've added logic to not send dups and
from what I see it's working.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
 > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
 >>   > Here are the things which could be improved:
 >>
 >> Oh, and another thing: could you avoid re-sending the identical
 >> revisions/builds you've already sent? I.e. send them only once, unless you
 >> have fields to add? Each of those costs us a row in the database, which is not
 >> a big deal, but would be good to avoid.
 >
 > Do you see that we send dups? I've added logic to not send dups and
 > from what I see it's working.

Please find attached a zip archive with seven submissions from about six hours
ago. Each has revisions with the same id and git_commit_hash. Some of those
revisions have different discovery_time, and that alone shouldn't really be
the reason to resend. Each of those submissions have a build object, but some
of them are repeated.

Revision with that git_commit_hash was submitted at least 72 times.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On 10/1/20 6:49 PM, Nikolai Kondrashov wrote:
> On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
>  > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
>  >>   > Here are the things which could be improved:
>  >>
>  >> Oh, and another thing: could you avoid re-sending the identical
>  >> revisions/builds you've already sent? I.e. send them only once, unless you
>  >> have fields to add? Each of those costs us a row in the database, which is not
>  >> a big deal, but would be good to avoid.
>  >
>  > Do you see that we send dups? I've added logic to not send dups and
>  > from what I see it's working.
> 
> Please find attached a zip archive with seven submissions from about six hours
> ago. Each has revisions with the same id and git_commit_hash. Some of those
> revisions have different discovery_time, and that alone shouldn't really be
> the reason to resend. Each of those submissions have a build object, but some
> of them are repeated.
> 
> Revision with that git_commit_hash was submitted at least 72 times.

And now the file is attached :D

[-- Attachment #2: commit_hash_dups.tar.gz --]
[-- Type: application/gzip, Size: 1583 bytes --]

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 5:52 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/1/20 6:49 PM, Nikolai Kondrashov wrote:
> > On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
> >  > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
> >  > <Nikolai.Kondrashov@redhat.com> wrote:
> >  >>
> >  >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
> >  >>   > Here are the things which could be improved:
> >  >>
> >  >> Oh, and another thing: could you avoid re-sending the identical
> >  >> revisions/builds you've already sent? I.e. send them only once, unless you
> >  >> have fields to add? Each of those costs us a row in the database, which is not
> >  >> a big deal, but would be good to avoid.
> >  >
> >  > Do you see that we send dups? I've added logic to not send dups and
> >  > from what I see it's working.
> >
> > Please find attached a zip archive with seven submissions from about six hours
> > ago. Each has revisions with the same id and git_commit_hash. Some of those
> > revisions have different discovery_time, and that alone shouldn't really be
> > the reason to resend. Each of those submissions have a build object, but some
> > of them are repeated.
> >
> > Revision with that git_commit_hash was submitted at least 72 times.
>
> And now the file is attached :D

This is intentional (in the current implementation) and is a
consequence of the fact that we always send all 3 entities for each
issue/test failure. It's much simpler on our side this way. If we
would send a test failure multiple times it would be unintentional.

Identical builds should have the same id, though. And if they have
different id's, then these are different builds.

Revisions were discovered separately by different instances, that's
why they have different discovery time. Consider that as if different
origin systems would discover it separately. Since discovery time is
not an inherent property of the commit itself, there is no way they
can agree on it. Will it be better if we don't send discovery time at
all?

Amount of duplication for builds/revisions is capped by the number of
bugs we discover, since build/revision is only sent with a bug once.
So it's not that it's unlimited.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 7:00 PM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 5:52 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 10/1/20 6:49 PM, Nikolai Kondrashov wrote:
 >>> On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
 >>>   > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
 >>>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>>   >>
 >>>   >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
 >>>   >>   > Here are the things which could be improved:
 >>>   >>
 >>>   >> Oh, and another thing: could you avoid re-sending the identical
 >>>   >> revisions/builds you've already sent? I.e. send them only once, unless you
 >>>   >> have fields to add? Each of those costs us a row in the database, which is not
 >>>   >> a big deal, but would be good to avoid.
 >>>   >
 >>>   > Do you see that we send dups? I've added logic to not send dups and
 >>>   > from what I see it's working.
 >>>
 >>> Please find attached a zip archive with seven submissions from about six hours
 >>> ago. Each has revisions with the same id and git_commit_hash. Some of those
 >>> revisions have different discovery_time, and that alone shouldn't really be
 >>> the reason to resend. Each of those submissions have a build object, but some
 >>> of them are repeated.
 >>>
 >>> Revision with that git_commit_hash was submitted at least 72 times.
 >>
 >> And now the file is attached :D
 >
 > This is intentional (in the current implementation) and is a
 > consequence of the fact that we always send all 3 entities for each
 > issue/test failure. It's much simpler on our side this way. If we
 > would send a test failure multiple times it would be unintentional.

I see.

 > Identical builds should have the same id, though. And if they have
 > different id's, then these are different builds.

Sure.

 > Revisions were discovered separately by different instances, that's
 > why they have different discovery time. Consider that as if different
 > origin systems would discover it separately. Since discovery time is
 > not an inherent property of the commit itself, there is no way they
 > can agree on it.

Ah, I see now why. Yeah, it's similar to multiple systems submitting the same
revision. Although we don't have systems submitting the same build multiple
times.

 > Will it be better if we don't send discovery time at all?

No, it's OK to submit the discover time multiple times, but we can't say which
one will be displayed :)

 > Amount of duplication for builds/revisions is capped by the number of
 > bugs we discover, since build/revision is only sent with a bug once.
 > So it's not that it's unlimited.

I think it's OK that you have some duplication, and certainly OK for the
start. Let's get on with sending the data and see how much actual duplication
we get, how it affects performance, and how the de-duplication implementation
goes :)

Thank you!
Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 6:34 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/1/20 7:00 PM, Dmitry Vyukov wrote:
>  > On Thu, Oct 1, 2020 at 5:52 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> On 10/1/20 6:49 PM, Nikolai Kondrashov wrote:
>  >>> On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
>  >>>   > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
>  >>>   > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>>   >>
>  >>>   >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
>  >>>   >>   > Here are the things which could be improved:
>  >>>   >>
>  >>>   >> Oh, and another thing: could you avoid re-sending the identical
>  >>>   >> revisions/builds you've already sent? I.e. send them only once, unless you
>  >>>   >> have fields to add? Each of those costs us a row in the database, which is not
>  >>>   >> a big deal, but would be good to avoid.
>  >>>   >
>  >>>   > Do you see that we send dups? I've added logic to not send dups and
>  >>>   > from what I see it's working.
>  >>>
>  >>> Please find attached a zip archive with seven submissions from about six hours
>  >>> ago. Each has revisions with the same id and git_commit_hash. Some of those
>  >>> revisions have different discovery_time, and that alone shouldn't really be
>  >>> the reason to resend. Each of those submissions have a build object, but some
>  >>> of them are repeated.
>  >>>
>  >>> Revision with that git_commit_hash was submitted at least 72 times.
>  >>
>  >> And now the file is attached :D
>  >
>  > This is intentional (in the current implementation) and is a
>  > consequence of the fact that we always send all 3 entities for each
>  > issue/test failure. It's much simpler on our side this way. If we
>  > would send a test failure multiple times it would be unintentional.
>
> I see.
>
>  > Identical builds should have the same id, though. And if they have
>  > different id's, then these are different builds.
>
> Sure.
>
>  > Revisions were discovered separately by different instances, that's
>  > why they have different discovery time. Consider that as if different
>  > origin systems would discover it separately. Since discovery time is
>  > not an inherent property of the commit itself, there is no way they
>  > can agree on it.
>
> Ah, I see now why. Yeah, it's similar to multiple systems submitting the same
> revision. Although we don't have systems submitting the same build multiple
> times.
>
>  > Will it be better if we don't send discovery time at all?
>
> No, it's OK to submit the discover time multiple times, but we can't say which
> one will be displayed :)
>
>  > Amount of duplication for builds/revisions is capped by the number of
>  > bugs we discover, since build/revision is only sent with a bug once.
>  > So it's not that it's unlimited.
>
> I think it's OK that you have some duplication, and certainly OK for the
> start. Let's get on with sending the data and see how much actual duplication
> we get, how it affects performance, and how the de-duplication implementation
> goes :)

The new format is now published by syzbot itself:
https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzbot:60b38542a0dfb2e0f746

So, do we switch to prod instance? Do you receive notifications now?
Do we need to fix anything else before the switch?

I don't see anything from syzbot at:
https://groups.io/g/kernelci-results-staging/topics
How can I see/subscribe to these notifications?

Since syzbot database now memorizes what was published to kcidb to
avoid dups, the switch will require some manual database surgery to
make it re-publish bugs. So I want to make sure we did not miss
something simple that we could do before the switch.

The switch involves just changing the topic to "kernelci_new", right?

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Thu, Oct 1, 2020 at 6:34 PM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
> >
> > On 10/1/20 7:00 PM, Dmitry Vyukov wrote:
> >  > On Thu, Oct 1, 2020 at 5:52 PM Nikolai Kondrashov
> >  > <Nikolai.Kondrashov@redhat.com> wrote:
> >  >>
> >  >> On 10/1/20 6:49 PM, Nikolai Kondrashov wrote:
> >  >>> On 10/1/20 5:48 PM, Dmitry Vyukov via groups.io wrote:
> >  >>>   > On Thu, Oct 1, 2020 at 3:32 PM Nikolai Kondrashov
> >  >>>   > <Nikolai.Kondrashov@redhat.com> wrote:
> >  >>>   >>
> >  >>>   >> On 10/1/20 1:48 PM, Nikolai Kondrashov wrote:
> >  >>>   >>   > Here are the things which could be improved:
> >  >>>   >>
> >  >>>   >> Oh, and another thing: could you avoid re-sending the identical
> >  >>>   >> revisions/builds you've already sent? I.e. send them only once, unless you
> >  >>>   >> have fields to add? Each of those costs us a row in the database, which is not
> >  >>>   >> a big deal, but would be good to avoid.
> >  >>>   >
> >  >>>   > Do you see that we send dups? I've added logic to not send dups and
> >  >>>   > from what I see it's working.
> >  >>>
> >  >>> Please find attached a zip archive with seven submissions from about six hours
> >  >>> ago. Each has revisions with the same id and git_commit_hash. Some of those
> >  >>> revisions have different discovery_time, and that alone shouldn't really be
> >  >>> the reason to resend. Each of those submissions have a build object, but some
> >  >>> of them are repeated.
> >  >>>
> >  >>> Revision with that git_commit_hash was submitted at least 72 times.
> >  >>
> >  >> And now the file is attached :D
> >  >
> >  > This is intentional (in the current implementation) and is a
> >  > consequence of the fact that we always send all 3 entities for each
> >  > issue/test failure. It's much simpler on our side this way. If we
> >  > would send a test failure multiple times it would be unintentional.
> >
> > I see.
> >
> >  > Identical builds should have the same id, though. And if they have
> >  > different id's, then these are different builds.
> >
> > Sure.
> >
> >  > Revisions were discovered separately by different instances, that's
> >  > why they have different discovery time. Consider that as if different
> >  > origin systems would discover it separately. Since discovery time is
> >  > not an inherent property of the commit itself, there is no way they
> >  > can agree on it.
> >
> > Ah, I see now why. Yeah, it's similar to multiple systems submitting the same
> > revision. Although we don't have systems submitting the same build multiple
> > times.
> >
> >  > Will it be better if we don't send discovery time at all?
> >
> > No, it's OK to submit the discover time multiple times, but we can't say which
> > one will be displayed :)
> >
> >  > Amount of duplication for builds/revisions is capped by the number of
> >  > bugs we discover, since build/revision is only sent with a bug once.
> >  > So it's not that it's unlimited.
> >
> > I think it's OK that you have some duplication, and certainly OK for the
> > start. Let's get on with sending the data and see how much actual duplication
> > we get, how it affects performance, and how the de-duplication implementation
> > goes :)
>
> The new format is now published by syzbot itself:
> https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzbot:60b38542a0dfb2e0f746
>
> So, do we switch to prod instance? Do you receive notifications now?
> Do we need to fix anything else before the switch?
>
> I don't see anything from syzbot at:
> https://groups.io/g/kernelci-results-staging/topics
> How can I see/subscribe to these notifications?
>
> Since syzbot database now memorizes what was published to kcidb to
> avoid dups, the switch will require some manual database surgery to
> make it re-publish bugs. So I want to make sure we did not miss
> something simple that we could do before the switch.
>
> The switch involves just changing the topic to "kernelci_new", right?

I am preparing for the switch and will do it unless I hear from you soon.
Since you already gave green light above, I assume we are good now.

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/2/20 10:52 AM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >> The switch involves just changing the topic to "kernelci_new", right?
 >
 > I am preparing for the switch and will do it unless I hear from you soon.
 > Since you already gave green light above, I assume we are good now.

Yes, please go ahead :)!

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Fri, Oct 2, 2020 at 10:12 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/2/20 10:52 AM, Dmitry Vyukov wrote:
>  > On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>  >> The switch involves just changing the topic to "kernelci_new", right?
>  >
>  > I am preparing for the switch and will do it unless I hear from you soon.
>  > Since you already gave green light above, I assume we are good now.
>
> Yes, please go ahead :)!

We are live!
https://staging.kernelci.org:3000/d/home/home?orgId=1&var-origin=syzbot&var-dataset=kernelci04&from=now-90d&to=now

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/2/20 12:02 PM, Dmitry Vyukov wrote:
 > On Fri, Oct 2, 2020 at 10:12 AM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 10/2/20 10:52 AM, Dmitry Vyukov wrote:
 >>   > On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>   >> The switch involves just changing the topic to "kernelci_new", right?
 >>   >
 >>   > I am preparing for the switch and will do it unless I hear from you soon.
 >>   > Since you already gave green light above, I assume we are good now.
 >>
 >> Yes, please go ahead :)!
 >
 > We are live!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&var-origin=syzbot&var-dataset=kernelci04&from=now-90d&to=now

Whoaaa, awesooomeee!
Thank you, that was super-fast!

Now I'll go fix all the issues the extra traffic has uncovered, and then let's
do all the features we need :D

Nick

P.S. I thought you were on a US timezone so didn't hurry with my answer
      earlier, sorry, but now I'll know you're only one hour away :)

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

On 10/2/20 12:02 PM, Dmitry Vyukov via groups.io wrote:
 > On Fri, Oct 2, 2020 at 10:12 AM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 10/2/20 10:52 AM, Dmitry Vyukov wrote:
 >>   > On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>   >> The switch involves just changing the topic to "kernelci_new", right?
 >>   >
 >>   > I am preparing for the switch and will do it unless I hear from you soon.
 >>   > Since you already gave green light above, I assume we are good now.
 >>
 >> Yes, please go ahead :)!
 >
 > We are live!
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&var-origin=syzbot&var-dataset=kernelci04&from=now-90d&to=now

We got a weird report (attached), could you take a look?
It doesn't validate, and the data doesn't make much sense.

Nick

[-- Attachment #2: 165.data.json --]
[-- Type: application/json, Size: 1409 bytes --]

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Fri, Oct 2, 2020 at 12:40 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/2/20 12:02 PM, Dmitry Vyukov via groups.io wrote:
>  > On Fri, Oct 2, 2020 at 10:12 AM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> On 10/2/20 10:52 AM, Dmitry Vyukov wrote:
>  >>   > On Thu, Oct 1, 2020 at 7:02 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>  >>   >> The switch involves just changing the topic to "kernelci_new", right?
>  >>   >
>  >>   > I am preparing for the switch and will do it unless I hear from you soon.
>  >>   > Since you already gave green light above, I assume we are good now.
>  >>
>  >> Yes, please go ahead :)!
>  >
>  > We are live!
>  > https://staging.kernelci.org:3000/d/home/home?orgId=1&var-origin=syzbot&var-dataset=kernelci04&from=now-90d&to=now
>
> We got a weird report (attached), could you take a look?
> It doesn't validate, and the data doesn't make much sense.

Right. This is a syzkaller build/test error which we store equally
with kernel issues.
This should fix it:
https://github.com/google/syzkaller/pull/2161/commits/38eef6a19a85b0992c05109e8c6944eae2fd8e59

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

On 10/1/20 8:02 PM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 6:34 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> I think it's OK that you have some duplication, and certainly OK for the
 >> start. Let's get on with sending the data and see how much actual duplication
 >> we get, how it affects performance, and how the de-duplication implementation
 >> goes :)
 >
 > The new format is now published by syzbot itself:
 > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzbot:60b38542a0dfb2e0f746

Awesome!

 > So, do we switch to prod instance?

Yes :)!

 > Do you receive notifications now?

No, more on that below.

 > Do we need to fix anything else before the switch?

No, let's go ahead with the switch. We can figure out the problems as we go if
we see any. Our "production" instance is not much of a "production" yet.

 > I don't see anything from syzbot at:
 > https://groups.io/g/kernelci-results-staging/topics

Yes, I'm trying to investigate that currently. We have a few issues working
together it seems. We're facing database query limit issues, plus whatever
messages are generated are not getting delivered. You're already having an
impact :D!

I'll work through that and will notify you once notifications for syzbot
results are coming through.

 > How can I see/subscribe to these notifications?

You can subscribe to that maillist. Please note, however, even though e-mails
have real recipients, such as maillists, only this maillist is on their
envelope. This way we can see they're generated correctly, but don't send them
anywhere yet.

 > Since syzbot database now memorizes what was published to kcidb to
 > avoid dups, the switch will require some manual database surgery to
 > make it re-publish bugs. So I want to make sure we did not miss
 > something simple that we could do before the switch.

Sure.

 > The switch involves just changing the topic to "kernelci_new", right?

Yes.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
> * Could you move the "dashboard" link out of test's "output_files", and into
>    a "misc" field? It won't be presented as a link in Grafana unfortunately,
>    but "output_files" is for something which could be useful when downloaded
>    (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
>    Let's deal with the dashboard link issue in
>    https://github.com/kernelci/kcidb/issues/93

Any suggestions/precedents for the name?
dashboard_link? origin_link? master_link?

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> * Could you move the "dashboard" link out of test's "output_files", and into
 >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
 >>     but "output_files" is for something which could be useful when downloaded
 >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
 >>     Let's deal with the dashboard link issue in
 >>     https://github.com/kernelci/kcidb/issues/93
 >
 > Any suggestions/precedents for the name?
 > dashboard_link? origin_link? master_link?

There's a PR linked which adds "origin_url":
https://github.com/kernelci/kcidb/pull/94

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
>  > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >> * Could you move the "dashboard" link out of test's "output_files", and into
>  >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
>  >>     but "output_files" is for something which could be useful when downloaded
>  >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
>  >>     Let's deal with the dashboard link issue in
>  >>     https://github.com/kernelci/kcidb/issues/93
>  >
>  > Any suggestions/precedents for the name?
>  > dashboard_link? origin_link? master_link?
>
> There's a PR linked which adds "origin_url":
> https://github.com/kernelci/kcidb/pull/94

I've implemented all suggested improvements, please check this one:
https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656

Here is top-level link, look for the "https:" one:
https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04

Not sure why it's duplicate here, though:
https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
both have the same id. If they have the same id, shouldn't they be merged?

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 5:36 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
> >
> > On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
> >  > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
> >  > <Nikolai.Kondrashov@redhat.com> wrote:
> >  >> * Could you move the "dashboard" link out of test's "output_files", and into
> >  >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
> >  >>     but "output_files" is for something which could be useful when downloaded
> >  >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
> >  >>     Let's deal with the dashboard link issue in
> >  >>     https://github.com/kernelci/kcidb/issues/93
> >  >
> >  > Any suggestions/precedents for the name?
> >  > dashboard_link? origin_link? master_link?
> >
> > There's a PR linked which adds "origin_url":
> > https://github.com/kernelci/kcidb/pull/94
>
> I've implemented all suggested improvements, please check this one:
> https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656
>
> Here is top-level link, look for the "https:" one:
> https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04
>
> Not sure why it's duplicate here, though:
> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
> both have the same id. If they have the same id, shouldn't they be merged?

Oh, I submitted it again and now there 3 of them:
https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032

Should they be merged based on id?

These ones were submitted manually. Do we do dups for the "syzbot" origin?

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 6:40 PM, Dmitry Vyukov wrote:
 > On Thu, Oct 1, 2020 at 5:36 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>
 >> On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
 >> <Nikolai.Kondrashov@redhat.com> wrote:
 >>>
 >>> On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
 >>>   > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
 >>>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>>   >> * Could you move the "dashboard" link out of test's "output_files", and into
 >>>   >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
 >>>   >>     but "output_files" is for something which could be useful when downloaded
 >>>   >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
 >>>   >>     Let's deal with the dashboard link issue in
 >>>   >>     https://github.com/kernelci/kcidb/issues/93
 >>>   >
 >>>   > Any suggestions/precedents for the name?
 >>>   > dashboard_link? origin_link? master_link?
 >>>
 >>> There's a PR linked which adds "origin_url":
 >>> https://github.com/kernelci/kcidb/pull/94
 >>
 >> I've implemented all suggested improvements, please check this one:
 >> https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656
 >>
 >> Here is top-level link, look for the "https:" one:
 >> https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04
 >>
 >> Not sure why it's duplicate here, though:
 >> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
 >> both have the same id. If they have the same id, shouldn't they be merged?
 >
 > Oh, I submitted it again and now there 3 of them:
 > https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
 >
 > Should they be merged based on id?
 >
 > These ones were submitted manually. Do we do dups for the "syzbot" origin?

No, we don't do de-duping yet, this is what I'm going to work on next, as I
mention in the other message.

I agree, this needs to be handled, but we should try to not resubmit without a
need, in general :)

Nick

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 6:36 PM, Dmitry Vyukov via groups.io wrote:
 > On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
 >>   > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
 >>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   >> * Could you move the "dashboard" link out of test's "output_files", and into
 >>   >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
 >>   >>     but "output_files" is for something which could be useful when downloaded
 >>   >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
 >>   >>     Let's deal with the dashboard link issue in
 >>   >>     https://github.com/kernelci/kcidb/issues/93
 >>   >
 >>   > Any suggestions/precedents for the name?
 >>   > dashboard_link? origin_link? master_link?
 >>
 >> There's a PR linked which adds "origin_url":
 >> https://github.com/kernelci/kcidb/pull/94
 >
 > I've implemented all suggested improvements, please check this one:
 > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656
 >
 > Here is top-level link, look for the "https:" one:
 > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04

Niice!

Although, can you use just the "syzbot" origin? It's going to be easier to
identify and remember for everyone, as it matches your branding.

 > Not sure why it's duplicate here, though:
 > https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
 > both have the same id. If they have the same id, shouldn't they be merged?

That's what I'm going to work on next, as part of support for the
submit-extra-fields-later mechanism I mentioned earlier. So far it hasn't been
a problem as everyone generally took care to not resubmit, but I agree, it
shouldn't be that way.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Thu, Oct 1, 2020 at 5:55 PM Nikolai Kondrashov <spbnick@gmail.com> wrote:
>
> On 10/1/20 6:36 PM, Dmitry Vyukov via groups.io wrote:
>  > On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
>  >>   > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
>  >>   > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>   >> * Could you move the "dashboard" link out of test's "output_files", and into
>  >>   >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
>  >>   >>     but "output_files" is for something which could be useful when downloaded
>  >>   >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
>  >>   >>     Let's deal with the dashboard link issue in
>  >>   >>     https://github.com/kernelci/kcidb/issues/93
>  >>   >
>  >>   > Any suggestions/precedents for the name?
>  >>   > dashboard_link? origin_link? master_link?
>  >>
>  >> There's a PR linked which adds "origin_url":
>  >> https://github.com/kernelci/kcidb/pull/94
>  >
>  > I've implemented all suggested improvements, please check this one:
>  > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656
>  >
>  > Here is top-level link, look for the "https:" one:
>  > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04
>
> Niice!
>
> Although, can you use just the "syzbot" origin? It's going to be easier to
> identify and remember for everyone, as it matches your branding.

That's a different origin for entries I submit manually for testing purposes.
The actual production implementation will submit everything as "syzbot".
I am not sure what origin I should use for the manual tool. Using a
different origin has some advantages, e.g. we can see that there are 3
dups but that was submitted manually.

>  > Not sure why it's duplicate here, though:
>  > https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:1861b29bc5d979149e290f78e331f38f022ab032
>  > both have the same id. If they have the same id, shouldn't they be merged?
>
> That's what I'm going to work on next, as part of support for the
> submit-extra-fields-later mechanism I mentioned earlier. So far it hasn't been
> a problem as everyone generally took care to not resubmit, but I agree, it
> shouldn't be that way.
>
> Nick

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 10/1/20 7:03 PM, Dmitry Vyukov via groups.io wrote:
 > On Thu, Oct 1, 2020 at 5:55 PM Nikolai Kondrashov <spbnick@gmail.com> wrote:
 >>
 >> On 10/1/20 6:36 PM, Dmitry Vyukov via groups.io wrote:
 >>   > On Thu, Oct 1, 2020 at 5:12 PM Nikolai Kondrashov
 >>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   >>
 >>   >> On 10/1/20 6:01 PM, Dmitry Vyukov wrote:
 >>   >>   > On Thu, Oct 1, 2020 at 12:48 PM Nikolai Kondrashov
 >>   >>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   >>   >> * Could you move the "dashboard" link out of test's "output_files", and into
 >>   >>   >>     a "misc" field? It won't be presented as a link in Grafana unfortunately,
 >>   >>   >>     but "output_files" is for something which could be useful when downloaded
 >>   >>   >>     (and e.g. re-hosted by KCIDB), and the linked HTML doesn't seem to be.
 >>   >>   >>     Let's deal with the dashboard link issue in
 >>   >>   >>     https://github.com/kernelci/kcidb/issues/93
 >>   >>   >
 >>   >>   > Any suggestions/precedents for the name?
 >>   >>   > dashboard_link? origin_link? master_link?
 >>   >>
 >>   >> There's a PR linked which adds "origin_url":
 >>   >> https://github.com/kernelci/kcidb/pull/94
 >>   >
 >>   > I've implemented all suggested improvements, please check this one:
 >>   > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-dataset=playground_kernelci04&var-id=syzkcidb:0a89a7b56db04c21a656
 >>   >
 >>   > Here is top-level link, look for the "https:" one:
 >>   > https://staging.kernelci.org:3000/d/home/home?orgId=1&refresh=30m&from=now-90d&to=now&var-origin=syzkcidb&var-git_repository_url=All&var-dataset=playground_kernelci04
 >>
 >> Niice!
 >>
 >> Although, can you use just the "syzbot" origin? It's going to be easier to
 >> identify and remember for everyone, as it matches your branding.
 >
 > That's a different origin for entries I submit manually for testing purposes.
 > The actual production implementation will submit everything as "syzbot".
 > I am not sure what origin I should use for the manual tool. Using a
 > different origin has some advantages, e.g. we can see that there are 3
 > dups but that was submitted manually.

Ah, I see. Yes, that's useful.

Nick

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

On 9/28/20 7:09 PM, Dmitry Vyukov wrote:
 > There are no "issues" either...
 > Here is what I've got from kcidb-schema (on latest checkout):
 > https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt

Yes, sorry about that, when I get to them, I'll implement support for both
"issues" and "incidents".

Nick

On 9/28/20 7:09 PM, Dmitry Vyukov wrote:
 > On Mon, Sep 28, 2020 at 5:23 PM Dmitry Vyukov <dvyukov@google.com> wrote:
 >>
 >> On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
 >> <Nikolai.Kondrashov@redhat.com> wrote:
 >>>
 >>> Hi Dmitry,
 >>>
 >>> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 >>>   > The examples are helpful. I think this should be doable. The ability
 >>>   > to send everything in a single request is handy, and since we don't
 >>>   > send bulky data (only links), should not cause any issues.
 >>>
 >>> Awesome!
 >>>
 >>>   > I need an API key, right? How are they distributed? Could you send me
 >>>   > one to my email privately?
 >>>
 >>> Yep, just sent it.
 >>>
 >>>   > However, we don't use any Python and use of kcidb-submit will be
 >>>   > problematic as the app is a Go App Engine app.
 >>>
 >>> Do you mean you can't run executables, or that having the Python stack in
 >>> there as well is not possible/burdensome? Just curious.
 >>
 >> Appengine is what's called "serverless" today. There are no servers,
 >> filesystem, custom binaries. I just upload a set of source Go files
 >> and Appengine takes care of the rest. So there is no way to deploy a
 >> custom binary written in a different language.
 >>
 >>
 >>>   > I looked what's the actual protocol and got to here:
 >>>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 >>>   > and this "client" seems to come from google.cloud.pubsub.
 >>>   >
 >>>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 >>>   > it would be much more practical for syzbot to use that directly. But
 >>>   > what exactly is the protocol? I assume it's that JSON, but is it
 >>>   > wrapped into some other layers? What's the topic name to publish? Is
 >>>   > there description/example on this level?
 >>>
 >>> There's no documentation available for this level, yet, just the code.
 >>>
 >>> However, yes, it's just JSON submitted to a PubSub topic, currently, and the
 >>> schema is exactly as output by "kcidb-schema".
 >>>
 >>> Please validate JSON fully before submitting to the PubSub topic
 >>> ("kcidb-validate" could help you start). We don't have error reporting to
 >>> submitters from the server side yet, we'll have to read the logs to find out
 >>> what happened, if it breaks :)
 >>>
 >>> I sent you the project/topic names in the message with the credentials.
 >>>
 >>> Oh, and we would definitely appreciate a contribution of the Go submission
 >>> module, if you make one :)
 >>>
 >>>   > Also, how can I test things? Is there a test instance? Or if I push
 >>>   > something, can I observe it on the KernelCI dashboard or something?
 >>>
 >>> Yes, there's a "playground" instance, details are in the message with the
 >>> credentials. You can see the results on the "playground" dashboard, and query
 >>> them from the "playground" database with "kcidb-query".
 >>>
 >>>   > What would be the procedure for deploying it on the syzbot side?
 >>>
 >>> Do you mean "deploying the dashboard"?
 >>
 >> I meant when/how do we deploy this uploading to production. I mean if
 >> I do a prototype and it works with a playground, what's next?
 >> ... answering my own question, I guess we can make production syzbot
 >> upload to the playground kcidb first and then see how it works. If it
 >> works well, then we can switch to production kcidb.
 >>
 >>
 >>
 >> I have a question re schema. You mentioned "incidents" above. But I
 >> don't see them in the kcidb-schema output.
 >
 > There are no "issues" either...
 > Here is what I've got from kcidb-schema (on latest checkout):
 > https://gist.githubusercontent.com/dvyukov/a067b63e74399201df921d3846241c82/raw/3f3aa708bbb24f530d49f767a8006f7a51826a80/gistfile1.txt
 >

Re: [kernelci-members] Working with the KernelCI project

[Nikolai Kondrashov]

Hi Dmitry,

On 9/28/20 6:23 PM, Dmitry Vyukov wrote:
 > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> Do you mean you can't run executables, or that having the Python stack in
 >> there as well is not possible/burdensome? Just curious.
 >
 > Appengine is what's called "serverless" today. There are no servers,
 > filesystem, custom binaries. I just upload a set of source Go files
 > and Appengine takes care of the rest. So there is no way to deploy a
 > custom binary written in a different language.

I see, thank you.

We're using Google Cloud Functions written in Python to do submission
processing, and are considering using the Appengine, so I think can imagine
what you're talking about.

 >> Do you mean "deploying the dashboard"?
 >
 > I meant when/how do we deploy this uploading to production. I mean if
 > I do a prototype and it works with a playground, what's next?
 > ... answering my own question, I guess we can make production syzbot
 > upload to the playground kcidb first and then see how it works. If it
 > works well, then we can switch to production kcidb.

Yes, sounds like a good plan! I don't think we need to hang out in the
"playground" beyond the point where you see that your data is consistently
validated and submitted. As soon as it fits the schema and you're satisfied
with what you send, we can switch to production.

 > I have a question re schema. You mentioned "incidents" above. But I
 > don't see them in the kcidb-schema output.

Unfortunately I didn't have the time to work on that yet, but it's one of my
top priorities. Meanwhile you can start sending test results without
correllating them to issues and incidents - you will be able to see what else
we're missing, we'll see how much data you produce and whether we can handle
it, and so on.

If you'd like to be careful and actively avoid sending failure notifications,
you can set the "waived" flag for your tests to "true", which would ask the
system to avoid considering your test results for reporting, but it would
still record them.

Nick

On 9/28/20 6:23 PM, Dmitry Vyukov wrote:
 > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >>
 >> Hi Dmitry,
 >>
 >> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 >>   > The examples are helpful. I think this should be doable. The ability
 >>   > to send everything in a single request is handy, and since we don't
 >>   > send bulky data (only links), should not cause any issues.
 >>
 >> Awesome!
 >>
 >>   > I need an API key, right? How are they distributed? Could you send me
 >>   > one to my email privately?
 >>
 >> Yep, just sent it.
 >>
 >>   > However, we don't use any Python and use of kcidb-submit will be
 >>   > problematic as the app is a Go App Engine app.
 >>
 >> Do you mean you can't run executables, or that having the Python stack in
 >> there as well is not possible/burdensome? Just curious.
 >
 > Appengine is what's called "serverless" today. There are no servers,
 > filesystem, custom binaries. I just upload a set of source Go files
 > and Appengine takes care of the rest. So there is no way to deploy a
 > custom binary written in a different language.
 >
 >
 >>   > I looked what's the actual protocol and got to here:
 >>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 >>   > and this "client" seems to come from google.cloud.pubsub.
 >>   >
 >>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 >>   > it would be much more practical for syzbot to use that directly. But
 >>   > what exactly is the protocol? I assume it's that JSON, but is it
 >>   > wrapped into some other layers? What's the topic name to publish? Is
 >>   > there description/example on this level?
 >>
 >> There's no documentation available for this level, yet, just the code.
 >>
 >> However, yes, it's just JSON submitted to a PubSub topic, currently, and the
 >> schema is exactly as output by "kcidb-schema".
 >>
 >> Please validate JSON fully before submitting to the PubSub topic
 >> ("kcidb-validate" could help you start). We don't have error reporting to
 >> submitters from the server side yet, we'll have to read the logs to find out
 >> what happened, if it breaks :)
 >>
 >> I sent you the project/topic names in the message with the credentials.
 >>
 >> Oh, and we would definitely appreciate a contribution of the Go submission
 >> module, if you make one :)
 >>
 >>   > Also, how can I test things? Is there a test instance? Or if I push
 >>   > something, can I observe it on the KernelCI dashboard or something?
 >>
 >> Yes, there's a "playground" instance, details are in the message with the
 >> credentials. You can see the results on the "playground" dashboard, and query
 >> them from the "playground" database with "kcidb-query".
 >>
 >>   > What would be the procedure for deploying it on the syzbot side?
 >>
 >> Do you mean "deploying the dashboard"?
 >
 > I meant when/how do we deploy this uploading to production. I mean if
 > I do a prototype and it works with a playground, what's next?
 > ... answering my own question, I guess we can make production syzbot
 > upload to the playground kcidb first and then see how it works. If it
 > works well, then we can switch to production kcidb.
 >
 >
 >
 > I have a question re schema. You mentioned "incidents" above. But I
 > don't see them in the kcidb-schema output.
 >
 >
 >
 >> Excited to have you join the effort!
 >>
 >> Nick
 >>
 >> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
 >>   > On Fri, Aug 21, 2020 at 12:10 PM Nikolai Kondrashov
 >>   > <Nikolai.Kondrashov@redhat.com> wrote:
 >>   >>
 >>   >> Hi Dmitry,
 >>   >>
 >>   >> On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
 >>   >>   > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
 >>   >>   >> Did you have time to take a look at my mock-up? It probably looks like too
 >>   >>   >> much text, but please don't hesitate to reach me with any questions or
 >>   >>   >> suggestions you have as you look through it, if you haven't already.
 >>   >>   >>
 >>   >>   >> Please note that you don't have to start with sending all this data. In fact,
 >>   >>   >> only a few structural fields are required and we can start with the absolute
 >>   >>   >> minimum. The mock-up is showing what you *could* already send, but you don't
 >>   >>   >> have to go that far.
 >>   >>   >>
 >>   >>   >> I'd like to schedule a hacking session at Plumbers where we go and try to get
 >>   >>   >> some data out of participants' systems and into KCIDB. Would you be interested
 >>   >>   >> to attend, perhaps?
 >>   >>   >
 >>   >>   > Hi Nick,
 >>   >>   >
 >>   >>   > I've seen your reply and seen that you provided an example of how to
 >>   >>   > fill in the data. I was starred in my inbox, but I did not have time
 >>   >>   > to look at the details yet (we have somewhat crazy intern season). I
 >>   >>   > still want to do this.
 >>   >>
 >>   >> Glad to hear that, and I understand about the time.
 >>   >>
 >>   >>   > I am not sure how a remote virtual hacking session is different from
 >>   >>   > just doing this normally and some email exchange :)
 >>   >>
 >>   >> I would say the low response latency helps move faster :)
 >>   >>
 >>   >> We could have a video call in background and/or share a screen or a tmate
 >>   >> session, as we try to make things work. Even just having someone available in
 >>   >> an IRC chat for quick responses helps, I'd say.
 >>   >>
 >>   >> Everyone has their own preferences, of course, I'm just offering to be there
 >>   >> and discuss things and maybe work on something :)
 >>   >>
 >>   >> As I just wrote to Philip, I've finally published the schema introduction
 >>   >> article:
 >>   >>
 >>   >>       https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
 >>   >>
 >>   >> Here's our talk, same MC as yours, but later:
 >>   >>
 >>   >>       https://linuxplumbersconf.org/event/7/contributions/730/
 >>   >>
 >>   >> and the BoF session next day:
 >>   >>
 >>   >>       https://linuxplumbersconf.org/event/7/contributions/731/
 >>   >>
 >>   >> Will be glad to meet you there, even though we won't be able to do
 >>   >> much in the 45 minutes we were allocated.
 >>   >>
 >>   >> So please also drop by our "hallway chat", whenever you have time:
 >>   >>
 >>   >>       https://chat.2020.linuxplumbersconf.org/channel/kcidb
 >>   >>
 >>   >> I'll be there every day of the conference, during Europe's working hours.
 >>   >
 >>   >
 >>   > Hi Nikolai,
 >>   >
 >>   > The examples are helpful. I think this should be doable. The ability
 >>   > to send everything in a single request is handy, and since we don't
 >>   > send bulky data (only links), should not cause any issues.
 >>   >
 >>   > I need an API key, right? How are they distributed? Could you send me
 >>   > one to my email privately?
 >>   >
 >>   > However, we don't use any Python and use of kcidb-submit will be
 >>   > problematic as the app is a Go App Engine app.
 >>   > I looked what's the actual protocol and got to here:
 >>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
 >>   > and this "client" seems to come from google.cloud.pubsub.
 >>   >
 >>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
 >>   > it would be much more practical for syzbot to use that directly. But
 >>   > what exactly is the protocol? I assume it's that JSON, but is it
 >>   > wrapped into some other layers? What's the topic name to publish? Is
 >>   > there description/example on this level?
 >>   >
 >>   > Also, how can I test things? Is there a test instance? Or if I push
 >>   > something, can I observe it on the KernelCI dashboard or something?
 >>   > What would be the procedure for deploying it on the syzbot side?
 >>   >
 >>   > Thanks
 >>   >
 >>
 >

Re: [kernelci-members] Working with the KernelCI project

[Dmitry Vyukov]

On Mon, Sep 28, 2020 at 6:49 PM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Dmitry,
>
> On 9/28/20 6:23 PM, Dmitry Vyukov wrote:
>  > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >> Do you mean you can't run executables, or that having the Python stack in
>  >> there as well is not possible/burdensome? Just curious.
>  >
>  > Appengine is what's called "serverless" today. There are no servers,
>  > filesystem, custom binaries. I just upload a set of source Go files
>  > and Appengine takes care of the rest. So there is no way to deploy a
>  > custom binary written in a different language.
>
> I see, thank you.
>
> We're using Google Cloud Functions written in Python to do submission
> processing, and are considering using the Appengine, so I think can imagine
> what you're talking about.

For an AppEngine project in Python, it should be perfectly possible to
use a Python library.
Or, deploy 2 AppEngine apps and make them talk to each... but at this
point it moves in the area of burdensome.


>  >> Do you mean "deploying the dashboard"?
>  >
>  > I meant when/how do we deploy this uploading to production. I mean if
>  > I do a prototype and it works with a playground, what's next?
>  > ... answering my own question, I guess we can make production syzbot
>  > upload to the playground kcidb first and then see how it works. If it
>  > works well, then we can switch to production kcidb.
>
> Yes, sounds like a good plan! I don't think we need to hang out in the
> "playground" beyond the point where you see that your data is consistently
> validated and submitted. As soon as it fits the schema and you're satisfied
> with what you send, we can switch to production.
>
>  > I have a question re schema. You mentioned "incidents" above. But I
>  > don't see them in the kcidb-schema output.
>
> Unfortunately I didn't have the time to work on that yet, but it's one of my
> top priorities. Meanwhile you can start sending test results without
> correllating them to issues and incidents - you will be able to see what else
> we're missing, we'll see how much data you produce and whether we can handle
> it, and so on.
>
> If you'd like to be careful and actively avoid sending failure notifications,
> you can set the "waived" flag for your tests to "true", which would ask the
> system to avoid considering your test results for reporting, but it would
> still record them.

Ack.

> Nick
>
> On 9/28/20 6:23 PM, Dmitry Vyukov wrote:
>  > On Mon, Sep 28, 2020 at 3:16 PM Nikolai Kondrashov
>  > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>
>  >> Hi Dmitry,
>  >>
>  >> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
>  >>   > The examples are helpful. I think this should be doable. The ability
>  >>   > to send everything in a single request is handy, and since we don't
>  >>   > send bulky data (only links), should not cause any issues.
>  >>
>  >> Awesome!
>  >>
>  >>   > I need an API key, right? How are they distributed? Could you send me
>  >>   > one to my email privately?
>  >>
>  >> Yep, just sent it.
>  >>
>  >>   > However, we don't use any Python and use of kcidb-submit will be
>  >>   > problematic as the app is a Go App Engine app.
>  >>
>  >> Do you mean you can't run executables, or that having the Python stack in
>  >> there as well is not possible/burdensome? Just curious.
>  >
>  > Appengine is what's called "serverless" today. There are no servers,
>  > filesystem, custom binaries. I just upload a set of source Go files
>  > and Appengine takes care of the rest. So there is no way to deploy a
>  > custom binary written in a different language.
>  >
>  >
>  >>   > I looked what's the actual protocol and got to here:
>  >>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
>  >>   > and this "client" seems to come from google.cloud.pubsub.
>  >>   >
>  >>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
>  >>   > it would be much more practical for syzbot to use that directly. But
>  >>   > what exactly is the protocol? I assume it's that JSON, but is it
>  >>   > wrapped into some other layers? What's the topic name to publish? Is
>  >>   > there description/example on this level?
>  >>
>  >> There's no documentation available for this level, yet, just the code.
>  >>
>  >> However, yes, it's just JSON submitted to a PubSub topic, currently, and the
>  >> schema is exactly as output by "kcidb-schema".
>  >>
>  >> Please validate JSON fully before submitting to the PubSub topic
>  >> ("kcidb-validate" could help you start). We don't have error reporting to
>  >> submitters from the server side yet, we'll have to read the logs to find out
>  >> what happened, if it breaks :)
>  >>
>  >> I sent you the project/topic names in the message with the credentials.
>  >>
>  >> Oh, and we would definitely appreciate a contribution of the Go submission
>  >> module, if you make one :)
>  >>
>  >>   > Also, how can I test things? Is there a test instance? Or if I push
>  >>   > something, can I observe it on the KernelCI dashboard or something?
>  >>
>  >> Yes, there's a "playground" instance, details are in the message with the
>  >> credentials. You can see the results on the "playground" dashboard, and query
>  >> them from the "playground" database with "kcidb-query".
>  >>
>  >>   > What would be the procedure for deploying it on the syzbot side?
>  >>
>  >> Do you mean "deploying the dashboard"?
>  >
>  > I meant when/how do we deploy this uploading to production. I mean if
>  > I do a prototype and it works with a playground, what's next?
>  > ... answering my own question, I guess we can make production syzbot
>  > upload to the playground kcidb first and then see how it works. If it
>  > works well, then we can switch to production kcidb.
>  >
>  >
>  >
>  > I have a question re schema. You mentioned "incidents" above. But I
>  > don't see them in the kcidb-schema output.
>  >
>  >
>  >
>  >> Excited to have you join the effort!
>  >>
>  >> Nick
>  >>
>  >> On 9/28/20 3:48 PM, Dmitry Vyukov wrote:
>  >>   > On Fri, Aug 21, 2020 at 12:10 PM Nikolai Kondrashov
>  >>   > <Nikolai.Kondrashov@redhat.com> wrote:
>  >>   >>
>  >>   >> Hi Dmitry,
>  >>   >>
>  >>   >> On 8/5/20 9:44 PM, Dmitry Vyukov wrote:
>  >>   >>   > On Mon, Aug 3, 2020 at 11:25 AM Nikolai Kondrashov
>  >>   >>   >> Did you have time to take a look at my mock-up? It probably looks like too
>  >>   >>   >> much text, but please don't hesitate to reach me with any questions or
>  >>   >>   >> suggestions you have as you look through it, if you haven't already.
>  >>   >>   >>
>  >>   >>   >> Please note that you don't have to start with sending all this data. In fact,
>  >>   >>   >> only a few structural fields are required and we can start with the absolute
>  >>   >>   >> minimum. The mock-up is showing what you *could* already send, but you don't
>  >>   >>   >> have to go that far.
>  >>   >>   >>
>  >>   >>   >> I'd like to schedule a hacking session at Plumbers where we go and try to get
>  >>   >>   >> some data out of participants' systems and into KCIDB. Would you be interested
>  >>   >>   >> to attend, perhaps?
>  >>   >>   >
>  >>   >>   > Hi Nick,
>  >>   >>   >
>  >>   >>   > I've seen your reply and seen that you provided an example of how to
>  >>   >>   > fill in the data. I was starred in my inbox, but I did not have time
>  >>   >>   > to look at the details yet (we have somewhat crazy intern season). I
>  >>   >>   > still want to do this.
>  >>   >>
>  >>   >> Glad to hear that, and I understand about the time.
>  >>   >>
>  >>   >>   > I am not sure how a remote virtual hacking session is different from
>  >>   >>   > just doing this normally and some email exchange :)
>  >>   >>
>  >>   >> I would say the low response latency helps move faster :)
>  >>   >>
>  >>   >> We could have a video call in background and/or share a screen or a tmate
>  >>   >> session, as we try to make things work. Even just having someone available in
>  >>   >> an IRC chat for quick responses helps, I'd say.
>  >>   >>
>  >>   >> Everyone has their own preferences, of course, I'm just offering to be there
>  >>   >> and discuss things and maybe work on something :)
>  >>   >>
>  >>   >> As I just wrote to Philip, I've finally published the schema introduction
>  >>   >> article:
>  >>   >>
>  >>   >>       https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>  >>   >>
>  >>   >> Here's our talk, same MC as yours, but later:
>  >>   >>
>  >>   >>       https://linuxplumbersconf.org/event/7/contributions/730/
>  >>   >>
>  >>   >> and the BoF session next day:
>  >>   >>
>  >>   >>       https://linuxplumbersconf.org/event/7/contributions/731/
>  >>   >>
>  >>   >> Will be glad to meet you there, even though we won't be able to do
>  >>   >> much in the 45 minutes we were allocated.
>  >>   >>
>  >>   >> So please also drop by our "hallway chat", whenever you have time:
>  >>   >>
>  >>   >>       https://chat.2020.linuxplumbersconf.org/channel/kcidb
>  >>   >>
>  >>   >> I'll be there every day of the conference, during Europe's working hours.
>  >>   >
>  >>   >
>  >>   > Hi Nikolai,
>  >>   >
>  >>   > The examples are helpful. I think this should be doable. The ability
>  >>   > to send everything in a single request is handy, and since we don't
>  >>   > send bulky data (only links), should not cause any issues.
>  >>   >
>  >>   > I need an API key, right? How are they distributed? Could you send me
>  >>   > one to my email privately?
>  >>   >
>  >>   > However, we don't use any Python and use of kcidb-submit will be
>  >>   > problematic as the app is a Go App Engine app.
>  >>   > I looked what's the actual protocol and got to here:
>  >>   > https://github.com/kernelci/kcidb/blob/master/kcidb/mq/__init__.py#L69
>  >>   > and this "client" seems to come from google.cloud.pubsub.
>  >>   >
>  >>   > Go has https://pkg.go.dev/cloud.google.com/go/pubsub package as well,
>  >>   > it would be much more practical for syzbot to use that directly. But
>  >>   > what exactly is the protocol? I assume it's that JSON, but is it
>  >>   > wrapped into some other layers? What's the topic name to publish? Is
>  >>   > there description/example on this level?
>  >>   >
>  >>   > Also, how can I test things? Is there a test instance? Or if I push
>  >>   > something, can I observe it on the KernelCI dashboard or something?
>  >>   > What would be the procedure for deploying it on the syzbot side?
>  >>   >
>  >>   > Thanks
>  >>   >
>  >>
>  >
>

Re: Working with the KernelCI project

[Nikolai Kondrashov]

Hi Philip,

Re-sending this to add the kernelci maillist and a colleague, sorry.
Please reply to *this* message instead of the first one.

On 7/13/20 3:19 AM, Philip Li wrote:
> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
>> How about I try to take a 0-day report and express it as a KCIDB submission,
>> as an illustration of how this could work? Would that help you understand what
>> we're trying to do? If yes, could you give me a link to one?
> Right, the "accuracy" for all single branch we test is more related to technical
> problem if we need look for a way to solve it. Here assume we have a report, it
> does have chance to be aggeragated. You can pick up any link from
> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.

Alright, first I'll copy the KCIDB intro I sent Dmitry in another branch of
this thread, in case you didn't read it (otherwise skip past END-OF-INTRO):

You can submit all your data by just piping
JSON to the "kcidb-submit" tool (providing some options for authentication and
destination), or you could use the Python 3 API.

The expected JSON data is a dictionary of arrays, along with the schema
version identifier. Each of the arrays in the dictionary could be missing or
empty, but otherwise they contain various report objects. The database will
accept them in any (reasonable) amount, order, or combination, although
submission order can affect when and how notifications are sent out.

Every object has an ID which can be used to refer to it and link objects
together, but all the IDs are generated by the submitter, who is making sure
they're unique. For most of the objects (except revisions at this moment) you
can just use your CI system's ID for it and prefix it with your CI system's
name. This way you don't have to maintain a mapping between your system's IDs
and our IDs when you report results gradually. If you don't have that, you can
just generate them, for example hash some key fields or, as the last resort,
use UUIDs. Revisions are just using commit hashes at the moment.

Every object has a special property called "misc" which can contain arbitrary
data. You could use that to submit data we don't have a schema for yet. The
users/developers will be able to see it, and we can use that as samples for
implementing future support.

See example use of "misc" for a KernelCI test and its execution environment
(yes, it could be formatted better):
https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec

                                END-OF-INTRO

I think it's great that 0-day e-mails contain everything needed to investigate
and reproduce the issue and are self-sufficient. However, at the moment KCIDB
doesn't allow embedding artifacts or logs into submissions, but instead
expects them to be stored somewhere else and have the URLs provided.

At the moment we would need you to do that, at least with .config files, but
if you can't, we can work on supporting embedding them. We planned on copying
linked files to KernelCI-managed storage anyway, eventually.

KCIDB also doesn't support embedding the nice error summaries and log excerpts
you include in your reports, but that should be quite easy to amend by storing
them in the database itself (provided they're not too big). Just needs a
little thinking about the exact schema to use.

For now, though, my examples below assume you can provide links to files
(spoofed here), and you're supplying the excerpts or complete logs with those.

KCIDB doesn't support source code linters and static analysis (such as
checkpatch, or coverity) at the moment, and we should add that. However,
a sparse run could be expressed in KCIDB as a build.

Taking this report as a sample:

     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/GUFRHHPKTWFYNLRH4LE2E2YELI6XG2IE/

this is how a submission could look:

     {
         "revisions": [
             {
                 "id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
                 "origin": "0day",
                 "discovery_time": "2020-07-08T07:57:24+03:00",
                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git",
                 "git_repository_commit_hash": "391e437eedc0dab0a9f2c26997e68e040ae04ea3",
                 "git_repository_branch": "master",
                 "patch_mboxes": [
                     {
                         "name": "0001-irqchip-stm32-exti-map-direct-event-to-irq-parent.patch",
                         "url": "https://github.com/0day-ci/linux/commit/3f47dd3217f24edfd442b35784001979e7aeacc7.patch"
                     }
                 ],
                 "message_id": "20200706081106.25125-1-alexandre.torgue@st.com",
                 "contacts": [
                     "Alexandre Torgue <alexandre.torgue@st.com>",
                     "kbuild-all <kbuild-all@lists.01.org>",
                     "Marc Zyngier <maz@kernel.org>",
                     "Thomas Gleixner <tglx@linutronix.de>",
                     "Jason Cooper <jason@lakedaemon.net>",
                     "LKML <linux-kernel@vger.kernel.org>"
                 ],
                 "valid": true
             }
         ],
         "builds": [
             {
                 "id": "0day:391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c:sparse",
                 "origin": "0day",
                 "revision_id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
                 "architecture": "arm",
                 "compiler": "arm-linux-gnueabi-gcc (GCC) 9.3.0",
                 "start_time": "2020-07-08T07:57:24+03:00",
                 "config_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/config",
                 "log_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_build.log",
                 "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=arm",
                 "input_files": [
                     {
                         "name": "make.cross",
                         "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
                     },
                     {
                         "name": "instructions.txt",
                         "url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_repro.txt"
                     }
                 ],
                 "valid": false
             }
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }


The above describes the revision you're testing as a patch being applied to a
particular commit in the stm32 repo's master branch. The revision has a build,
which failed, the build has the config URL and the log linked, as well as the
reproduction instructions linked as one of the "input files". We can work on
adding a dedicated field for reproduction instructinos for both builds and
tests, since they're very useful and syzbot also produces them.

A failed W=1 build:

     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/WEKO2YRAZIPZFUQAY2D4XAOWJGC3HGBD/

Would look similar:

     {
         "revisions": [
             {
                 "id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
                 "origin": "0day",
                 "discovery_time": "2020-07-08T07:57:24+03:00",
                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git",
                 "git_repository_commit_hash": "c46ed28dbe95844c1d15addd26ff05499057c4d5",
                 "git_repository_branch": "for-next",
                 "patch_mboxes": [
                     {
                         "name": "0001-arm64-dts-qcom-sc7180-Add-lpass-cpu-node-for-I2S-dri.patch",
                         "url": "https://github.com/0day-ci/linux/commit/d20696ca206ae45d9d27fbeffb23fe5431b5de9d.patch"
                     }
                 ],
                 "message_id": "20200716061445.628709-1-cychiang@chromium.org",
                 "contacts": [
                     "Ajit Pandey <ajitp@codeaurora.org>",
                     "Cheng-Yi Chiang <cychiang@chromium.org>",
                     "kbuild-all <kbuild-all@lists.01.org>",
                     "Andy Gross <agross@kernel.org>",
                     "Bjorn Andersson <bjorn.andersson@linaro.org>",
                     "Rob Herring <robh+dt@kernel.org>",
                     "linux-arm-msm@vger.kernel.org",
                     "devicetree@vger.kernel.org"
                 ],
                 "valid": true
             }
         ],
         "builds": [
             {
                 "id": "0day:c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2:sparse",
                 "origin": "0day",
                 "revision_id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
                 "architecture": "arm",
                 "compiler": "clang version 12.0.0",
                 "start_time": "2020-07-08T07:57:24+03:00",
                 "config_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/config",
                 "log_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_build.log",
                 "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64",
                 "input_files": [
                     {
                         "name": "make.cross",
                         "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
                     },
                     {
                         "name": "instructions.txt",
                         "url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_repro.txt"
                     }
                 ],
                 "valid": false
             }
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

KCIDB also doesn't support non-runtime tests for compiled kernels (such as
size regression tests you're running), and we should add that, but meanwhile
we can accommodate them as tests without "environments".

This one:

     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/PX2O2OJZT2TZ7SU4VUB5ODM4KRBPTXD7/

Could look like this:

     {
         "revisions": [
             {
                 "id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
                 "origin": "0day",
                 "discovery_time": "2020-07-17T14:41:52+03:00",
                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git",
                 "git_repository_commit_hash": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
                 "git_repository_branch": "mount_setattr",
                 "contacts": [
                     "Christian Brauner <christian@brauner.io>",
                     "kbuild-all <kbuild-all@lists.01.org>"
                 ],
                 "valid": true
             }
         ],
         "builds": [
             {
                 "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
                 "origin": "0day",
                 "revision_id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
                 "start_time": "2020-07-17T14:41:52+03:00",
                 "valid": true
             }
         ],
         "tests": [
             {
                 "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2:size",
                 "origin": "0day",
                 "build_id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
                 "path": "size_regression",
                 "output_files": [
                     {
                         "name": "details.txt",
                         "url": "https://01.org/0day/aa63af1b08246bd31b77d056bf1d47f775cecbe2/size_regression_details.log",
                     }
                 ],
                 "start_time": "2020-07-17T14:41:52+03:00",
                 "status": "FAIL",
                 "waived": false
             },
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

Since you don't provide any build information in that report, the build object
doesn't have any data. However, that's still valid according to the current
schema.

Finally, this runtime test failure:

     https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/ULKTFB6NGLORWH2WLEKFSFEQFSIWLT5F/

you can report like this:

     {
         "revisions": [
             {
                 "id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
                 "origin": "0day",
                 "discovery_time": "2020-07-17T09:04:55+03:00",
                 "git_repository_url": "https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git"
                 "git_repository_commit_hash": "5155be9994e557618a8312389fb4e52dfbf28a3c",
                 "git_repository_branch": "master",
                 "contacts": [
                     "Paul E. McKenney <paulmck@kernel.org>",
                     "LKP <lkp@lists.01.org>"
                 ],
                 "valid": true
             }
         ],
         "builds": [
             {
                 "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
                 "origin": "0day",
                 "revision_id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
                 "start_time": "2020-07-17T09:04:55+03:00",
                 "architecture": "i386",
                 "command": "make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage",
                 "config_url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/config",
                 "valid": true
             }
         ],
         "tests": [
             {
                 "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c:trinity",
                 "origin": "0day",
                 "build_id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
                 "path": "trinity",
                 "output_files": [
                     {
                         "name": "dmesg.xz",
                         "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/dmesg.xz",
                     },
                     {
                         "name": "details.txt",
                         "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/trinity_details.log",
                     }
                 ],
                 "environment": {
                     "description": "qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 16G"
                 },
                 "start_time": "2020-07-17T09:04:55+03:00",
                 "status": "FAIL",
                 "waived": false
             },
         ],
         "version": {
             "major": 3,
             "minor": 0
         }
     }

Note the "contacts" field all revisions have: this will help us determine who
to send the reports to.

Perhaps we need to add support for test input files to accommodate your
reproduction instructions and custom scripts.

I have omitted some fields I could've added, and we need to improve the schema
to accommodate your reports better, of course.

However, if you'd be interested, we could help you set up forwarding your
reports to KernelCI. You can start very simple and small, as the schema only
requires a handful of fields. This will help us see your needs: what data you
want in reports and on the dashboard, how many reports you want to push (both
positive and negative), etc.

Don't hesitate to write with questions, suggestions, and hope to "see" you at
this year's Plumbers, where we hopefully will be presenting more about this
effort. I'll also be writing a separate article introducing the schema this
week, will copy both you and Dmitry here.

Nick

On 7/13/20 3:19 AM, Philip Li wrote:
> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
>> Hi Philip,
>>
>> On 7/9/20 2:00 PM, Philip Li wrote:
>>> The 0-day ci mostly focus on regression and then bisection, and the strategy
>>> is to merge various branches to run the test. This is not exactly as the
>>> tranditional CI. The worry here is to know exactly one branch is pass or
>>> fail currently is not 100% available. For instance, the final merged branch
>>> is fail doesn't provide fail/pass info of each individual branch. This triggers
>>> bisection to kick out the bad branch.
>>>
>>> Then it need redo the testing of remaining ones, which is not always
>>> feasible for us (considering the computing power).
>>
>> Yes, I think everyone here could sympathise with limited hardware resources :)
>>
>>> Especially, sometimes the bisection would fail.
>>
>> Can you give an example of how would a bisection fail?
>> Would that be a flaky test failing on a previously assumed-good commit, for
>> example?
> thanks, one example is build issue that breaks the bisectability which can
> lead to bisect fail.
>
>>
>>> As we focus on regression a lot to bisect to first bad commit, there would
>>> be uncertainty to draw conclusion for single branch.
>>
>> I think not having complete certainty for a project as large as the Linux
>> kernel is normal. Kernel CI has the bisection system as well, and syzbot is
>> going to even greater lengths with identifying similar failures. We at CKI
>> have test maintainers constantly looking at test failures and deciding whether
>> they're false or not. These are things we just have to handle for common
>> reporting to work.
>>
>>> This requires more careful thinking for us without increasing the needs of
>>> computing resource. This is one bottleneck I can see so far. Not sure any
>>> idea or recommendation for this.
>>
>> Our aim with common reporting is simply to provide a unified way to reach
>> developers with testing results, essentially to send them a single e-mail
>> report, instead of one report per CI system, to make a single database
>> available for analyzis and a single dashboard UI.
>>
>> I.e. instead of sending an e-mail report to a developer we ask you to send a
>> JSON report to us, and then we try to handle analyzing and reporting for you.
>>
>> It is up to the submitting CI system to choose how many, or how little tests
>> to run, or how much or how little data to send. Kernel CI is not going to ask
>> you to run any tests, it is up to you.
>>
>> In the end, we trust you want the developers to notice and fix the problems
>> you find, you'll try to provide enough data, and we'd like to make a system
>> which will help you do that. If you can pinpoint the exact commit - great! If
>> not, we'll just have some data from you which can be analyzed otherwise.
>>
>> How about I try to take a 0-day report and express it as a KCIDB submission,
>> as an illustration of how this could work? Would that help you understand what
>> we're trying to do? If yes, could you give me a link to one?
> Right, the "accuracy" for all single branch we test is more related to technical
> problem if we need look for a way to solve it. Here assume we have a report, it
> does have chance to be aggeragated. You can pick up any link from
> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
>
> Thanks
>
>>
>> Nick
>>
>

Re: Working with the KernelCI project

[Nikolai Kondrashov]

Hi Philip,

On 7/22/20 3:42 PM, Nikolai Kondrashov wrote:
 > On 7/13/20 3:19 AM, Philip Li wrote:
 >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >>> How about I try to take a 0-day report and express it as a KCIDB submission,
 >>> as an illustration of how this could work? Would that help you understand what
 >>> we're trying to do? If yes, could you give me a link to one?
 >> Right, the "accuracy" for all single branch we test is more related to technical
 >> problem if we need look for a way to solve it. Here assume we have a report, it
 >> does have chance to be aggeragated. You can pick up any link from
 >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
 >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
 >
 > I have omitted some fields I could've added, and we need to improve the schema
 > to accommodate your reports better, of course.
 >
 > However, if you'd be interested, we could help you set up forwarding your
 > reports to KernelCI. You can start very simple and small, as the schema only
 > requires a handful of fields. This will help us see your needs: what data you
 > want in reports and on the dashboard, how many reports you want to push (both
 > positive and negative), etc.
 >
 > Don't hesitate to write with questions, suggestions, and hope to "see" you at
 > this year's Plumbers, where we hopefully will be presenting more about this
 > effort. I'll also be writing a separate article introducing the schema this
 > week, will copy both you and Dmitry here.

Did you have time to take a look at my mock-up? It probably looks like too
much text, but please don't hesitate to reach me with any questions or
suggestions you have as you look through it, if you haven't already.

Please note that you don't have to start with sending all this data. In fact,
only a few structural fields are required and we can start with the absolute
minimum. The mock-up is showing what you *could* already send, but you don't
have to go that far.

I'd like to schedule a hacking session at Plumbers where we go and try to get
some data out of participants' systems and into KCIDB. Would you be interested
to attend, perhaps?

Nick

On 7/22/20 3:42 PM, Nikolai Kondrashov wrote:
 > Hi Philip,
 >
 > Re-sending this to add the kernelci maillist and a colleague, sorry.
 > Please reply to *this* message instead of the first one.
 >
 > On 7/13/20 3:19 AM, Philip Li wrote:
 >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >>> How about I try to take a 0-day report and express it as a KCIDB submission,
 >>> as an illustration of how this could work? Would that help you understand what
 >>> we're trying to do? If yes, could you give me a link to one?
 >> Right, the "accuracy" for all single branch we test is more related to technical
 >> problem if we need look for a way to solve it. Here assume we have a report, it
 >> does have chance to be aggeragated. You can pick up any link from
 >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
 >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
 >
 > Alright, first I'll copy the KCIDB intro I sent Dmitry in another branch of
 > this thread, in case you didn't read it (otherwise skip past END-OF-INTRO):
 >
 > You can submit all your data by just piping
 > JSON to the "kcidb-submit" tool (providing some options for authentication and
 > destination), or you could use the Python 3 API.
 >
 > The expected JSON data is a dictionary of arrays, along with the schema
 > version identifier. Each of the arrays in the dictionary could be missing or
 > empty, but otherwise they contain various report objects. The database will
 > accept them in any (reasonable) amount, order, or combination, although
 > submission order can affect when and how notifications are sent out.
 >
 > Every object has an ID which can be used to refer to it and link objects
 > together, but all the IDs are generated by the submitter, who is making sure
 > they're unique. For most of the objects (except revisions at this moment) you
 > can just use your CI system's ID for it and prefix it with your CI system's
 > name. This way you don't have to maintain a mapping between your system's IDs
 > and our IDs when you report results gradually. If you don't have that, you can
 > just generate them, for example hash some key fields or, as the last resort,
 > use UUIDs. Revisions are just using commit hashes at the moment.
 >
 > Every object has a special property called "misc" which can contain arbitrary
 > data. You could use that to submit data we don't have a schema for yet. The
 > users/developers will be able to see it, and we can use that as samples for
 > implementing future support.
 >
 > See example use of "misc" for a KernelCI test and its execution environment
 > (yes, it could be formatted better):
 > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
 >
 >                                 END-OF-INTRO
 >
 > I think it's great that 0-day e-mails contain everything needed to investigate
 > and reproduce the issue and are self-sufficient. However, at the moment KCIDB
 > doesn't allow embedding artifacts or logs into submissions, but instead
 > expects them to be stored somewhere else and have the URLs provided.
 >
 > At the moment we would need you to do that, at least with .config files, but
 > if you can't, we can work on supporting embedding them. We planned on copying
 > linked files to KernelCI-managed storage anyway, eventually.
 >
 > KCIDB also doesn't support embedding the nice error summaries and log excerpts
 > you include in your reports, but that should be quite easy to amend by storing
 > them in the database itself (provided they're not too big). Just needs a
 > little thinking about the exact schema to use.
 >
 > For now, though, my examples below assume you can provide links to files
 > (spoofed here), and you're supplying the excerpts or complete logs with those.
 >
 > KCIDB doesn't support source code linters and static analysis (such as
 > checkpatch, or coverity) at the moment, and we should add that. However,
 > a sparse run could be expressed in KCIDB as a build.
 >
 > Taking this report as a sample:
 >
 >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/GUFRHHPKTWFYNLRH4LE2E2YELI6XG2IE/
 >
 > this is how a submission could look:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
 >                  "origin": "0day",
 >                  "discovery_time": "2020-07-08T07:57:24+03:00",
 >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git",
 >                  "git_repository_commit_hash": "391e437eedc0dab0a9f2c26997e68e040ae04ea3",
 >                  "git_repository_branch": "master",
 >                  "patch_mboxes": [
 >                      {
 >                          "name": "0001-irqchip-stm32-exti-map-direct-event-to-irq-parent.patch",
 >                          "url": "https://github.com/0day-ci/linux/commit/3f47dd3217f24edfd442b35784001979e7aeacc7.patch"
 >                      }
 >                  ],
 >                  "message_id": "20200706081106.25125-1-alexandre.torgue@st.com",
 >                  "contacts": [
 >                      "Alexandre Torgue <alexandre.torgue@st.com>",
 >                      "kbuild-all <kbuild-all@lists.01.org>",
 >                      "Marc Zyngier <maz@kernel.org>",
 >                      "Thomas Gleixner <tglx@linutronix.de>",
 >                      "Jason Cooper <jason@lakedaemon.net>",
 >                      "LKML <linux-kernel@vger.kernel.org>"
 >                  ],
 >                  "valid": true
 >              }
 >          ],
 >          "builds": [
 >              {
 >                  "id": "0day:391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c:sparse",
 >                  "origin": "0day",
 >                  "revision_id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
 >                  "architecture": "arm",
 >                  "compiler": "arm-linux-gnueabi-gcc (GCC) 9.3.0",
 >                  "start_time": "2020-07-08T07:57:24+03:00",
 >                  "config_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/config",
 >                  "log_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_build.log",
 >                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=arm",
 >                  "input_files": [
 >                      {
 >                          "name": "make.cross",
 >                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
 >                      },
 >                      {
 >                          "name": "instructions.txt",
 >                          "url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_repro.txt"
 >                      }
 >                  ],
 >                  "valid": false
 >              }
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 >
 > The above describes the revision you're testing as a patch being applied to a
 > particular commit in the stm32 repo's master branch. The revision has a build,
 > which failed, the build has the config URL and the log linked, as well as the
 > reproduction instructions linked as one of the "input files". We can work on
 > adding a dedicated field for reproduction instructinos for both builds and
 > tests, since they're very useful and syzbot also produces them.
 >
 > A failed W=1 build:
 >
 >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/WEKO2YRAZIPZFUQAY2D4XAOWJGC3HGBD/
 >
 > Would look similar:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
 >                  "origin": "0day",
 >                  "discovery_time": "2020-07-08T07:57:24+03:00",
 >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git",
 >                  "git_repository_commit_hash": "c46ed28dbe95844c1d15addd26ff05499057c4d5",
 >                  "git_repository_branch": "for-next",
 >                  "patch_mboxes": [
 >                      {
 >                          "name": "0001-arm64-dts-qcom-sc7180-Add-lpass-cpu-node-for-I2S-dri.patch",
 >                          "url": "https://github.com/0day-ci/linux/commit/d20696ca206ae45d9d27fbeffb23fe5431b5de9d.patch"
 >                      }
 >                  ],
 >                  "message_id": "20200716061445.628709-1-cychiang@chromium.org",
 >                  "contacts": [
 >                      "Ajit Pandey <ajitp@codeaurora.org>",
 >                      "Cheng-Yi Chiang <cychiang@chromium.org>",
 >                      "kbuild-all <kbuild-all@lists.01.org>",
 >                      "Andy Gross <agross@kernel.org>",
 >                      "Bjorn Andersson <bjorn.andersson@linaro.org>",
 >                      "Rob Herring <robh+dt@kernel.org>",
 >                      "linux-arm-msm@vger.kernel.org",
 >                      "devicetree@vger.kernel.org"
 >                  ],
 >                  "valid": true
 >              }
 >          ],
 >          "builds": [
 >              {
 >                  "id": "0day:c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2:sparse",
 >                  "origin": "0day",
 >                  "revision_id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
 >                  "architecture": "arm",
 >                  "compiler": "clang version 12.0.0",
 >                  "start_time": "2020-07-08T07:57:24+03:00",
 >                  "config_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/config",
 >                  "log_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_build.log",
 >                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64",
 >                  "input_files": [
 >                      {
 >                          "name": "make.cross",
 >                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
 >                      },
 >                      {
 >                          "name": "instructions.txt",
 >                          "url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_repro.txt"
 >                      }
 >                  ],
 >                  "valid": false
 >              }
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > KCIDB also doesn't support non-runtime tests for compiled kernels (such as
 > size regression tests you're running), and we should add that, but meanwhile
 > we can accommodate them as tests without "environments".
 >
 > This one:
 >
 >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/PX2O2OJZT2TZ7SU4VUB5ODM4KRBPTXD7/
 >
 > Could look like this:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >                  "origin": "0day",
 >                  "discovery_time": "2020-07-17T14:41:52+03:00",
 >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git",
 >                  "git_repository_commit_hash": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >                  "git_repository_branch": "mount_setattr",
 >                  "contacts": [
 >                      "Christian Brauner <christian@brauner.io>",
 >                      "kbuild-all <kbuild-all@lists.01.org>"
 >                  ],
 >                  "valid": true
 >              }
 >          ],
 >          "builds": [
 >              {
 >                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >                  "origin": "0day",
 >                  "revision_id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >                  "start_time": "2020-07-17T14:41:52+03:00",
 >                  "valid": true
 >              }
 >          ],
 >          "tests": [
 >              {
 >                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2:size",
 >                  "origin": "0day",
 >                  "build_id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >                  "path": "size_regression",
 >                  "output_files": [
 >                      {
 >                          "name": "details.txt",
 >                          "url": "https://01.org/0day/aa63af1b08246bd31b77d056bf1d47f775cecbe2/size_regression_details.log",
 >                      }
 >                  ],
 >                  "start_time": "2020-07-17T14:41:52+03:00",
 >                  "status": "FAIL",
 >                  "waived": false
 >              },
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > Since you don't provide any build information in that report, the build object
 > doesn't have any data. However, that's still valid according to the current
 > schema.
 >
 > Finally, this runtime test failure:
 >
 >      https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/ULKTFB6NGLORWH2WLEKFSFEQFSIWLT5F/
 >
 > you can report like this:
 >
 >      {
 >          "revisions": [
 >              {
 >                  "id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >                  "origin": "0day",
 >                  "discovery_time": "2020-07-17T09:04:55+03:00",
 >                  "git_repository_url": "https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git"
 >                  "git_repository_commit_hash": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >                  "git_repository_branch": "master",
 >                  "contacts": [
 >                      "Paul E. McKenney <paulmck@kernel.org>",
 >                      "LKP <lkp@lists.01.org>"
 >                  ],
 >                  "valid": true
 >              }
 >          ],
 >          "builds": [
 >              {
 >                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
 >                  "origin": "0day",
 >                  "revision_id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >                  "start_time": "2020-07-17T09:04:55+03:00",
 >                  "architecture": "i386",
 >                  "command": "make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage",
 >                  "config_url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/config",
 >                  "valid": true
 >              }
 >          ],
 >          "tests": [
 >              {
 >                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c:trinity",
 >                  "origin": "0day",
 >                  "build_id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
 >                  "path": "trinity",
 >                  "output_files": [
 >                      {
 >                          "name": "dmesg.xz",
 >                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/dmesg.xz",
 >                      },
 >                      {
 >                          "name": "details.txt",
 >                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/trinity_details.log",
 >                      }
 >                  ],
 >                  "environment": {
 >                      "description": "qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 16G"
 >                  },
 >                  "start_time": "2020-07-17T09:04:55+03:00",
 >                  "status": "FAIL",
 >                  "waived": false
 >              },
 >          ],
 >          "version": {
 >              "major": 3,
 >              "minor": 0
 >          }
 >      }
 >
 > Note the "contacts" field all revisions have: this will help us determine who
 > to send the reports to.
 >
 > Perhaps we need to add support for test input files to accommodate your
 > reproduction instructions and custom scripts.
 >
 > I have omitted some fields I could've added, and we need to improve the schema
 > to accommodate your reports better, of course.
 >
 > However, if you'd be interested, we could help you set up forwarding your
 > reports to KernelCI. You can start very simple and small, as the schema only
 > requires a handful of fields. This will help us see your needs: what data you
 > want in reports and on the dashboard, how many reports you want to push (both
 > positive and negative), etc.
 >
 > Don't hesitate to write with questions, suggestions, and hope to "see" you at
 > this year's Plumbers, where we hopefully will be presenting more about this
 > effort. I'll also be writing a separate article introducing the schema this
 > week, will copy both you and Dmitry here.
 >
 > Nick
 >
 > On 7/13/20 3:19 AM, Philip Li wrote:
 >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >>> Hi Philip,
 >>>
 >>> On 7/9/20 2:00 PM, Philip Li wrote:
 >>>> The 0-day ci mostly focus on regression and then bisection, and the strategy
 >>>> is to merge various branches to run the test. This is not exactly as the
 >>>> tranditional CI. The worry here is to know exactly one branch is pass or
 >>>> fail currently is not 100% available. For instance, the final merged branch
 >>>> is fail doesn't provide fail/pass info of each individual branch. This triggers
 >>>> bisection to kick out the bad branch.
 >>>>
 >>>> Then it need redo the testing of remaining ones, which is not always
 >>>> feasible for us (considering the computing power).
 >>>
 >>> Yes, I think everyone here could sympathise with limited hardware resources :)
 >>>
 >>>> Especially, sometimes the bisection would fail.
 >>>
 >>> Can you give an example of how would a bisection fail?
 >>> Would that be a flaky test failing on a previously assumed-good commit, for
 >>> example?
 >> thanks, one example is build issue that breaks the bisectability which can
 >> lead to bisect fail.
 >>
 >>>
 >>>> As we focus on regression a lot to bisect to first bad commit, there would
 >>>> be uncertainty to draw conclusion for single branch.
 >>>
 >>> I think not having complete certainty for a project as large as the Linux
 >>> kernel is normal. Kernel CI has the bisection system as well, and syzbot is
 >>> going to even greater lengths with identifying similar failures. We at CKI
 >>> have test maintainers constantly looking at test failures and deciding whether
 >>> they're false or not. These are things we just have to handle for common
 >>> reporting to work.
 >>>
 >>>> This requires more careful thinking for us without increasing the needs of
 >>>> computing resource. This is one bottleneck I can see so far. Not sure any
 >>>> idea or recommendation for this.
 >>>
 >>> Our aim with common reporting is simply to provide a unified way to reach
 >>> developers with testing results, essentially to send them a single e-mail
 >>> report, instead of one report per CI system, to make a single database
 >>> available for analyzis and a single dashboard UI.
 >>>
 >>> I.e. instead of sending an e-mail report to a developer we ask you to send a
 >>> JSON report to us, and then we try to handle analyzing and reporting for you.
 >>>
 >>> It is up to the submitting CI system to choose how many, or how little tests
 >>> to run, or how much or how little data to send. Kernel CI is not going to ask
 >>> you to run any tests, it is up to you.
 >>>
 >>> In the end, we trust you want the developers to notice and fix the problems
 >>> you find, you'll try to provide enough data, and we'd like to make a system
 >>> which will help you do that. If you can pinpoint the exact commit - great! If
 >>> not, we'll just have some data from you which can be analyzed otherwise.
 >>>
 >>> How about I try to take a 0-day report and express it as a KCIDB submission,
 >>> as an illustration of how this could work? Would that help you understand what
 >>> we're trying to do? If yes, could you give me a link to one?
 >> Right, the "accuracy" for all single branch we test is more related to technical
 >> problem if we need look for a way to solve it. Here assume we have a report, it
 >> does have chance to be aggeragated. You can pick up any link from
 >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
 >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
 >>
 >> Thanks
 >>
 >>>
 >>> Nick
 >>>
 >>
 >

Re: Working with the KernelCI project

[Philip Li]

On Mon, Aug 03, 2020 at 12:11:59PM +0300, Nikolai Kondrashov wrote:
> Hi Philip,
> 
> On 7/22/20 3:42 PM, Nikolai Kondrashov wrote:
> > On 7/13/20 3:19 AM, Philip Li wrote:
> >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
> >>> How about I try to take a 0-day report and express it as a KCIDB submission,
> >>> as an illustration of how this could work? Would that help you understand what
> >>> we're trying to do? If yes, could you give me a link to one?
> >> Right, the "accuracy" for all single branch we test is more related to technical
> >> problem if we need look for a way to solve it. Here assume we have a report, it
> >> does have chance to be aggeragated. You can pick up any link from
> >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
> >
> > I have omitted some fields I could've added, and we need to improve the schema
> > to accommodate your reports better, of course.
> >
> > However, if you'd be interested, we could help you set up forwarding your
> > reports to KernelCI. You can start very simple and small, as the schema only
> > requires a handful of fields. This will help us see your needs: what data you
> > want in reports and on the dashboard, how many reports you want to push (both
> > positive and negative), etc.
> >
> > Don't hesitate to write with questions, suggestions, and hope to "see" you at
> > this year's Plumbers, where we hopefully will be presenting more about this
> > effort. I'll also be writing a separate article introducing the schema this
> > week, will copy both you and Dmitry here.
> 
> Did you have time to take a look at my mock-up? It probably looks like too
Sorry for late notice, I was occupied by internal stuff. Please allow more time
for me to further follow up. The mock up is very clear and helpful. I will
provide a few more inputs/needs to discuss with you within the week.

> much text, but please don't hesitate to reach me with any questions or
> suggestions you have as you look through it, if you haven't already.
> 
> Please note that you don't have to start with sending all this data. In fact,
> only a few structural fields are required and we can start with the absolute
> minimum. The mock-up is showing what you *could* already send, but you don't
> have to go that far.
> 
> I'd like to schedule a hacking session at Plumbers where we go and try to get
> some data out of participants' systems and into KCIDB. Would you be interested
> to attend, perhaps?
yes, i'm interested to attend this, though I'm still under approval process
for plumbers. I will let you know whether i can attend finally.

> 
> Nick
> 
> On 7/22/20 3:42 PM, Nikolai Kondrashov wrote:
> > Hi Philip,
> >
> > Re-sending this to add the kernelci maillist and a colleague, sorry.
> > Please reply to *this* message instead of the first one.
> >
> > On 7/13/20 3:19 AM, Philip Li wrote:
> >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
> >>> How about I try to take a 0-day report and express it as a KCIDB submission,
> >>> as an illustration of how this could work? Would that help you understand what
> >>> we're trying to do? If yes, could you give me a link to one?
> >> Right, the "accuracy" for all single branch we test is more related to technical
> >> problem if we need look for a way to solve it. Here assume we have a report, it
> >> does have chance to be aggeragated. You can pick up any link from
> >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
> >
> > Alright, first I'll copy the KCIDB intro I sent Dmitry in another branch of
> > this thread, in case you didn't read it (otherwise skip past END-OF-INTRO):
> >
> > You can submit all your data by just piping
> > JSON to the "kcidb-submit" tool (providing some options for authentication and
> > destination), or you could use the Python 3 API.
> >
> > The expected JSON data is a dictionary of arrays, along with the schema
> > version identifier. Each of the arrays in the dictionary could be missing or
> > empty, but otherwise they contain various report objects. The database will
> > accept them in any (reasonable) amount, order, or combination, although
> > submission order can affect when and how notifications are sent out.
> >
> > Every object has an ID which can be used to refer to it and link objects
> > together, but all the IDs are generated by the submitter, who is making sure
> > they're unique. For most of the objects (except revisions at this moment) you
> > can just use your CI system's ID for it and prefix it with your CI system's
> > name. This way you don't have to maintain a mapping between your system's IDs
> > and our IDs when you report results gradually. If you don't have that, you can
> > just generate them, for example hash some key fields or, as the last resort,
> > use UUIDs. Revisions are just using commit hashes at the moment.
> >
> > Every object has a special property called "misc" which can contain arbitrary
> > data. You could use that to submit data we don't have a schema for yet. The
> > users/developers will be able to see it, and we can use that as samples for
> > implementing future support.
> >
> > See example use of "misc" for a KernelCI test and its execution environment
> > (yes, it could be formatted better):
> > https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
> >
> >                                 END-OF-INTRO
> >
> > I think it's great that 0-day e-mails contain everything needed to investigate
> > and reproduce the issue and are self-sufficient. However, at the moment KCIDB
> > doesn't allow embedding artifacts or logs into submissions, but instead
> > expects them to be stored somewhere else and have the URLs provided.
> >
> > At the moment we would need you to do that, at least with .config files, but
> > if you can't, we can work on supporting embedding them. We planned on copying
> > linked files to KernelCI-managed storage anyway, eventually.
> >
> > KCIDB also doesn't support embedding the nice error summaries and log excerpts
> > you include in your reports, but that should be quite easy to amend by storing
> > them in the database itself (provided they're not too big). Just needs a
> > little thinking about the exact schema to use.
> >
> > For now, though, my examples below assume you can provide links to files
> > (spoofed here), and you're supplying the excerpts or complete logs with those.
> >
> > KCIDB doesn't support source code linters and static analysis (such as
> > checkpatch, or coverity) at the moment, and we should add that. However,
> > a sparse run could be expressed in KCIDB as a build.
> >
> > Taking this report as a sample:
> >
> >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/GUFRHHPKTWFYNLRH4LE2E2YELI6XG2IE/
> >
> > this is how a submission could look:
> >
> >      {
> >          "revisions": [
> >              {
> >                  "id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
> >                  "origin": "0day",
> >                  "discovery_time": "2020-07-08T07:57:24+03:00",
> >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git",
> >                  "git_repository_commit_hash": "391e437eedc0dab0a9f2c26997e68e040ae04ea3",
> >                  "git_repository_branch": "master",
> >                  "patch_mboxes": [
> >                      {
> >                          "name": "0001-irqchip-stm32-exti-map-direct-event-to-irq-parent.patch",
> >                          "url": "https://github.com/0day-ci/linux/commit/3f47dd3217f24edfd442b35784001979e7aeacc7.patch"
> >                      }
> >                  ],
> >                  "message_id": "20200706081106.25125-1-alexandre.torgue@st.com",
> >                  "contacts": [
> >                      "Alexandre Torgue <alexandre.torgue@st.com>",
> >                      "kbuild-all <kbuild-all@lists.01.org>",
> >                      "Marc Zyngier <maz@kernel.org>",
> >                      "Thomas Gleixner <tglx@linutronix.de>",
> >                      "Jason Cooper <jason@lakedaemon.net>",
> >                      "LKML <linux-kernel@vger.kernel.org>"
> >                  ],
> >                  "valid": true
> >              }
> >          ],
> >          "builds": [
> >              {
> >                  "id": "0day:391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c:sparse",
> >                  "origin": "0day",
> >                  "revision_id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
> >                  "architecture": "arm",
> >                  "compiler": "arm-linux-gnueabi-gcc (GCC) 9.3.0",
> >                  "start_time": "2020-07-08T07:57:24+03:00",
> >                  "config_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/config",
> >                  "log_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_build.log",
> >                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=arm",
> >                  "input_files": [
> >                      {
> >                          "name": "make.cross",
> >                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
> >                      },
> >                      {
> >                          "name": "instructions.txt",
> >                          "url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_repro.txt"
> >                      }
> >                  ],
> >                  "valid": false
> >              }
> >          ],
> >          "version": {
> >              "major": 3,
> >              "minor": 0
> >          }
> >      }
> >
> >
> > The above describes the revision you're testing as a patch being applied to a
> > particular commit in the stm32 repo's master branch. The revision has a build,
> > which failed, the build has the config URL and the log linked, as well as the
> > reproduction instructions linked as one of the "input files". We can work on
> > adding a dedicated field for reproduction instructinos for both builds and
> > tests, since they're very useful and syzbot also produces them.
> >
> > A failed W=1 build:
> >
> >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/WEKO2YRAZIPZFUQAY2D4XAOWJGC3HGBD/
> >
> > Would look similar:
> >
> >      {
> >          "revisions": [
> >              {
> >                  "id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
> >                  "origin": "0day",
> >                  "discovery_time": "2020-07-08T07:57:24+03:00",
> >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git",
> >                  "git_repository_commit_hash": "c46ed28dbe95844c1d15addd26ff05499057c4d5",
> >                  "git_repository_branch": "for-next",
> >                  "patch_mboxes": [
> >                      {
> >                          "name": "0001-arm64-dts-qcom-sc7180-Add-lpass-cpu-node-for-I2S-dri.patch",
> >                          "url": "https://github.com/0day-ci/linux/commit/d20696ca206ae45d9d27fbeffb23fe5431b5de9d.patch"
> >                      }
> >                  ],
> >                  "message_id": "20200716061445.628709-1-cychiang@chromium.org",
> >                  "contacts": [
> >                      "Ajit Pandey <ajitp@codeaurora.org>",
> >                      "Cheng-Yi Chiang <cychiang@chromium.org>",
> >                      "kbuild-all <kbuild-all@lists.01.org>",
> >                      "Andy Gross <agross@kernel.org>",
> >                      "Bjorn Andersson <bjorn.andersson@linaro.org>",
> >                      "Rob Herring <robh+dt@kernel.org>",
> >                      "linux-arm-msm@vger.kernel.org",
> >                      "devicetree@vger.kernel.org"
> >                  ],
> >                  "valid": true
> >              }
> >          ],
> >          "builds": [
> >              {
> >                  "id": "0day:c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2:sparse",
> >                  "origin": "0day",
> >                  "revision_id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
> >                  "architecture": "arm",
> >                  "compiler": "clang version 12.0.0",
> >                  "start_time": "2020-07-08T07:57:24+03:00",
> >                  "config_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/config",
> >                  "log_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_build.log",
> >                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64",
> >                  "input_files": [
> >                      {
> >                          "name": "make.cross",
> >                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
> >                      },
> >                      {
> >                          "name": "instructions.txt",
> >                          "url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_repro.txt"
> >                      }
> >                  ],
> >                  "valid": false
> >              }
> >          ],
> >          "version": {
> >              "major": 3,
> >              "minor": 0
> >          }
> >      }
> >
> > KCIDB also doesn't support non-runtime tests for compiled kernels (such as
> > size regression tests you're running), and we should add that, but meanwhile
> > we can accommodate them as tests without "environments".
> >
> > This one:
> >
> >      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/PX2O2OJZT2TZ7SU4VUB5ODM4KRBPTXD7/
> >
> > Could look like this:
> >
> >      {
> >          "revisions": [
> >              {
> >                  "id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
> >                  "origin": "0day",
> >                  "discovery_time": "2020-07-17T14:41:52+03:00",
> >                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git",
> >                  "git_repository_commit_hash": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
> >                  "git_repository_branch": "mount_setattr",
> >                  "contacts": [
> >                      "Christian Brauner <christian@brauner.io>",
> >                      "kbuild-all <kbuild-all@lists.01.org>"
> >                  ],
> >                  "valid": true
> >              }
> >          ],
> >          "builds": [
> >              {
> >                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
> >                  "origin": "0day",
> >                  "revision_id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
> >                  "start_time": "2020-07-17T14:41:52+03:00",
> >                  "valid": true
> >              }
> >          ],
> >          "tests": [
> >              {
> >                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2:size",
> >                  "origin": "0day",
> >                  "build_id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
> >                  "path": "size_regression",
> >                  "output_files": [
> >                      {
> >                          "name": "details.txt",
> >                          "url": "https://01.org/0day/aa63af1b08246bd31b77d056bf1d47f775cecbe2/size_regression_details.log",
> >                      }
> >                  ],
> >                  "start_time": "2020-07-17T14:41:52+03:00",
> >                  "status": "FAIL",
> >                  "waived": false
> >              },
> >          ],
> >          "version": {
> >              "major": 3,
> >              "minor": 0
> >          }
> >      }
> >
> > Since you don't provide any build information in that report, the build object
> > doesn't have any data. However, that's still valid according to the current
> > schema.
> >
> > Finally, this runtime test failure:
> >
> >      https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/ULKTFB6NGLORWH2WLEKFSFEQFSIWLT5F/
> >
> > you can report like this:
> >
> >      {
> >          "revisions": [
> >              {
> >                  "id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
> >                  "origin": "0day",
> >                  "discovery_time": "2020-07-17T09:04:55+03:00",
> >                  "git_repository_url": "https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git"
> >                  "git_repository_commit_hash": "5155be9994e557618a8312389fb4e52dfbf28a3c",
> >                  "git_repository_branch": "master",
> >                  "contacts": [
> >                      "Paul E. McKenney <paulmck@kernel.org>",
> >                      "LKP <lkp@lists.01.org>"
> >                  ],
> >                  "valid": true
> >              }
> >          ],
> >          "builds": [
> >              {
> >                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
> >                  "origin": "0day",
> >                  "revision_id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
> >                  "start_time": "2020-07-17T09:04:55+03:00",
> >                  "architecture": "i386",
> >                  "command": "make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage",
> >                  "config_url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/config",
> >                  "valid": true
> >              }
> >          ],
> >          "tests": [
> >              {
> >                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c:trinity",
> >                  "origin": "0day",
> >                  "build_id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
> >                  "path": "trinity",
> >                  "output_files": [
> >                      {
> >                          "name": "dmesg.xz",
> >                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/dmesg.xz",
> >                      },
> >                      {
> >                          "name": "details.txt",
> >                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/trinity_details.log",
> >                      }
> >                  ],
> >                  "environment": {
> >                      "description": "qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 16G"
> >                  },
> >                  "start_time": "2020-07-17T09:04:55+03:00",
> >                  "status": "FAIL",
> >                  "waived": false
> >              },
> >          ],
> >          "version": {
> >              "major": 3,
> >              "minor": 0
> >          }
> >      }
> >
> > Note the "contacts" field all revisions have: this will help us determine who
> > to send the reports to.
> >
> > Perhaps we need to add support for test input files to accommodate your
> > reproduction instructions and custom scripts.
> >
> > I have omitted some fields I could've added, and we need to improve the schema
> > to accommodate your reports better, of course.
> >
> > However, if you'd be interested, we could help you set up forwarding your
> > reports to KernelCI. You can start very simple and small, as the schema only
> > requires a handful of fields. This will help us see your needs: what data you
> > want in reports and on the dashboard, how many reports you want to push (both
> > positive and negative), etc.
> >
> > Don't hesitate to write with questions, suggestions, and hope to "see" you at
> > this year's Plumbers, where we hopefully will be presenting more about this
> > effort. I'll also be writing a separate article introducing the schema this
> > week, will copy both you and Dmitry here.
> >
> > Nick
> >
> > On 7/13/20 3:19 AM, Philip Li wrote:
> >> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
> >>> Hi Philip,
> >>>
> >>> On 7/9/20 2:00 PM, Philip Li wrote:
> >>>> The 0-day ci mostly focus on regression and then bisection, and the strategy
> >>>> is to merge various branches to run the test. This is not exactly as the
> >>>> tranditional CI. The worry here is to know exactly one branch is pass or
> >>>> fail currently is not 100% available. For instance, the final merged branch
> >>>> is fail doesn't provide fail/pass info of each individual branch. This triggers
> >>>> bisection to kick out the bad branch.
> >>>>
> >>>> Then it need redo the testing of remaining ones, which is not always
> >>>> feasible for us (considering the computing power).
> >>>
> >>> Yes, I think everyone here could sympathise with limited hardware resources :)
> >>>
> >>>> Especially, sometimes the bisection would fail.
> >>>
> >>> Can you give an example of how would a bisection fail?
> >>> Would that be a flaky test failing on a previously assumed-good commit, for
> >>> example?
> >> thanks, one example is build issue that breaks the bisectability which can
> >> lead to bisect fail.
> >>
> >>>
> >>>> As we focus on regression a lot to bisect to first bad commit, there would
> >>>> be uncertainty to draw conclusion for single branch.
> >>>
> >>> I think not having complete certainty for a project as large as the Linux
> >>> kernel is normal. Kernel CI has the bisection system as well, and syzbot is
> >>> going to even greater lengths with identifying similar failures. We at CKI
> >>> have test maintainers constantly looking at test failures and deciding whether
> >>> they're false or not. These are things we just have to handle for common
> >>> reporting to work.
> >>>
> >>>> This requires more careful thinking for us without increasing the needs of
> >>>> computing resource. This is one bottleneck I can see so far. Not sure any
> >>>> idea or recommendation for this.
> >>>
> >>> Our aim with common reporting is simply to provide a unified way to reach
> >>> developers with testing results, essentially to send them a single e-mail
> >>> report, instead of one report per CI system, to make a single database
> >>> available for analyzis and a single dashboard UI.
> >>>
> >>> I.e. instead of sending an e-mail report to a developer we ask you to send a
> >>> JSON report to us, and then we try to handle analyzing and reporting for you.
> >>>
> >>> It is up to the submitting CI system to choose how many, or how little tests
> >>> to run, or how much or how little data to send. Kernel CI is not going to ask
> >>> you to run any tests, it is up to you.
> >>>
> >>> In the end, we trust you want the developers to notice and fix the problems
> >>> you find, you'll try to provide enough data, and we'd like to make a system
> >>> which will help you do that. If you can pinpoint the exact commit - great! If
> >>> not, we'll just have some data from you which can be analyzed otherwise.
> >>>
> >>> How about I try to take a 0-day report and express it as a KCIDB submission,
> >>> as an illustration of how this could work? Would that help you understand what
> >>> we're trying to do? If yes, could you give me a link to one?
> >> Right, the "accuracy" for all single branch we test is more related to technical
> >> problem if we need look for a way to solve it. Here assume we have a report, it
> >> does have chance to be aggeragated. You can pick up any link from
> >> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> >> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
> >>
> >> Thanks
> >>
> >>>
> >>> Nick
> >>>
> >>
> >
> 

Re: Working with the KernelCI project

[Philip Li]

On Wed, Jul 22, 2020 at 03:42:41PM +0300, Nikolai Kondrashov wrote:
> Hi Philip,
> 
> Re-sending this to add the kernelci maillist and a colleague, sorry.
> Please reply to *this* message instead of the first one.
> 
> On 7/13/20 3:19 AM, Philip Li wrote:
> > On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
> > > How about I try to take a 0-day report and express it as a KCIDB submission,
> > > as an illustration of how this could work? Would that help you understand what
> > > we're trying to do? If yes, could you give me a link to one?
> > Right, the "accuracy" for all single branch we test is more related to technical
> > problem if we need look for a way to solve it. Here assume we have a report, it
> > does have chance to be aggeragated. You can pick up any link from
> > https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> > https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
> 
> Alright, first I'll copy the KCIDB intro I sent Dmitry in another branch of
> this thread, in case you didn't read it (otherwise skip past END-OF-INTRO):
> 
> You can submit all your data by just piping
> JSON to the "kcidb-submit" tool (providing some options for authentication and
> destination), or you could use the Python 3 API.
> 
> The expected JSON data is a dictionary of arrays, along with the schema
> version identifier. Each of the arrays in the dictionary could be missing or
> empty, but otherwise they contain various report objects. The database will
> accept them in any (reasonable) amount, order, or combination, although
> submission order can affect when and how notifications are sent out.
> 
> Every object has an ID which can be used to refer to it and link objects
> together, but all the IDs are generated by the submitter, who is making sure
> they're unique. For most of the objects (except revisions at this moment) you
> can just use your CI system's ID for it and prefix it with your CI system's
> name. This way you don't have to maintain a mapping between your system's IDs
> and our IDs when you report results gradually. If you don't have that, you can
> just generate them, for example hash some key fields or, as the last resort,
> use UUIDs. Revisions are just using commit hashes at the moment.
> 
> Every object has a special property called "misc" which can contain arbitrary
> data. You could use that to submit data we don't have a schema for yet. The
> users/developers will be able to see it, and we can use that as samples for
> implementing future support.
> 
> See example use of "misc" for a KernelCI test and its execution environment
> (yes, it could be formatted better):
> https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
> 
>                                END-OF-INTRO
> 
> I think it's great that 0-day e-mails contain everything needed to investigate
> and reproduce the issue and are self-sufficient. However, at the moment KCIDB
> doesn't allow embedding artifacts or logs into submissions, but instead
> expects them to be stored somewhere else and have the URLs provided.
> 
> At the moment we would need you to do that, at least with .config files, but
> if you can't, we can work on supporting embedding them. We planned on copying
> linked files to KernelCI-managed storage anyway, eventually.
> 
> KCIDB also doesn't support embedding the nice error summaries and log excerpts
> you include in your reports, but that should be quite easy to amend by storing
> them in the database itself (provided they're not too big). Just needs a
> little thinking about the exact schema to use.
> 
> For now, though, my examples below assume you can provide links to files
> (spoofed here), and you're supplying the excerpts or complete logs with those.
thanks, the samples below are helpful, i have added a few comments to discuss.

> 
> KCIDB doesn't support source code linters and static analysis (such as
> checkpatch, or coverity) at the moment, and we should add that. However,
> a sparse run could be expressed in KCIDB as a build.
> 
> Taking this report as a sample:
> 
>     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/GUFRHHPKTWFYNLRH4LE2E2YELI6XG2IE/
> 
> this is how a submission could look:
> 
>     {
>         "revisions": [
>             {
>                 "id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
>                 "origin": "0day",
>                 "discovery_time": "2020-07-08T07:57:24+03:00",
>                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git",
>                 "git_repository_commit_hash": "391e437eedc0dab0a9f2c26997e68e040ae04ea3",
>                 "git_repository_branch": "master",
>                 "patch_mboxes": [
>                     {
>                         "name": "0001-irqchip-stm32-exti-map-direct-event-to-irq-parent.patch",
>                         "url": "https://github.com/0day-ci/linux/commit/3f47dd3217f24edfd442b35784001979e7aeacc7.patch"
>                     }
>                 ],
>                 "message_id": "20200706081106.25125-1-alexandre.torgue@st.com",
>                 "contacts": [
>                     "Alexandre Torgue <alexandre.torgue@st.com>",
>                     "kbuild-all <kbuild-all@lists.01.org>",
>                     "Marc Zyngier <maz@kernel.org>",
>                     "Thomas Gleixner <tglx@linutronix.de>",
>                     "Jason Cooper <jason@lakedaemon.net>",
>                     "LKML <linux-kernel@vger.kernel.org>"
>                 ],
>                 "valid": true
>             }
>         ],
>         "builds": [
>             {
>                 "id": "0day:391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c:sparse",
>                 "origin": "0day",
>                 "revision_id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
>                 "architecture": "arm",
>                 "compiler": "arm-linux-gnueabi-gcc (GCC) 9.3.0",
>                 "start_time": "2020-07-08T07:57:24+03:00",
>                 "config_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/config",
>                 "log_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_build.log",
>                 "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=arm",
>                 "input_files": [
>                     {
>                         "name": "make.cross",
>                         "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
>                     },
>                     {
>                         "name": "instructions.txt",
>                         "url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_repro.txt"
>                     }
>                 ],
>                 "valid": false
>             }
>         ],
>         "version": {
>             "major": 3,
>             "minor": 0
>         }
>     }
> 
> 
> The above describes the revision you're testing as a patch being applied to a
> particular commit in the stm32 repo's master branch. The revision has a build,
> which failed, the build has the config URL and the log linked, as well as the
Thanks, besides the final result like success or failed, does the db currently
support start but not completed build (or planned but not start)?

> reproduction instructions linked as one of the "input files". We can work on
> adding a dedicated field for reproduction instructinos for both builds and
> tests, since they're very useful and syzbot also produces them.
> 
> A failed W=1 build:
> 
>     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/WEKO2YRAZIPZFUQAY2D4XAOWJGC3HGBD/
> 
> Would look similar:
> 
>     {
>         "revisions": [
>             {
>                 "id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
>                 "origin": "0day",
>                 "discovery_time": "2020-07-08T07:57:24+03:00",
>                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git",
>                 "git_repository_commit_hash": "c46ed28dbe95844c1d15addd26ff05499057c4d5",
>                 "git_repository_branch": "for-next",
>                 "patch_mboxes": [
>                     {
>                         "name": "0001-arm64-dts-qcom-sc7180-Add-lpass-cpu-node-for-I2S-dri.patch",
>                         "url": "https://github.com/0day-ci/linux/commit/d20696ca206ae45d9d27fbeffb23fe5431b5de9d.patch"
>                     }
>                 ],
>                 "message_id": "20200716061445.628709-1-cychiang@chromium.org",
>                 "contacts": [
>                     "Ajit Pandey <ajitp@codeaurora.org>",
>                     "Cheng-Yi Chiang <cychiang@chromium.org>",
>                     "kbuild-all <kbuild-all@lists.01.org>",
>                     "Andy Gross <agross@kernel.org>",
>                     "Bjorn Andersson <bjorn.andersson@linaro.org>",
>                     "Rob Herring <robh+dt@kernel.org>",
>                     "linux-arm-msm@vger.kernel.org",
>                     "devicetree@vger.kernel.org"
>                 ],
>                 "valid": true
>             }
>         ],
>         "builds": [
>             {
>                 "id": "0day:c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2:sparse",
>                 "origin": "0day",
>                 "revision_id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
>                 "architecture": "arm",
>                 "compiler": "clang version 12.0.0",
>                 "start_time": "2020-07-08T07:57:24+03:00",
>                 "config_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/config",
>                 "log_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_build.log",
>                 "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64",
>                 "input_files": [
>                     {
>                         "name": "make.cross",
>                         "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
>                     },
>                     {
>                         "name": "instructions.txt",
>                         "url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_repro.txt"
>                     }
>                 ],
>                 "valid": false
>             }
>         ],
>         "version": {
>             "major": 3,
>             "minor": 0
>         }
>     }
> 
> KCIDB also doesn't support non-runtime tests for compiled kernels (such as
> size regression tests you're running), and we should add that, but meanwhile
> we can accommodate them as tests without "environments".
> 
> This one:
> 
>     https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/PX2O2OJZT2TZ7SU4VUB5ODM4KRBPTXD7/
> 
> Could look like this:
> 
>     {
>         "revisions": [
>             {
>                 "id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
>                 "origin": "0day",
>                 "discovery_time": "2020-07-17T14:41:52+03:00",
>                 "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git",
>                 "git_repository_commit_hash": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
>                 "git_repository_branch": "mount_setattr",
>                 "contacts": [
>                     "Christian Brauner <christian@brauner.io>",
>                     "kbuild-all <kbuild-all@lists.01.org>"
>                 ],
>                 "valid": true
>             }
>         ],
>         "builds": [
>             {
>                 "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
>                 "origin": "0day",
>                 "revision_id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
>                 "start_time": "2020-07-17T14:41:52+03:00",
>                 "valid": true
>             }
>         ],
>         "tests": [
>             {
>                 "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2:size",
>                 "origin": "0day",
>                 "build_id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
>                 "path": "size_regression",
>                 "output_files": [
>                     {
>                         "name": "details.txt",
>                         "url": "https://01.org/0day/aa63af1b08246bd31b77d056bf1d47f775cecbe2/size_regression_details.log",
>                     }
>                 ],
>                 "start_time": "2020-07-17T14:41:52+03:00",
>                 "status": "FAIL",
>                 "waived": false
>             },
>         ],
>         "version": {
>             "major": 3,
>             "minor": 0
>         }
>     }
> 
> Since you don't provide any build information in that report, the build object
> doesn't have any data. However, that's still valid according to the current
> schema.
If the commit from test result is same as build report's commit, does it mean
when uploading test result, we need figure out the revision's id and builds' id
to be part of test result's joson, if they have been uploaded during build result?

> 
> Finally, this runtime test failure:
> 
>     https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/ULKTFB6NGLORWH2WLEKFSFEQFSIWLT5F/
> 
> you can report like this:
> 
>     {
>         "revisions": [
>             {
>                 "id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
>                 "origin": "0day",
>                 "discovery_time": "2020-07-17T09:04:55+03:00",
>                 "git_repository_url": "https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git"
>                 "git_repository_commit_hash": "5155be9994e557618a8312389fb4e52dfbf28a3c",
>                 "git_repository_branch": "master",
>                 "contacts": [
>                     "Paul E. McKenney <paulmck@kernel.org>",
>                     "LKP <lkp@lists.01.org>"
>                 ],
>                 "valid": true
>             }
>         ],
>         "builds": [
>             {
>                 "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
>                 "origin": "0day",
>                 "revision_id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
>                 "start_time": "2020-07-17T09:04:55+03:00",
>                 "architecture": "i386",
>                 "command": "make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage",
>                 "config_url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/config",
>                 "valid": true
>             }
>         ],
>         "tests": [
>             {
>                 "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c:trinity",
>                 "origin": "0day",
>                 "build_id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
>                 "path": "trinity",
>                 "output_files": [
>                     {
>                         "name": "dmesg.xz",
>                         "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/dmesg.xz",
>                     },
>                     {
>                         "name": "details.txt",
>                         "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/trinity_details.log",
>                     }
>                 ],
>                 "environment": {
>                     "description": "qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 16G"
>                 },
>                 "start_time": "2020-07-17T09:04:55+03:00",
>                 "status": "FAIL",
>                 "waived": false
>             },
>         ],
>         "version": {
>             "major": 3,
>             "minor": 0
>         }
>     }
> 
> Note the "contacts" field all revisions have: this will help us determine who
> to send the reports to.
is this a feature to be triggered by the db to send reports? As 0day sends
mail to notify the bad commit to related contacts to be aware of the problem.

> 
> Perhaps we need to add support for test input files to accommodate your
> reproduction instructions and custom scripts.
> 
> I have omitted some fields I could've added, and we need to improve the schema
> to accommodate your reports better, of course.
> 
> However, if you'd be interested, we could help you set up forwarding your
> reports to KernelCI. You can start very simple and small, as the schema only
> requires a handful of fields. This will help us see your needs: what data you
> want in reports and on the dashboard, how many reports you want to push (both
> positive and negative), etc.
Thanks for the offering, the aggregation is a nice feature, but we are fully
occupied by planned effort and it's hard to do actual implementation level thing
in short term. Meanwhile I'd like to know more about it, and look fwd to the
hacking session if i could attend it.

> 
> Don't hesitate to write with questions, suggestions, and hope to "see" you at
> this year's Plumbers, where we hopefully will be presenting more about this
> effort. I'll also be writing a separate article introducing the schema this
> week, will copy both you and Dmitry here.
> 
> Nick
> 
> On 7/13/20 3:19 AM, Philip Li wrote:
> > On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
> > > Hi Philip,
> > > 
> > > On 7/9/20 2:00 PM, Philip Li wrote:
> > > > The 0-day ci mostly focus on regression and then bisection, and the strategy
> > > > is to merge various branches to run the test. This is not exactly as the
> > > > tranditional CI. The worry here is to know exactly one branch is pass or
> > > > fail currently is not 100% available. For instance, the final merged branch
> > > > is fail doesn't provide fail/pass info of each individual branch. This triggers
> > > > bisection to kick out the bad branch.
> > > > 
> > > > Then it need redo the testing of remaining ones, which is not always
> > > > feasible for us (considering the computing power).
> > > 
> > > Yes, I think everyone here could sympathise with limited hardware resources :)
> > > 
> > > > Especially, sometimes the bisection would fail.
> > > 
> > > Can you give an example of how would a bisection fail?
> > > Would that be a flaky test failing on a previously assumed-good commit, for
> > > example?
> > thanks, one example is build issue that breaks the bisectability which can
> > lead to bisect fail.
> > 
> > > 
> > > > As we focus on regression a lot to bisect to first bad commit, there would
> > > > be uncertainty to draw conclusion for single branch.
> > > 
> > > I think not having complete certainty for a project as large as the Linux
> > > kernel is normal. Kernel CI has the bisection system as well, and syzbot is
> > > going to even greater lengths with identifying similar failures. We at CKI
> > > have test maintainers constantly looking at test failures and deciding whether
> > > they're false or not. These are things we just have to handle for common
> > > reporting to work.
> > > 
> > > > This requires more careful thinking for us without increasing the needs of
> > > > computing resource. This is one bottleneck I can see so far. Not sure any
> > > > idea or recommendation for this.
> > > 
> > > Our aim with common reporting is simply to provide a unified way to reach
> > > developers with testing results, essentially to send them a single e-mail
> > > report, instead of one report per CI system, to make a single database
> > > available for analyzis and a single dashboard UI.
> > > 
> > > I.e. instead of sending an e-mail report to a developer we ask you to send a
> > > JSON report to us, and then we try to handle analyzing and reporting for you.
> > > 
> > > It is up to the submitting CI system to choose how many, or how little tests
> > > to run, or how much or how little data to send. Kernel CI is not going to ask
> > > you to run any tests, it is up to you.
> > > 
> > > In the end, we trust you want the developers to notice and fix the problems
> > > you find, you'll try to provide enough data, and we'd like to make a system
> > > which will help you do that. If you can pinpoint the exact commit - great! If
> > > not, we'll just have some data from you which can be analyzed otherwise.
> > > 
> > > How about I try to take a 0-day report and express it as a KCIDB submission,
> > > as an illustration of how this could work? Would that help you understand what
> > > we're trying to do? If yes, could you give me a link to one?
> > Right, the "accuracy" for all single branch we test is more related to technical
> > problem if we need look for a way to solve it. Here assume we have a report, it
> > does have chance to be aggeragated. You can pick up any link from
> > https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
> > https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
> > 
> > Thanks
> > 
> > > 
> > > Nick
> > > 
> > 
> 

Re: Working with the KernelCI project

[Nikolai Kondrashov]

On 8/9/20 5:25 AM, Philip Li wrote:
 > On Wed, Jul 22, 2020 at 03:42:41PM +0300, Nikolai Kondrashov wrote:
 >> On 7/13/20 3:19 AM, Philip Li wrote:
 >>> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >> The above describes the revision you're testing as a patch being applied to a
 >> particular commit in the stm32 repo's master branch. The revision has a build,
 >> which failed, the build has the config URL and the log linked, as well as the
 >
 > Thanks, besides the final result like success or failed, does the db currently
 > support start but not completed build (or planned but not start)?

Not yet, and we don't support updating the submitted objects yet either. The
database considers the results final, although you can send them in gradually.

This lets us keep the design simpler, so we can concentrate on figuring out
more controversial topics, such as which objects we need, which data we need
for each object and how we can represent it. For this reason we need your data
now, whatever it is, so we can figure it out :) Once we more-or-less settle on
that, we can consider making reporting more gradual and flexible.

 >> Since you don't provide any build information in that report, the build object
 >> doesn't have any data. However, that's still valid according to the current
 >> schema.
 >
 > If the commit from test result is same as build report's commit, does it mean
 > when uploading test result, we need figure out the revision's id and builds' id
 > to be part of test result's joson, if they have been uploaded during build result?

You don't need to know the revision ID (the commit hash) when submitting a
test result, only the build ID. But yes, you have to know it to link your test
to the build (which links to the revision, in turn). You don't have to include
the revision/build objects themselves in the same submission.

It's up to you what you use for the build ID, as long as it's unique across
all the builds you submit. E.g. you could use an existing ID from your system
which corresponds to a build, and which is available to you when you report
the test.

 >> Note the "contacts" field all revisions have: this will help us determine who
 >> to send the reports to.
 >
 > is this a feature to be triggered by the db to send reports? As 0day sends
 > mail to notify the bad commit to related contacts to be aware of the problem.

Yes, this is intended for that. Although we are currently developing the
notification system and are not sending the reports to real people yet.

 >> Perhaps we need to add support for test input files to accommodate your
 >> reproduction instructions and custom scripts.
 >>
 >> I have omitted some fields I could've added, and we need to improve the schema
 >> to accommodate your reports better, of course.
 >>
 >> However, if you'd be interested, we could help you set up forwarding your
 >> reports to KernelCI. You can start very simple and small, as the schema only
 >> requires a handful of fields. This will help us see your needs: what data you
 >> want in reports and on the dashboard, how many reports you want to push (both
 >> positive and negative), etc.
 >
 > Thanks for the offering, the aggregation is a nice feature, but we are fully
 > occupied by planned effort and it's hard to do actual implementation level thing
 > in short term. Meanwhile I'd like to know more about it, and look fwd to the
 > hacking session if i could attend it.

That's understandable, of course. If you make it, I hope I'll be able to show
how easy it is to start submitting :)

As a primer, this is literally how the interface could look:

     cat report.json | kcidb-submit -p kernelci-production -t kernelci_new

See you at Plumbers!

Nick

On 8/9/20 5:25 AM, Philip Li wrote:
 > On Wed, Jul 22, 2020 at 03:42:41PM +0300, Nikolai Kondrashov wrote:
 >> Hi Philip,
 >>
 >> Re-sending this to add the kernelci maillist and a colleague, sorry.
 >> Please reply to *this* message instead of the first one.
 >>
 >> On 7/13/20 3:19 AM, Philip Li wrote:
 >>> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >>>> How about I try to take a 0-day report and express it as a KCIDB submission,
 >>>> as an illustration of how this could work? Would that help you understand what
 >>>> we're trying to do? If yes, could you give me a link to one?
 >>> Right, the "accuracy" for all single branch we test is more related to technical
 >>> problem if we need look for a way to solve it. Here assume we have a report, it
 >>> does have chance to be aggeragated. You can pick up any link from
 >>> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
 >>> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
 >>
 >> Alright, first I'll copy the KCIDB intro I sent Dmitry in another branch of
 >> this thread, in case you didn't read it (otherwise skip past END-OF-INTRO):
 >>
 >> You can submit all your data by just piping
 >> JSON to the "kcidb-submit" tool (providing some options for authentication and
 >> destination), or you could use the Python 3 API.
 >>
 >> The expected JSON data is a dictionary of arrays, along with the schema
 >> version identifier. Each of the arrays in the dictionary could be missing or
 >> empty, but otherwise they contain various report objects. The database will
 >> accept them in any (reasonable) amount, order, or combination, although
 >> submission order can affect when and how notifications are sent out.
 >>
 >> Every object has an ID which can be used to refer to it and link objects
 >> together, but all the IDs are generated by the submitter, who is making sure
 >> they're unique. For most of the objects (except revisions at this moment) you
 >> can just use your CI system's ID for it and prefix it with your CI system's
 >> name. This way you don't have to maintain a mapping between your system's IDs
 >> and our IDs when you report results gradually. If you don't have that, you can
 >> just generate them, for example hash some key fields or, as the last resort,
 >> use UUIDs. Revisions are just using commit hashes at the moment.
 >>
 >> Every object has a special property called "misc" which can contain arbitrary
 >> data. You could use that to submit data we don't have a schema for yet. The
 >> users/developers will be able to see it, and we can use that as samples for
 >> implementing future support.
 >>
 >> See example use of "misc" for a KernelCI test and its execution environment
 >> (yes, it could be formatted better):
 >> https://staging.kernelci.org:3000/d/test/test?orgId=1&var-id=kernelci:staging.kernelci.org:5ef9ab28baa38e14753eeeec
 >>
 >>                                 END-OF-INTRO
 >>
 >> I think it's great that 0-day e-mails contain everything needed to investigate
 >> and reproduce the issue and are self-sufficient. However, at the moment KCIDB
 >> doesn't allow embedding artifacts or logs into submissions, but instead
 >> expects them to be stored somewhere else and have the URLs provided.
 >>
 >> At the moment we would need you to do that, at least with .config files, but
 >> if you can't, we can work on supporting embedding them. We planned on copying
 >> linked files to KernelCI-managed storage anyway, eventually.
 >>
 >> KCIDB also doesn't support embedding the nice error summaries and log excerpts
 >> you include in your reports, but that should be quite easy to amend by storing
 >> them in the database itself (provided they're not too big). Just needs a
 >> little thinking about the exact schema to use.
 >>
 >> For now, though, my examples below assume you can provide links to files
 >> (spoofed here), and you're supplying the excerpts or complete logs with those.
 > thanks, the samples below are helpful, i have added a few comments to discuss.
 >
 >>
 >> KCIDB doesn't support source code linters and static analysis (such as
 >> checkpatch, or coverity) at the moment, and we should add that. However,
 >> a sparse run could be expressed in KCIDB as a build.
 >>
 >> Taking this report as a sample:
 >>
 >>      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/GUFRHHPKTWFYNLRH4LE2E2YELI6XG2IE/
 >>
 >> this is how a submission could look:
 >>
 >>      {
 >>          "revisions": [
 >>              {
 >>                  "id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
 >>                  "origin": "0day",
 >>                  "discovery_time": "2020-07-08T07:57:24+03:00",
 >>                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/atorgue/stm32.git",
 >>                  "git_repository_commit_hash": "391e437eedc0dab0a9f2c26997e68e040ae04ea3",
 >>                  "git_repository_branch": "master",
 >>                  "patch_mboxes": [
 >>                      {
 >>                          "name": "0001-irqchip-stm32-exti-map-direct-event-to-irq-parent.patch",
 >>                          "url": "https://github.com/0day-ci/linux/commit/3f47dd3217f24edfd442b35784001979e7aeacc7.patch"
 >>                      }
 >>                  ],
 >>                  "message_id": "20200706081106.25125-1-alexandre.torgue@st.com",
 >>                  "contacts": [
 >>                      "Alexandre Torgue <alexandre.torgue@st.com>",
 >>                      "kbuild-all <kbuild-all@lists.01.org>",
 >>                      "Marc Zyngier <maz@kernel.org>",
 >>                      "Thomas Gleixner <tglx@linutronix.de>",
 >>                      "Jason Cooper <jason@lakedaemon.net>",
 >>                      "LKML <linux-kernel@vger.kernel.org>"
 >>                  ],
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "builds": [
 >>              {
 >>                  "id": "0day:391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c:sparse",
 >>                  "origin": "0day",
 >>                  "revision_id": "391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c",
 >>                  "architecture": "arm",
 >>                  "compiler": "arm-linux-gnueabi-gcc (GCC) 9.3.0",
 >>                  "start_time": "2020-07-08T07:57:24+03:00",
 >>                  "config_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/config",
 >>                  "log_url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_build.log",
 >>                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=arm",
 >>                  "input_files": [
 >>                      {
 >>                          "name": "make.cross",
 >>                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
 >>                      },
 >>                      {
 >>                          "name": "instructions.txt",
 >>                          "url": "https://01.org/0day/391e437eedc0dab0a9f2c26997e68e040ae04ea3+88b9f2633622d8ca67ee0357740daa51095e59fa672db64895e871d8195d777c/sparse_repro.txt"
 >>                      }
 >>                  ],
 >>                  "valid": false
 >>              }
 >>          ],
 >>          "version": {
 >>              "major": 3,
 >>              "minor": 0
 >>          }
 >>      }
 >>
 >>
 >> The above describes the revision you're testing as a patch being applied to a
 >> particular commit in the stm32 repo's master branch. The revision has a build,
 >> which failed, the build has the config URL and the log linked, as well as the
 > Thanks, besides the final result like success or failed, does the db currently
 > support start but not completed build (or planned but not start)?
 >
 >> reproduction instructions linked as one of the "input files". We can work on
 >> adding a dedicated field for reproduction instructinos for both builds and
 >> tests, since they're very useful and syzbot also produces them.
 >>
 >> A failed W=1 build:
 >>
 >>      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/WEKO2YRAZIPZFUQAY2D4XAOWJGC3HGBD/
 >>
 >> Would look similar:
 >>
 >>      {
 >>          "revisions": [
 >>              {
 >>                  "id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
 >>                  "origin": "0day",
 >>                  "discovery_time": "2020-07-08T07:57:24+03:00",
 >>                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git",
 >>                  "git_repository_commit_hash": "c46ed28dbe95844c1d15addd26ff05499057c4d5",
 >>                  "git_repository_branch": "for-next",
 >>                  "patch_mboxes": [
 >>                      {
 >>                          "name": "0001-arm64-dts-qcom-sc7180-Add-lpass-cpu-node-for-I2S-dri.patch",
 >>                          "url": "https://github.com/0day-ci/linux/commit/d20696ca206ae45d9d27fbeffb23fe5431b5de9d.patch"
 >>                      }
 >>                  ],
 >>                  "message_id": "20200716061445.628709-1-cychiang@chromium.org",
 >>                  "contacts": [
 >>                      "Ajit Pandey <ajitp@codeaurora.org>",
 >>                      "Cheng-Yi Chiang <cychiang@chromium.org>",
 >>                      "kbuild-all <kbuild-all@lists.01.org>",
 >>                      "Andy Gross <agross@kernel.org>",
 >>                      "Bjorn Andersson <bjorn.andersson@linaro.org>",
 >>                      "Rob Herring <robh+dt@kernel.org>",
 >>                      "linux-arm-msm@vger.kernel.org",
 >>                      "devicetree@vger.kernel.org"
 >>                  ],
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "builds": [
 >>              {
 >>                  "id": "0day:c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2:sparse",
 >>                  "origin": "0day",
 >>                  "revision_id": "c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2",
 >>                  "architecture": "arm",
 >>                  "compiler": "clang version 12.0.0",
 >>                  "start_time": "2020-07-08T07:57:24+03:00",
 >>                  "config_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/config",
 >>                  "log_url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_build.log",
 >>                  "command": "COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64",
 >>                  "input_files": [
 >>                      {
 >>                          "name": "make.cross",
 >>                          "url": "https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross"
 >>                      },
 >>                      {
 >>                          "name": "instructions.txt",
 >>                          "url": "https://01.org/0day/c46ed28dbe95844c1d15addd26ff05499057c4d5+3a6bc6d39be8f6c3acc50a89c648859f1ee0d638f4969ec4d2cab6c7135518c2/w=1_repro.txt"
 >>                      }
 >>                  ],
 >>                  "valid": false
 >>              }
 >>          ],
 >>          "version": {
 >>              "major": 3,
 >>              "minor": 0
 >>          }
 >>      }
 >>
 >> KCIDB also doesn't support non-runtime tests for compiled kernels (such as
 >> size regression tests you're running), and we should add that, but meanwhile
 >> we can accommodate them as tests without "environments".
 >>
 >> This one:
 >>
 >>      https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/thread/PX2O2OJZT2TZ7SU4VUB5ODM4KRBPTXD7/
 >>
 >> Could look like this:
 >>
 >>      {
 >>          "revisions": [
 >>              {
 >>                  "id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >>                  "origin": "0day",
 >>                  "discovery_time": "2020-07-17T14:41:52+03:00",
 >>                  "git_repository_url": "https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git",
 >>                  "git_repository_commit_hash": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >>                  "git_repository_branch": "mount_setattr",
 >>                  "contacts": [
 >>                      "Christian Brauner <christian@brauner.io>",
 >>                      "kbuild-all <kbuild-all@lists.01.org>"
 >>                  ],
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "builds": [
 >>              {
 >>                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >>                  "origin": "0day",
 >>                  "revision_id": "aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >>                  "start_time": "2020-07-17T14:41:52+03:00",
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "tests": [
 >>              {
 >>                  "id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2:size",
 >>                  "origin": "0day",
 >>                  "build_id": "0day:aa63af1b08246bd31b77d056bf1d47f775cecbe2",
 >>                  "path": "size_regression",
 >>                  "output_files": [
 >>                      {
 >>                          "name": "details.txt",
 >>                          "url": "https://01.org/0day/aa63af1b08246bd31b77d056bf1d47f775cecbe2/size_regression_details.log",
 >>                      }
 >>                  ],
 >>                  "start_time": "2020-07-17T14:41:52+03:00",
 >>                  "status": "FAIL",
 >>                  "waived": false
 >>              },
 >>          ],
 >>          "version": {
 >>              "major": 3,
 >>              "minor": 0
 >>          }
 >>      }
 >>
 >> Since you don't provide any build information in that report, the build object
 >> doesn't have any data. However, that's still valid according to the current
 >> schema.
 > If the commit from test result is same as build report's commit, does it mean
 > when uploading test result, we need figure out the revision's id and builds' id
 > to be part of test result's joson, if they have been uploaded during build result?
 >
 >>
 >> Finally, this runtime test failure:
 >>
 >>      https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/ULKTFB6NGLORWH2WLEKFSFEQFSIWLT5F/
 >>
 >> you can report like this:
 >>
 >>      {
 >>          "revisions": [
 >>              {
 >>                  "id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >>                  "origin": "0day",
 >>                  "discovery_time": "2020-07-17T09:04:55+03:00",
 >>                  "git_repository_url": "https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git"
 >>                  "git_repository_commit_hash": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >>                  "git_repository_branch": "master",
 >>                  "contacts": [
 >>                      "Paul E. McKenney <paulmck@kernel.org>",
 >>                      "LKP <lkp@lists.01.org>"
 >>                  ],
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "builds": [
 >>              {
 >>                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
 >>                  "origin": "0day",
 >>                  "revision_id": "5155be9994e557618a8312389fb4e52dfbf28a3c",
 >>                  "start_time": "2020-07-17T09:04:55+03:00",
 >>                  "architecture": "i386",
 >>                  "command": "make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage",
 >>                  "config_url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/config",
 >>                  "valid": true
 >>              }
 >>          ],
 >>          "tests": [
 >>              {
 >>                  "id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c:trinity",
 >>                  "origin": "0day",
 >>                  "build_id": "0day:5155be9994e557618a8312389fb4e52dfbf28a3c",
 >>                  "path": "trinity",
 >>                  "output_files": [
 >>                      {
 >>                          "name": "dmesg.xz",
 >>                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/dmesg.xz",
 >>                      },
 >>                      {
 >>                          "name": "details.txt",
 >>                          "url": "https://01.org/0day/5155be9994e557618a8312389fb4e52dfbf28a3c/trinity_details.log",
 >>                      }
 >>                  ],
 >>                  "environment": {
 >>                      "description": "qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 16G"
 >>                  },
 >>                  "start_time": "2020-07-17T09:04:55+03:00",
 >>                  "status": "FAIL",
 >>                  "waived": false
 >>              },
 >>          ],
 >>          "version": {
 >>              "major": 3,
 >>              "minor": 0
 >>          }
 >>      }
 >>
 >> Note the "contacts" field all revisions have: this will help us determine who
 >> to send the reports to.
 > is this a feature to be triggered by the db to send reports? As 0day sends
 > mail to notify the bad commit to related contacts to be aware of the problem.
 >
 >>
 >> Perhaps we need to add support for test input files to accommodate your
 >> reproduction instructions and custom scripts.
 >>
 >> I have omitted some fields I could've added, and we need to improve the schema
 >> to accommodate your reports better, of course.
 >>
 >> However, if you'd be interested, we could help you set up forwarding your
 >> reports to KernelCI. You can start very simple and small, as the schema only
 >> requires a handful of fields. This will help us see your needs: what data you
 >> want in reports and on the dashboard, how many reports you want to push (both
 >> positive and negative), etc.
 > Thanks for the offering, the aggregation is a nice feature, but we are fully
 > occupied by planned effort and it's hard to do actual implementation level thing
 > in short term. Meanwhile I'd like to know more about it, and look fwd to the
 > hacking session if i could attend it.
 >
 >>
 >> Don't hesitate to write with questions, suggestions, and hope to "see" you at
 >> this year's Plumbers, where we hopefully will be presenting more about this
 >> effort. I'll also be writing a separate article introducing the schema this
 >> week, will copy both you and Dmitry here.
 >>
 >> Nick
 >>
 >> On 7/13/20 3:19 AM, Philip Li wrote:
 >>> On Thu, Jul 09, 2020 at 10:05:04PM +0300, Nikolai Kondrashov wrote:
 >>>> Hi Philip,
 >>>>
 >>>> On 7/9/20 2:00 PM, Philip Li wrote:
 >>>>> The 0-day ci mostly focus on regression and then bisection, and the strategy
 >>>>> is to merge various branches to run the test. This is not exactly as the
 >>>>> tranditional CI. The worry here is to know exactly one branch is pass or
 >>>>> fail currently is not 100% available. For instance, the final merged branch
 >>>>> is fail doesn't provide fail/pass info of each individual branch. This triggers
 >>>>> bisection to kick out the bad branch.
 >>>>>
 >>>>> Then it need redo the testing of remaining ones, which is not always
 >>>>> feasible for us (considering the computing power).
 >>>>
 >>>> Yes, I think everyone here could sympathise with limited hardware resources :)
 >>>>
 >>>>> Especially, sometimes the bisection would fail.
 >>>>
 >>>> Can you give an example of how would a bisection fail?
 >>>> Would that be a flaky test failing on a previously assumed-good commit, for
 >>>> example?
 >>> thanks, one example is build issue that breaks the bisectability which can
 >>> lead to bisect fail.
 >>>
 >>>>
 >>>>> As we focus on regression a lot to bisect to first bad commit, there would
 >>>>> be uncertainty to draw conclusion for single branch.
 >>>>
 >>>> I think not having complete certainty for a project as large as the Linux
 >>>> kernel is normal. Kernel CI has the bisection system as well, and syzbot is
 >>>> going to even greater lengths with identifying similar failures. We at CKI
 >>>> have test maintainers constantly looking at test failures and deciding whether
 >>>> they're false or not. These are things we just have to handle for common
 >>>> reporting to work.
 >>>>
 >>>>> This requires more careful thinking for us without increasing the needs of
 >>>>> computing resource. This is one bottleneck I can see so far. Not sure any
 >>>>> idea or recommendation for this.
 >>>>
 >>>> Our aim with common reporting is simply to provide a unified way to reach
 >>>> developers with testing results, essentially to send them a single e-mail
 >>>> report, instead of one report per CI system, to make a single database
 >>>> available for analyzis and a single dashboard UI.
 >>>>
 >>>> I.e. instead of sending an e-mail report to a developer we ask you to send a
 >>>> JSON report to us, and then we try to handle analyzing and reporting for you.
 >>>>
 >>>> It is up to the submitting CI system to choose how many, or how little tests
 >>>> to run, or how much or how little data to send. Kernel CI is not going to ask
 >>>> you to run any tests, it is up to you.
 >>>>
 >>>> In the end, we trust you want the developers to notice and fix the problems
 >>>> you find, you'll try to provide enough data, and we'd like to make a system
 >>>> which will help you do that. If you can pinpoint the exact commit - great! If
 >>>> not, we'll just have some data from you which can be analyzed otherwise.
 >>>>
 >>>> How about I try to take a 0-day report and express it as a KCIDB submission,
 >>>> as an illustration of how this could work? Would that help you understand what
 >>>> we're trying to do? If yes, could you give me a link to one?
 >>> Right, the "accuracy" for all single branch we test is more related to technical
 >>> problem if we need look for a way to solve it. Here assume we have a report, it
 >>> does have chance to be aggeragated. You can pick up any link from
 >>> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org/ (build) or
 >>> https://lists.01.org/hyperkitty/list/lkp@lists.01.org/ (runtime) for example.
 >>>
 >>> Thanks
 >>>
 >>>>
 >>>> Nick
 >>>>
 >>>
 >>
 >

Re: Working with the KernelCI project

[Nikolai Kondrashov]

Hi Philip,

On 8/9/20 5:25 AM, Philip Li wrote:
 > On Wed, Jul 22, 2020 at 03:42:41PM +0300, Nikolai Kondrashov wrote:
 >> The above describes the revision you're testing as a patch being applied to a
 >> particular commit in the stm32 repo's master branch. The revision has a build,
 >> which failed, the build has the config URL and the log linked, as well as the
 > Thanks, besides the final result like success or failed, does the db currently
 > support start but not completed build (or planned but not start)?

I've been thinking about this a bit and had a similar request from my
teammates at CKI Project for our internal implementation of the protocol.

What it seems we can do is to allow sending the same object multiple times,
with the condition that every new instance has the same amount of, or more
optional fields.

This would allow you to send your build or test with some basic information
first, and e.g. without output files and (importantly) with the status, and
when the build/test is done, send the same, but with previously missing fields
added.

The dashboard would be able to display the interim object as "in progress",
for example.

We don't have a working implementation yet, but it seems it should be
possible to do with minimal effort and performance hit.

 >> However, if you'd be interested, we could help you set up forwarding your
 >> reports to KernelCI. You can start very simple and small, as the schema only
 >> requires a handful of fields. This will help us see your needs: what data you
 >> want in reports and on the dashboard, how many reports you want to push (both
 >> positive and negative), etc.
 > Thanks for the offering, the aggregation is a nice feature, but we are fully
 > occupied by planned effort and it's hard to do actual implementation level thing
 > in short term. Meanwhile I'd like to know more about it, and look fwd to the
 > hacking session if i could attend it.

I've finally published the schema introduction article I promised earlier:

     https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

LPC finalized the schedule. Our talk is on Wednesday:

     https://linuxplumbersconf.org/event/7/contributions/730/

Our BoF session is on Thursday:

     https://linuxplumbersconf.org/event/7/contributions/731/

Unfortunately, we were only given 45 minutes for the latter, so we won't be
able to fit much there. However, we've been allocated a separate chat room:

     https://chat.2020.linuxplumbersconf.org/channel/kcidb

We'll be hanging out there during the whole conference, and will be able to
commandeer one of the shared video call rooms, if necessary, or start a video
chat on a 3rd party service, and discuss anything related to common reporting,
walk you through the submission process, help you set up minimal reporting,
you name it.

Hope to see you there, but if you can't make it, we're always there on the
#kernelci channel on Freenode, reachable over email, and can organize a video
call, of course.

Nick

Re: Working with the KernelCI project

[Nikolai Kondrashov]

On 8/21/20 12:50 PM, Nikolai Kondrashov wrote:
 > This would allow you to send your build or test with some basic information
 > first, and e.g. without output files and (importantly) with the status

I meant "*without* the status", of course 🤦‍♂️.

Nick