Re: Contributing ARM tests results to KCIDB

["Nikolai Kondrashov" <Nikolai.Kondrashov@redhat.com>]

On 9/18/20 6:30 PM, Nikolai Kondrashov wrote:
 > Yes, I think it's one of the problems you uncovered :)
 >
 > The schema allows for fully-compliant RFC3339 timestamps, but the BigQuery
 > database on the backend doesn't understand some of them. In particular it
 > doesn't understand the date-only timestamps you send. E.g. "2020-09-13".
 > That's what I wanted to fix today, but ran out of time.

Looking at this more it seems that Python's jsonschema module simply doesn't
enforce the requirements we put on those fields 🤦. You can send essentially
what you want and then hit BigQuery, which is serious about them.

Sorry about that.

I opened an issue for this: https://github.com/kernelci/kcidb/issues/108

For now please just make sure your timestamp comply with RFC3339.

You can produce such a timestamp e.g. using "date --rfc-3339=s".

Nick

On 9/18/20 6:30 PM, Nikolai Kondrashov wrote:
 > On 9/18/20 6:21 PM, Cristian Marussi wrote:
 >  > So in order to carry on my experiments, I've just tried to push a new dataset
 >  > with a few changes in my data-layout to mimic what I see other origins do; this
 >  > contained something like 38 builds across 4 different revisions (with brand new
 >  > revisions IDs), but I cannot see anything on the UI: I just keep seeing the old
 >  > push from yesterday.
 >  >
 >  > JSON seems valid and kcidb-submit does not report any error even using -l DEBUG.
 >  > (I pushed >30mins ago)
 >  >
 >  > Any idea ?
 >
 > Yes, I think it's one of the problems you uncovered :)
 >
 > The schema allows for fully-compliant RFC3339 timestamps, but the BigQuery
 > database on the backend doesn't understand some of them. In particular it
 > doesn't understand the date-only timestamps you send. E.g. "2020-09-13".
 > That's what I wanted to fix today, but ran out of time.
 >
 > Additionally, the backend doesn't have a way to report a problem to the
 > submitter at the moment. We intend to fix that, but for now it's possible only
 > through us looking at the logs and sending a message to the submitter :)
 >
 > To work around this you can pad your timestamps with dummy date and time
 > data.
 >
 > E.g. instead of sending:
 >
 >      2020-09-13
 >
 > you can send:
 >
 >      2020-09-13 00:00:00+00:00
 >
 > Hopefully that's the only problem. It could be, since you managed to send data
 > before :)
 >
 > Nick
 >
 > On 9/18/20 6:21 PM, Cristian Marussi wrote:
 >  > Hi Nikolai,
 >  >
 >  > On Thu, Sep 17, 2020 at 08:26:15PM +0300, Nikolai Kondrashov wrote:
 >  >> On 9/17/20 7:22 PM, Cristian Marussi wrote:
 >  >>> It works too ... :D
 >  >>>
 >  >>> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=arm:2020-07-07:d3d7689c2cc9503266cac3bc777bb4ddae2e5f2e
 >  >>
 >  >> Whoa, awesome!
 >  >>
 >  >> And you have already uncovered a few issues we need to fix, too!
 >  >> I will deal with them tomorrow.
 >  >>
 >  >>> ..quick question though....given that now I'll have to play quite a bit
 >  >>> with it and see how's better to present our data, if anythinjg missing etc etc,
 >  >>> is there any chance (or way) that if I submmit the same JSON report multiple
 >  >>> times with slight differences here and there (but with the same IDs clearly)
 >  >>> I'll get my DB updated in the bits I have changed: as an example I've just
 >  >>> resubmitted the same report with added discovery_time and descriptions, and got
 >  >>> NO errors, but I cannot see the changes in the UI (unless they have still to
 >  >>> propagate...)..or maybe I can obtain the same effect by dropping my dataset
 >  >>> before re-submitting ?
 >  >>
 >  >> Right now it's not supported (with various possible quirks if attempted).
 >  >> So, preferably, submit only one, complete and final instance of each object
 >  >> (with unique ID) for now.
 >  >>
 >  >> We have a plan to support merging missing properties across multiple reported
 >  >> objects with the same ID.
 >  >>
 >  >>              Object A        Object B    Dashboard/Notifications
 >  >>
 >  >> FieldX:     Foo             Foo         Foo
 >  >> FieldY:                     Bar         Bar
 >  >> FieldZ:     Baz                         Baz
 >  >> FieldU:     Red             Blue        Red/Blue
 >  >>
 >  >> Since we're using a distributed database we cannot really maintain order
 >  >> (without introducing artificial global lock), so the order of the reports
 >  >> doesn't matter. We can only guarantee that a present value would override
 >  >> missing value. It would be undefined which value would be picked among
 >  >> multiple different values.
 >  >>
 >  >> This would allow gradual reporting of each object, but no editing, sorry.
 >  >>
 >  >> However, once again, this is a plan with some research done, only.
 >  >> I plan to start implementing it within a few weeks.
 >  >>
 >  >
 >  > So in order to carry on my experiments, I've just tried to push a new dataset
 >  > with a few changes in my data-layout to mimic what I see other origins do; this
 >  > contained something like 38 builds across 4 different revisions (with brand new
 >  > revisions IDs), but I cannot see anything on the UI: I just keep seeing the old
 >  > push from yesterday.
 >  >
 >  > JSON seems valid and kcidb-submit does not report any error even using -l DEBUG.
 >  > (I pushed >30mins ago)
 >  >
 >  > Any idea ?
 >  >
 >  > Thanks
 >  >
 >  > Cristian
 >  >
 >  >> Nick
 >  >>
 >  >> On 9/17/20 7:22 PM, Cristian Marussi wrote:
 >  >>> On Thu, Sep 17, 2020 at 04:52:30PM +0300, Nikolai Kondrashov wrote:
 >  >>>> Hi Christian,
 >  >>>>
 >  >>>> On 9/17/20 3:50 PM, Cristian Marussi wrote:
 >  >>>>> Hi Nikolai,
 >  >>>>>
 >  >>>>> I work at ARM in the Kernel team and, in short, we'd like certainly to
 >  >>>>> contribute our internal Kernel test results to KCIDB.
 >  >>>>
 >  >>>> Wonderful!
 >  >>>>
 >  >>>>> After having attended your LPC2020 TestMC and KernelCI/BoF, I've now cooked
 >  >>>>> up some KCIDB JSON test report (seemingly valid against your KCIDB v3 schema)
 >  >>>>> and I'd like to start experimenting with kci-submit (on non-production
 >  >>>>> instances), so as to assess how to fit our results into your schema and maybe
 >  >>>>> contribute with some new KCIDB requirements if strictly needed.
 >  >>>>
 >  >>>> Great, this is exactly what we need, welcome aboard :)
 >  >>>>
 >  >>>> Please don't hesitate to reach out on kernelci@groups.io or on #kernelci on
 >  >>>> freenode.net, if you have any questions, problems, or requirements.
 >  >>>>
 >  >>>>> Is it possible to get some valid credentials and a playground instance to
 >  >>>>> point at ?
 >  >>>>
 >  >>>> Absolutely, I created credentials for you and sent them in a separate message.
 >  >>>>
 >  >>>> You can use origin "arm" for the start, unless you have multiple CI systems
 >  >>>> and want to differentiate them somehow in your reports.
 >  >>>>
 >  >>>> Nick
 >  >>>>
 >  >>>    Thanks !
 >  >>>
 >  >>> It works too ... :D
 >  >>>
 >  >>> https://staging.kernelci.org:3000/d/build/build?orgId=1&var-dataset=playground_kernelci04&var-id=arm:2020-07-07:d3d7689c2cc9503266cac3bc777bb4ddae2e5f2e
 >  >>>
 >  >>> ..quick question though....given that now I'll have to play quite a bit
 >  >>> with it and see how's better to present our data, if anythinjg missing etc etc,
 >  >>> is there any chance (or way) that if I submmit the same JSON report multiple
 >  >>> times with slight differences here and there (but with the same IDs clearly)
 >  >>> I'll get my DB updated in the bits I have changed: as an example I've just
 >  >>> resubmitted the same report with added discovery_time and descriptions, and got
 >  >>> NO errors, but I cannot see the changes in the UI (unless they have still to
 >  >>> propagate...)..or maybe I can obtain the same effect by dropping my dataset
 >  >>> before re-submitting ?
 >  >>>
 >  >>> Regards
 >  >>>
 >  >>> Thanks
 >  >>>
 >  >>> Cristian
 >  >>>
 >  >>>> On 9/17/20 3:50 PM, Cristian Marussi wrote:
 >  >>>>> Hi Nikolai,
 >  >>>>>
 >  >>>>> I work at ARM in the Kernel team and, in short, we'd like certainly to
 >  >>>>> contribute our internal Kernel test results to KCIDB.
 >  >>>>>
 >  >>>>> After having attended your LPC2020 TestMC and KernelCI/BoF, I've now cooked
 >  >>>>> up some KCIDB JSON test report (seemingly valid against your KCIDB v3 schema)
 >  >>>>> and I'd like to start experimenting with kci-submit (on non-production
 >  >>>>> instances), so as to assess how to fit our results into your schema and maybe
 >  >>>>> contribute with some new KCIDB requirements if strictly needed.
 >  >>>>>
 >  >>>>> Is it possible to get some valid credentials and a playground instance to
 >  >>>>> point at ?
 >  >>>>>
 >  >>>>> Thanks
 >  >>>>>
 >  >>>>> Regards
 >  >>>>>
 >  >>>>> Cristian
 >  >>>>>
 >  >>>>>
 >  >>>>>
 >  >>>>>
 >  >>>>>
 >  >>>>
 >  >>>
 >  >>
 >  >
 >  >
 >  > >
 >  >
 >
 >
 >
 > 
 >
 >