Re: Structured feeds

From: Veronika Kabatova <vkabatov@redhat.com>
To: Daniel Axtens <dja@axtens.net>, Dmitry Vyukov <dvyukov@google.com>
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
	workflows@vger.kernel.org, automated-testing@yoctoproject.org,
	Brendan Higgins <brendanhiggins@google.com>,
	Han-Wen Nienhuys <hanwen@google.com>,
	Kevin Hilman <khilman@baylibre.com>
Subject: Re: Structured feeds
Date: Thu, 7 Nov 2019 06:26:17 -0500 (EST)	[thread overview]
Message-ID: <1033825454.8368196.1573125977348.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <87tv7ggdio.fsf@dja-thinkpad.axtens.net>

----- Original Message -----
> From: "Daniel Axtens" <dja@axtens.net>
> To: "Dmitry Vyukov" <dvyukov@google.com>, "Konstantin Ryabitsev" <konstantin@linuxfoundation.org>
> Cc: workflows@vger.kernel.org, automated-testing@yoctoproject.org, "Brendan Higgins" <brendanhiggins@google.com>,
> "Han-Wen Nienhuys" <hanwen@google.com>, "Kevin Hilman" <khilman@baylibre.com>, "Veronika Kabatova"
> <vkabatov@redhat.com>
> Sent: Thursday, November 7, 2019 11:57:19 AM
> Subject: Re: Structured feeds
> 
> Dmitry Vyukov <dvyukov@google.com> writes:
> 
> > On Wed, Nov 6, 2019 at 9:50 PM Konstantin Ryabitsev
> > <konstantin@linuxfoundation.org> wrote:
> >>
> >> On Thu, Nov 07, 2019 at 02:35:08AM +1100, Daniel Axtens wrote:
> >> >This is an non-trivial problem, fwiw. Patchwork's email parser clocks
> >> >in
> >> >at almost thirteen hundred lines, and that's with the benefit of the
> >> >Python standard library. It also regularly gets patched to handle
> >> >changes to email systems (e.g. DMARC), changes to git (git request-pull
> >> >format changed subtly in 2.14.3), the bizzare ways people send email,
> >> >and so on.
> >>
> >> I'm actually very interested in seeing patchwork switch from being fed
> >> mail directly from postfix to using public-inbox repositories as its
> >> source of patches. I know it's easy enough to accomplish as-is, by
> >> piping things from public-inbox to parsemail.sh, but it would be even
> >> more awesome if patchwork learned to work with these repos natively.
> >>
> >> The way I see it:
> >>
> >> - site administrator configures upstream public-inbox feeds
> >> - a backend process clones these repositories
> >>    - if it doesn't find a refs/heads/json, then it does its own parsing
> >>      to generate a structured feed with patches/series/trailers/pull
> >>      requests, cross-referencing them by series as necessary. Something
> >>      like a subset of this, excluding patchwork-specific data:
> >>      https://patchwork.kernel.org/api/1.1/patches/11177661/
> >>    - if it does find an existing structured feed, it simply uses it (e.g.
> >>      it was made available by another patchwork instance)
> >
> > It's an interesting feature if a patchwork instance would convert and
> > export text emails to structured info. Then it can be consumed by CIs
> > for precommit testing and other systems without the need to duplicate
> > conversion.
> 
> This already happens.
> 
> Snowpatch does this and uses it to run CI checks on patch series as soon
> as they arrive, and sends them back to patchwork as test results. It has
> been running on linuxppc-dev for over a year.
> 
> Snowpatch is at https://github.com/ruscur/snowpatch
> 
> An example patch showing the checks having been run is
> https://patchwork.ozlabs.org/patch/1190589/
> 

CKI does something similar too [0].

The code contains some RHEL-specific checks as we are not running patch
testing for upstream yet. The PW checks can be submitted from the pipeline.
We should probably update the trigger to use the events API...

The only "structured information" CKI requires is to have the patch in the
correct PW project, which is mapped to a git tree/branch so we know where
to apply the patch. However there are cases when more information is needed,
such as if multiple different branches can be used with the same project, or
the patch depends on another change.

This situation should be resolved with the freeform tagging feature I
proposed a while ago (blocked by DB refactoring; original series can be found
at [1]). This feature would allow developers to add any tags to their patches,
similar to the signed-off-by line. The extracted tags can then be queried in
the API and used by CI.

I'll be totally honest and admit I ignored most of the implementation details
of public inbox feeds (will take a look when I have some free time) but as
long as they contain the original email, the feature should be usable with
them too.

[0] https://gitlab.com/cki-project/pipeline-trigger/blob/master/triggers/patch_trigger.py
[1] https://patchwork.ozlabs.org/project/patchwork/list/?series=66057

Veronika

> I think there's a different CI system used for some device-tree patches:
> e.g. https://patchwork.ozlabs.org/patch/1190714/ - I have no idea how
> this works in the backend, but it also uses the patchwork API.
> 
> Regards,
> Daniel
> 