Re: [Automated-testing] Structured feeds

From: Daniel Axtens <dja@axtens.net>
To: Don Zickus <dzickus@redhat.com>, patchwork@lists.ozlabs.org
Cc: Dmitry Vyukov <dvyukov@google.com>,
	workflows@vger.kernel.org, automated-testing@yoctoproject.org,
	Han-Wen Nienhuys <hanwen@google.com>,
	Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Subject: Re: [Automated-testing] Structured feeds
Date: Fri, 08 Nov 2019 22:44:37 +1100	[thread overview]
Message-ID: <87k18ah9sq.fsf@dja-thinkpad.axtens.net> (raw)
In-Reply-To: <20191107204356.kg3ddamtx74b6q4p@redhat.com>

Don Zickus <dzickus@redhat.com> writes:

> On Thu, Nov 07, 2019 at 02:35:08AM +1100, Daniel Axtens wrote:
>> > As soon as we have a bridge from plain-text emails into the structured
>> > form, we can start building everything else in the structured world.
>> > Such bridge needs to parse new incoming emails, try to make sense out
>> > of them (new patch, new patch version, comment, etc) and then push the
>> > information in structured form. Then e.g. CIs can fetch info about
>> 
>> This is an non-trivial problem, fwiw. Patchwork's email parser clocks in
>> at almost thirteen hundred lines, and that's with the benefit of the
>> Python standard library. It also regularly gets patched to handle
>> changes to email systems (e.g. DMARC), changes to git (git request-pull
>> format changed subtly in 2.14.3), the bizzare ways people send email,
>> and so on.
>
> Does it ever make sense to just use git to do the translation to structured
> json?  Git has similar logic and can easily handle its own changes.  Tools
> like git-mailinfo and git-mailsplit probably do a good chunk of the
> work today.
>
+patchwork@

So patchwork, in theory at least, is VCS-agnostic: if a mail contains a
unified-diff, we can treat it as a patch. We do have some special
handling for git pull requests, but we also have tests for parsing of
CVS and if memory serves Mercurial too. So we haven't wanted to depend
on git-specific tools. Maybe in future we will give up on that, but we
haven't yet.

Regards,
Daniel

> It wouldn't pull together series info.
>
> Just a thought.
>
> Cheers,
> Don
>
>
>
>> 
>> Patchwork does expose much of this as an API, for example for patches:
>> https://patchwork.ozlabs.org/api/patches/?order=-id so if you want to
>> build on that feel free. We can possibly add data to the API if that
>> would be helpful. (Patches are always welcome too, if you don't want to
>> wait an indeterminate amount of time.)
>> 
>> Regards,
>> Daniel
>> 
>> 
>> -- 
>> _______________________________________________
>> automated-testing mailing list
>> automated-testing@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/automated-testing