Re: Structured feeds

From: Johannes Berg <johannes@sipsolutions.net>
To: Daniel Axtens <dja@axtens.net>,
	Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
	patchwork@lists.ozlabs.org
Cc: workflows@vger.kernel.org, Kevin Hilman <khilman@baylibre.com>,
	Brendan Higgins <brendanhiggins@google.com>,
	Han-Wen Nienhuys <hanwen@google.com>,
	automated-testing@yoctoproject.org,
	Dmitry Vyukov <dvyukov@google.com>
Subject: Re: Structured feeds
Date: Sat, 09 Nov 2019 08:41:08 +0100	[thread overview]
Message-ID: <223b3b048f644b4d0fd854f9e360ed6d451c8241.camel@sipsolutions.net> (raw)
In-Reply-To: <87h83eh2op.fsf@dja-thinkpad.axtens.net>

On Sat, 2019-11-09 at 01:18 +1100, Daniel Axtens wrote:
> > 
>  - code that efficiently reads a public-inbox git repository/folder of
>    git repositories and feeds it into the existing parser. I have very
>    inefficient code that converts public-inbox to an mbox and then
>    parses that, but I'm sure you can do better with a git library.

Somebody (Daniel Borkmann?) posted a (very fast) public-inbox git to
maildir converter, with procmail support. I assume that would actually
satisfy this step already, since you can just substitute the patchwork
parser for procmail.

>  - careful thought about how to do this incrementally. It's obvious how
>    to do email incrementally, but I think you need to keep an extra bit
>    of state around to incrementally parse the git archive. I think.

Not sure he had an incremental mode figured out there, but that can't
really be all *that* hard, just store the last-successfully-parsed git
sha1?

johannes