From: Daniel Borkmann <daniel@iogearbox.net>
To: Johannes Berg <johannes@sipsolutions.net>,
Daniel Axtens <dja@axtens.net>,
Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
patchwork@lists.ozlabs.org
Cc: workflows@vger.kernel.org, Kevin Hilman <khilman@baylibre.com>,
Brendan Higgins <brendanhiggins@google.com>,
Han-Wen Nienhuys <hanwen@google.com>,
automated-testing@yoctoproject.org,
Dmitry Vyukov <dvyukov@google.com>
Subject: Re: Structured feeds
Date: Tue, 12 Nov 2019 11:44:29 +0100 [thread overview]
Message-ID: <a12245be-194d-4db3-a225-30d636946589@iogearbox.net> (raw)
In-Reply-To: <223b3b048f644b4d0fd854f9e360ed6d451c8241.camel@sipsolutions.net>
On 11/9/19 8:41 AM, Johannes Berg wrote:
> On Sat, 2019-11-09 at 01:18 +1100, Daniel Axtens wrote:
>>>
>> - code that efficiently reads a public-inbox git repository/folder of
>> git repositories and feeds it into the existing parser. I have very
>> inefficient code that converts public-inbox to an mbox and then
>> parses that, but I'm sure you can do better with a git library.
>
> Somebody (Daniel Borkmann?) posted a (very fast) public-inbox git to
> maildir converter, with procmail support. I assume that would actually
> satisfy this step already, since you can just substitute the patchwork
> parser for procmail.
>
>> - careful thought about how to do this incrementally. It's obvious how
>> to do email incrementally, but I think you need to keep an extra bit
>> of state around to incrementally parse the git archive. I think.
>
> Not sure he had an incremental mode figured out there, but that can't
> really be all *that* hard, just store the last-successfully-parsed git
> sha1?
Yep, that is what it is doing, so that we only need to walk the repo(s)
upon a new git fetch to the point where we stopped last time.
Thanks,
Daniel
next prev parent reply other threads:[~2019-11-12 10:44 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-05 10:02 Structured feeds Dmitry Vyukov
2019-11-06 15:35 ` Daniel Axtens
2019-11-06 20:50 ` Konstantin Ryabitsev
2019-11-07 9:08 ` Dmitry Vyukov
2019-11-07 10:57 ` Daniel Axtens
2019-11-07 11:26 ` Veronika Kabatova
2019-11-08 0:24 ` Eric Wong
2019-11-07 11:09 ` Daniel Axtens
2019-11-08 14:18 ` Daniel Axtens
2019-11-09 7:41 ` Johannes Berg
2019-11-12 10:44 ` Daniel Borkmann [this message]
[not found] ` <208edf06eb4c56a4f376caf0feced65f09d23f93.camel@that.guru>
2019-11-30 18:16 ` Johannes Berg
2019-11-30 18:36 ` Stephen Finucane
2019-11-07 8:53 ` Dmitry Vyukov
2019-11-07 10:40 ` Daniel Axtens
2019-11-07 10:43 ` Dmitry Vyukov
2019-11-07 20:43 ` [Automated-testing] " Don Zickus
2019-11-08 7:58 ` Dmitry Vyukov
2019-11-08 15:26 ` Don Zickus
2019-11-08 11:44 ` Daniel Axtens
2019-11-08 14:54 ` Don Zickus
2019-11-06 19:54 ` Han-Wen Nienhuys
2019-11-06 20:31 ` Sean Whitton
2019-11-07 9:04 ` Dmitry Vyukov
2019-11-07 8:48 ` [Automated-testing] " Tim.Bird
2019-11-07 9:13 ` Dmitry Vyukov
2019-11-07 9:20 ` Tim.Bird
2019-11-07 20:53 ` Don Zickus
2019-11-08 8:05 ` Dmitry Vyukov
2019-11-08 14:52 ` Don Zickus
2019-11-11 9:20 ` Dmitry Vyukov
2019-11-11 15:14 ` Don Zickus
2019-11-12 22:54 ` Konstantin Ryabitsev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a12245be-194d-4db3-a225-30d636946589@iogearbox.net \
--to=daniel@iogearbox.net \
--cc=automated-testing@yoctoproject.org \
--cc=brendanhiggins@google.com \
--cc=dja@axtens.net \
--cc=dvyukov@google.com \
--cc=hanwen@google.com \
--cc=johannes@sipsolutions.net \
--cc=khilman@baylibre.com \
--cc=konstantin@linuxfoundation.org \
--cc=patchwork@lists.ozlabs.org \
--cc=workflows@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).