grokmirror and pi-piper for all emails on a list

grokmirror and pi-piper for all emails on a list

[Luis Chamberlain]

Dearly super-duper admins,

I just read for the first time:

https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror

And I am extremely excited. After trying it, it seems to work great!
Just one thing, say I want to actually get all emails on the archive
to be run through .procmail, how do I do that? So that it would seem I
subscribed to a mailing list from the very beginning? It would be cool
if I could do that per mailing list. For instance, I want to enable
that for all my mailing lists except lkml.

  Luis

Re: grokmirror and pi-piper for all emails on a list

[Konstantin Ryabitsev]

On Fri, May 28, 2021 at 12:14:56PM -0700, Luis Chamberlain wrote:
> Dearly super-duper admins,
> 
> I just read for the first time:
> 
> https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror
> 
> And I am extremely excited. After trying it, it seems to work great!
> Just one thing, say I want to actually get all emails on the archive
> to be run through .procmail, how do I do that? So that it would seem I
> subscribed to a mailing list from the very beginning? It would be cool
> if I could do that per mailing list. For instance, I want to enable
> that for all my mailing lists except lkml.

In each repository, you will find a file called pi-piper.latest. If you set it
to the hash of the very first commit, the next run of pi-piper will replay all
of the messages (may take ages, you've been warned). Note, that if you
configured pi-piper to reshallow the repository, you will need to unshallow it
first (see the article you linked above).

Also of note is that the upcoming "lei" tool from public-inbox is going to be
a much more robust approach to doing what pi-piper does. I really wrote it
mostly so we could feed lore lists into patchwork. :)

Best regards,
-K

Re: grokmirror and pi-piper for all emails on a list

[Luis Chamberlain]

On Fri, May 28, 2021 at 12:38 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Fri, May 28, 2021 at 12:14:56PM -0700, Luis Chamberlain wrote:
> > Dearly super-duper admins,
> >
> > I just read for the first time:
> >
> > https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror
> >
> > And I am extremely excited. After trying it, it seems to work great!
> > Just one thing, say I want to actually get all emails on the archive
> > to be run through .procmail, how do I do that? So that it would seem I
> > subscribed to a mailing list from the very beginning? It would be cool
> > if I could do that per mailing list. For instance, I want to enable
> > that for all my mailing lists except lkml.
>
> In each repository, you will find a file called pi-piper.latest. If you set it
> to the hash of the very first commit, the next run of pi-piper will replay all
> of the messages (may take ages, you've been warned).

Groovyness!

> Note, that if you
> configured pi-piper to reshallow the repository, you will need to unshallow it
> first (see the article you linked above).

OK so do you mean that if I'm starting from scratch I should use the
shallow setting or make sure to not use it?

> Also of note is that the upcoming "lei" tool from public-inbox is going to be
> a much more robust approach to doing what pi-piper does. I really wrote it
> mostly so we could feed lore lists into patchwork. :)

Whoa, ok so, before I go on raving about this, I should wait instead
and rave about lei later?

  Luis

Re: grokmirror and pi-piper for all emails on a list

[Konstantin Ryabitsev]

On Fri, May 28, 2021 at 12:46:03PM -0700, Luis Chamberlain wrote:
> > Note, that if you
> > configured pi-piper to reshallow the repository, you will need to unshallow it
> > first (see the article you linked above).
> 
> OK so do you mean that if I'm starting from scratch I should use the
> shallow setting or make sure to not use it?

If you're starting from scratch, make sure you don't use it, or set it after
you're done passing everything through pi-piper.

> > Also of note is that the upcoming "lei" tool from public-inbox is going to be
> > a much more robust approach to doing what pi-piper does. I really wrote it
> > mostly so we could feed lore lists into patchwork. :)
> 
> Whoa, ok so, before I go on raving about this, I should wait instead
> and rave about lei later?

You can use lei right now if you like -- I believe it is fairly stable.
I believe the docs are still pretty thin, though, and may require newer sever
deployments than what is currently running on lore.kernel.org.

I'm supposed to be giving a talk on lei, lore and b4 at the plumbers
conference this September, so I'm hoping that everything will be in a better
position to "just work out of the box" closer to that time.

-K

Re: grokmirror and pi-piper for all emails on a list

[Luis Chamberlain]

On Fri, May 28, 2021 at 12:53 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Fri, May 28, 2021 at 12:46:03PM -0700, Luis Chamberlain wrote:
> > > Note, that if you
> > > configured pi-piper to reshallow the repository, you will need to unshallow it
> > > first (see the article you linked above).
> >
> > OK so do you mean that if I'm starting from scratch I should use the
> > shallow setting or make sure to not use it?
>
> If you're starting from scratch, make sure you don't use it, or set it after
> you're done passing everything through pi-piper.

OK so if starting from scratch, I should have:

shallow = no
Then I do a run of:

grok-pull -c ~/.config/lore.conf

Then in so far as the pi-piper.latest files I see the following. I'm
being verbose in case it helps others:

$ find .local/ -name pi-piper.latest
.local/share/grokmirror/lore/lkml/git/3.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/6.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/0.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/8.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/7.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/10.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/9.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/1.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/5.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/2.git/pi-piper.latest
.local/share/grokmirror/lore/lkml/git/4.git/pi-piper.latest

.local/share/grokmirror/lore/linux-fsdevel/git/0.git/pi-piper.latest
.local/share/grokmirror/lore/linux-block/git/0.git/pi-piper.latest
.local/share/grokmirror/lore/linux-btrfs/git/0.git/pi-piper.latest
.local/share/grokmirror/lore/linux-xfs/git/0.git/pi-piper.latest

To not bother with epocs 0-9 of lkml, I just only edit then
.local/share/grokmirror/lore/lkml/git/10.git/pi-piper.latest to say
the first commit in that repository which is commit
82004facc255acc1b43a1d76ff6d6b7109ae64c8 ("Re: [PATCH -next]
dmaengine: sprd: Add missing MODULE_DEVICE_TABLE"), dated Mon May 10
19:46:24 2021.

Respectively I also set:

.local/share/grokmirror/lore/linux-fsdevel/git/0.git/pi-piper.latest
to commit 410a6ae1bea0eb6f1f932f5777ea5bb38d7fc953 ("2 messages
0621DbFD0--9") dated Wed Jun 12 15:56:20 2002 +0400

.local/share/grokmirror/lore/linux-block/git/0.git/pi-piper.latest to
commit 6f9146953e5e451dadd9bf0d926a44a088120528 ("[PATCH 25/47] nvme:
don't take the I/O queue q_lock in nvme_timeout") dated Fri Nov 20
17:35:20 2015 +0100

.local/share/grokmirror/lore/linux-btrfs/git/0.git/pi-piper.latest to
commit 7b8de12fa160a681c63e6572f1884b2cbc541260 ("Re: Checksums and
self-healing roadmap") Tue Apr 1 13:41:15 2008 -0400

.local/share/grokmirror/lore/linux-xfs/git/0.git/pi-piper.latest to
commit to commit e13ad025a1b542993b0b2f2e4e4125aeb13e6717 ("TAKE
954366 - prep for i_blksize removal") dated Mon Jul 17 20:46:51 2006
+1000.

I finally now do a second run:

time grok-pull -c ~/.config/lore.conf

And, indeed I see it now chugging on procmail doing all the emails.
Indeed, I can see this is going to take a while. If only mail
processing was parallelized :) Thanks for all this!

> > > Also of note is that the upcoming "lei" tool from public-inbox is going to be
> > > a much more robust approach to doing what pi-piper does. I really wrote it
> > > mostly so we could feed lore lists into patchwork. :)
> >
> > Whoa, ok so, before I go on raving about this, I should wait instead
> > and rave about lei later?
>
> You can use lei right now if you like -- I believe it is fairly stable.

Where do I find such a treasure?

> I believe the docs are still pretty thin, though, and may require newer sever
> deployments than what is currently running on lore.kernel.org.
>
> I'm supposed to be giving a talk on lei, lore and b4 at the plumbers
> conference this September, so I'm hoping that everything will be in a better
> position to "just work out of the box" closer to that time.

Neat! I'll look forward to the recordings!

 Luis

Re: grokmirror and pi-piper for all emails on a list

[Luis Chamberlain]

Figured folks might be interested... it took only about 5 minutes on
the first run, the second run, with my settings took about 8 minutes.
Not too shabby (nvme drives)!

  Luis

Re: grokmirror and pi-piper for all emails on a list

[Konstantin Ryabitsev]

On Fri, May 28, 2021 at 01:34:03PM -0700, Luis Chamberlain wrote:
> And, indeed I see it now chugging on procmail doing all the emails.
> Indeed, I can see this is going to take a while. If only mail
> processing was parallelized :) Thanks for all this!
> 
> > > > Also of note is that the upcoming "lei" tool from public-inbox is going to be
> > > > a much more robust approach to doing what pi-piper does. I really wrote it
> > > > mostly so we could feed lore lists into patchwork. :)
> > >
> > > Whoa, ok so, before I go on raving about this, I should wait instead
> > > and rave about lei later?
> >
> > You can use lei right now if you like -- I believe it is fairly stable.
> 
> Where do I find such a treasure?

For the moment, mostly by cloning https://public-inbox.org/public-inbox.git
and then reading Documentation/lei-*

The best place to ask follow-up questions would be on meta@public-inbox.org.

Best,
-K

Re: grokmirror and pi-piper for all emails on a list

[Luis Chamberlain]

On Fri, May 28, 2021 at 1:46 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> Figured folks might be interested... it took only about 5 minutes on
> the first run, the second run, with my settings took about 8 minutes.
> Not too shabby (nvme drives)!

It ran fast as it only processed lkml, but 8 minutes for processing
the latest epoch for lkml is also not too bad. Anyway let me know if
anyone has any idea of the following:

So after some debugging I am seeing that with a setup as follows,
running grok-pull -c ~/.config/lore.conf seems to not run pi-piper if
any of the repos being processed doesn't need an update. If I actually
also force the pi-paper.latest to each repo to say, the last 3rd
commit, I still don't see the last 3 emails coming up. But if I run
manually against reach repo like this:

grok-pi-piper -c ~/.config/pi-piper.conf
.local/share/grokmirror/lore/linux-fsdevel/git/0.git/

Then it triggers it to deliver mail as expected. So I am thinking the
following config may be missing something to force the post hook to
really run for each repo, or there may be a bug somewhere. I removed
the new lines from the include as it was not clear if having each
target repo per line is fine. And if you have each repo, if spaces are
allowed, or tabs are allowed.

$ cat .config/lore.conf
[core]
toplevel = ~/.local/share/grokmirror/lore
log = ${toplevel}/grokmirror.log

[remote]
site = https://lore.kernel.org
manifest = https://lore.kernel.org/manifest.js.gz

[pull]
post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf
refresh = 300
include = /linux-fsdevel/* /linux-block/* /linux-btrfs/* /linux-xfs/* /lkml/*