* grokmirror and pi-piper for all emails on a list @ 2021-05-28 19:14 Luis Chamberlain 2021-05-28 19:38 ` Konstantin Ryabitsev 0 siblings, 1 reply; 8+ messages in thread From: Luis Chamberlain @ 2021-05-28 19:14 UTC (permalink / raw) To: tools Dearly super-duper admins, I just read for the first time: https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror And I am extremely excited. After trying it, it seems to work great! Just one thing, say I want to actually get all emails on the archive to be run through .procmail, how do I do that? So that it would seem I subscribed to a mailing list from the very beginning? It would be cool if I could do that per mailing list. For instance, I want to enable that for all my mailing lists except lkml. Luis ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 19:14 grokmirror and pi-piper for all emails on a list Luis Chamberlain @ 2021-05-28 19:38 ` Konstantin Ryabitsev 2021-05-28 19:46 ` Luis Chamberlain 0 siblings, 1 reply; 8+ messages in thread From: Konstantin Ryabitsev @ 2021-05-28 19:38 UTC (permalink / raw) To: Luis Chamberlain; +Cc: tools On Fri, May 28, 2021 at 12:14:56PM -0700, Luis Chamberlain wrote: > Dearly super-duper admins, > > I just read for the first time: > > https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror > > And I am extremely excited. After trying it, it seems to work great! > Just one thing, say I want to actually get all emails on the archive > to be run through .procmail, how do I do that? So that it would seem I > subscribed to a mailing list from the very beginning? It would be cool > if I could do that per mailing list. For instance, I want to enable > that for all my mailing lists except lkml. In each repository, you will find a file called pi-piper.latest. If you set it to the hash of the very first commit, the next run of pi-piper will replay all of the messages (may take ages, you've been warned). Note, that if you configured pi-piper to reshallow the repository, you will need to unshallow it first (see the article you linked above). Also of note is that the upcoming "lei" tool from public-inbox is going to be a much more robust approach to doing what pi-piper does. I really wrote it mostly so we could feed lore lists into patchwork. :) Best regards, -K ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 19:38 ` Konstantin Ryabitsev @ 2021-05-28 19:46 ` Luis Chamberlain 2021-05-28 19:53 ` Konstantin Ryabitsev 0 siblings, 1 reply; 8+ messages in thread From: Luis Chamberlain @ 2021-05-28 19:46 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: tools On Fri, May 28, 2021 at 12:38 PM Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote: > > On Fri, May 28, 2021 at 12:14:56PM -0700, Luis Chamberlain wrote: > > Dearly super-duper admins, > > > > I just read for the first time: > > > > https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror > > > > And I am extremely excited. After trying it, it seems to work great! > > Just one thing, say I want to actually get all emails on the archive > > to be run through .procmail, how do I do that? So that it would seem I > > subscribed to a mailing list from the very beginning? It would be cool > > if I could do that per mailing list. For instance, I want to enable > > that for all my mailing lists except lkml. > > In each repository, you will find a file called pi-piper.latest. If you set it > to the hash of the very first commit, the next run of pi-piper will replay all > of the messages (may take ages, you've been warned). Groovyness! > Note, that if you > configured pi-piper to reshallow the repository, you will need to unshallow it > first (see the article you linked above). OK so do you mean that if I'm starting from scratch I should use the shallow setting or make sure to not use it? > Also of note is that the upcoming "lei" tool from public-inbox is going to be > a much more robust approach to doing what pi-piper does. I really wrote it > mostly so we could feed lore lists into patchwork. :) Whoa, ok so, before I go on raving about this, I should wait instead and rave about lei later? Luis ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 19:46 ` Luis Chamberlain @ 2021-05-28 19:53 ` Konstantin Ryabitsev 2021-05-28 20:34 ` Luis Chamberlain 0 siblings, 1 reply; 8+ messages in thread From: Konstantin Ryabitsev @ 2021-05-28 19:53 UTC (permalink / raw) To: Luis Chamberlain; +Cc: tools On Fri, May 28, 2021 at 12:46:03PM -0700, Luis Chamberlain wrote: > > Note, that if you > > configured pi-piper to reshallow the repository, you will need to unshallow it > > first (see the article you linked above). > > OK so do you mean that if I'm starting from scratch I should use the > shallow setting or make sure to not use it? If you're starting from scratch, make sure you don't use it, or set it after you're done passing everything through pi-piper. > > Also of note is that the upcoming "lei" tool from public-inbox is going to be > > a much more robust approach to doing what pi-piper does. I really wrote it > > mostly so we could feed lore lists into patchwork. :) > > Whoa, ok so, before I go on raving about this, I should wait instead > and rave about lei later? You can use lei right now if you like -- I believe it is fairly stable. I believe the docs are still pretty thin, though, and may require newer sever deployments than what is currently running on lore.kernel.org. I'm supposed to be giving a talk on lei, lore and b4 at the plumbers conference this September, so I'm hoping that everything will be in a better position to "just work out of the box" closer to that time. -K ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 19:53 ` Konstantin Ryabitsev @ 2021-05-28 20:34 ` Luis Chamberlain 2021-05-28 20:46 ` Luis Chamberlain 2021-05-28 21:09 ` Konstantin Ryabitsev 0 siblings, 2 replies; 8+ messages in thread From: Luis Chamberlain @ 2021-05-28 20:34 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: tools On Fri, May 28, 2021 at 12:53 PM Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote: > > On Fri, May 28, 2021 at 12:46:03PM -0700, Luis Chamberlain wrote: > > > Note, that if you > > > configured pi-piper to reshallow the repository, you will need to unshallow it > > > first (see the article you linked above). > > > > OK so do you mean that if I'm starting from scratch I should use the > > shallow setting or make sure to not use it? > > If you're starting from scratch, make sure you don't use it, or set it after > you're done passing everything through pi-piper. OK so if starting from scratch, I should have: shallow = no Then I do a run of: grok-pull -c ~/.config/lore.conf Then in so far as the pi-piper.latest files I see the following. I'm being verbose in case it helps others: $ find .local/ -name pi-piper.latest .local/share/grokmirror/lore/lkml/git/3.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/6.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/0.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/8.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/7.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/10.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/9.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/1.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/5.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/2.git/pi-piper.latest .local/share/grokmirror/lore/lkml/git/4.git/pi-piper.latest .local/share/grokmirror/lore/linux-fsdevel/git/0.git/pi-piper.latest .local/share/grokmirror/lore/linux-block/git/0.git/pi-piper.latest .local/share/grokmirror/lore/linux-btrfs/git/0.git/pi-piper.latest .local/share/grokmirror/lore/linux-xfs/git/0.git/pi-piper.latest To not bother with epocs 0-9 of lkml, I just only edit then .local/share/grokmirror/lore/lkml/git/10.git/pi-piper.latest to say the first commit in that repository which is commit 82004facc255acc1b43a1d76ff6d6b7109ae64c8 ("Re: [PATCH -next] dmaengine: sprd: Add missing MODULE_DEVICE_TABLE"), dated Mon May 10 19:46:24 2021. Respectively I also set: .local/share/grokmirror/lore/linux-fsdevel/git/0.git/pi-piper.latest to commit 410a6ae1bea0eb6f1f932f5777ea5bb38d7fc953 ("2 messages 0621DbFD0--9") dated Wed Jun 12 15:56:20 2002 +0400 .local/share/grokmirror/lore/linux-block/git/0.git/pi-piper.latest to commit 6f9146953e5e451dadd9bf0d926a44a088120528 ("[PATCH 25/47] nvme: don't take the I/O queue q_lock in nvme_timeout") dated Fri Nov 20 17:35:20 2015 +0100 .local/share/grokmirror/lore/linux-btrfs/git/0.git/pi-piper.latest to commit 7b8de12fa160a681c63e6572f1884b2cbc541260 ("Re: Checksums and self-healing roadmap") Tue Apr 1 13:41:15 2008 -0400 .local/share/grokmirror/lore/linux-xfs/git/0.git/pi-piper.latest to commit to commit e13ad025a1b542993b0b2f2e4e4125aeb13e6717 ("TAKE 954366 - prep for i_blksize removal") dated Mon Jul 17 20:46:51 2006 +1000. I finally now do a second run: time grok-pull -c ~/.config/lore.conf And, indeed I see it now chugging on procmail doing all the emails. Indeed, I can see this is going to take a while. If only mail processing was parallelized :) Thanks for all this! > > > Also of note is that the upcoming "lei" tool from public-inbox is going to be > > > a much more robust approach to doing what pi-piper does. I really wrote it > > > mostly so we could feed lore lists into patchwork. :) > > > > Whoa, ok so, before I go on raving about this, I should wait instead > > and rave about lei later? > > You can use lei right now if you like -- I believe it is fairly stable. Where do I find such a treasure? > I believe the docs are still pretty thin, though, and may require newer sever > deployments than what is currently running on lore.kernel.org. > > I'm supposed to be giving a talk on lei, lore and b4 at the plumbers > conference this September, so I'm hoping that everything will be in a better > position to "just work out of the box" closer to that time. Neat! I'll look forward to the recordings! Luis ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 20:34 ` Luis Chamberlain @ 2021-05-28 20:46 ` Luis Chamberlain 2021-05-28 22:42 ` Luis Chamberlain 2021-05-28 21:09 ` Konstantin Ryabitsev 1 sibling, 1 reply; 8+ messages in thread From: Luis Chamberlain @ 2021-05-28 20:46 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: tools Figured folks might be interested... it took only about 5 minutes on the first run, the second run, with my settings took about 8 minutes. Not too shabby (nvme drives)! Luis ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 20:46 ` Luis Chamberlain @ 2021-05-28 22:42 ` Luis Chamberlain 0 siblings, 0 replies; 8+ messages in thread From: Luis Chamberlain @ 2021-05-28 22:42 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: tools On Fri, May 28, 2021 at 1:46 PM Luis Chamberlain <mcgrof@kernel.org> wrote: > > Figured folks might be interested... it took only about 5 minutes on > the first run, the second run, with my settings took about 8 minutes. > Not too shabby (nvme drives)! It ran fast as it only processed lkml, but 8 minutes for processing the latest epoch for lkml is also not too bad. Anyway let me know if anyone has any idea of the following: So after some debugging I am seeing that with a setup as follows, running grok-pull -c ~/.config/lore.conf seems to not run pi-piper if any of the repos being processed doesn't need an update. If I actually also force the pi-paper.latest to each repo to say, the last 3rd commit, I still don't see the last 3 emails coming up. But if I run manually against reach repo like this: grok-pi-piper -c ~/.config/pi-piper.conf .local/share/grokmirror/lore/linux-fsdevel/git/0.git/ Then it triggers it to deliver mail as expected. So I am thinking the following config may be missing something to force the post hook to really run for each repo, or there may be a bug somewhere. I removed the new lines from the include as it was not clear if having each target repo per line is fine. And if you have each repo, if spaces are allowed, or tabs are allowed. $ cat .config/lore.conf [core] toplevel = ~/.local/share/grokmirror/lore log = ${toplevel}/grokmirror.log [remote] site = https://lore.kernel.org manifest = https://lore.kernel.org/manifest.js.gz [pull] post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf refresh = 300 include = /linux-fsdevel/* /linux-block/* /linux-btrfs/* /linux-xfs/* /lkml/* ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: grokmirror and pi-piper for all emails on a list 2021-05-28 20:34 ` Luis Chamberlain 2021-05-28 20:46 ` Luis Chamberlain @ 2021-05-28 21:09 ` Konstantin Ryabitsev 1 sibling, 0 replies; 8+ messages in thread From: Konstantin Ryabitsev @ 2021-05-28 21:09 UTC (permalink / raw) To: Luis Chamberlain; +Cc: tools On Fri, May 28, 2021 at 01:34:03PM -0700, Luis Chamberlain wrote: > And, indeed I see it now chugging on procmail doing all the emails. > Indeed, I can see this is going to take a while. If only mail > processing was parallelized :) Thanks for all this! > > > > > Also of note is that the upcoming "lei" tool from public-inbox is going to be > > > > a much more robust approach to doing what pi-piper does. I really wrote it > > > > mostly so we could feed lore lists into patchwork. :) > > > > > > Whoa, ok so, before I go on raving about this, I should wait instead > > > and rave about lei later? > > > > You can use lei right now if you like -- I believe it is fairly stable. > > Where do I find such a treasure? For the moment, mostly by cloning https://public-inbox.org/public-inbox.git and then reading Documentation/lei-* The best place to ask follow-up questions would be on meta@public-inbox.org. Best, -K ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-05-28 22:43 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-28 19:14 grokmirror and pi-piper for all emails on a list Luis Chamberlain 2021-05-28 19:38 ` Konstantin Ryabitsev 2021-05-28 19:46 ` Luis Chamberlain 2021-05-28 19:53 ` Konstantin Ryabitsev 2021-05-28 20:34 ` Luis Chamberlain 2021-05-28 20:46 ` Luis Chamberlain 2021-05-28 22:42 ` Luis Chamberlain 2021-05-28 21:09 ` Konstantin Ryabitsev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).