get-lore-mbox: quickly grab full threads from lore

get-lore-mbox: quickly grab full threads from lore

[Konstantin Ryabitsev]

Hi, all:

I'd like your opinion on this quick helper script I wrote that uses any 
message-id to grab a full thread from lore.kernel.org and save it as a 
mbox file.

What's more useful, it can also prepare a mbox that you can pass 
straight to git-am:

- it will tally up all "Acked-by", "Reviewed-by" etc trailers and add 
  them to the proper patch headers
- it will properly sort patches in the series, so you don't have to do 
  it manually

Examples:

Just get the mbox file, in case you got cc'd somewhere in the middle and 
want to view the whole context:

  $ ./get-lore-mbox.py -o/tmp 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  $ mutt -f /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx

Now let's pass '--am-ready/-a' to get the same thing but in a format 
that's ready to be consumed by "git am":

  $ ./get-lore-mbox.py -o/tmp -a 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  Analyzing 280 messages in the thread
	Processing: [PATCH 00/18] add support for Clang's Shadow Call Stack
	Processing: [PATCH 01/18] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
  ...
  Found new series version: v2
    Processing: [PATCH v2 00/17] add support for Clang's Shadow Call Stack
    Processing: [PATCH v2 02/17] arm64/lib: copy_page: avoid x18 register in assembler code
  ...
  Found new series version: v7
	Processing: [PATCH v7 00/11] add support for Clang's Shadow Call Stack
	Processing: [PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
  ---
  Writing /tmp/v7_add_support_for_clang_s_shadow_call_stack.mbx
	[PATCH v7 00/11] add support for Clang's Shadow Call Stack
	[PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
	[PATCH v7 02/11] scs: add accounting
	[PATCH v7 03/11] scs: add support for stack usage debugging
	[PATCH v7 04/11] scs: disable when function graph tracing is enabled
	  Adding trailer: Reviewed-by: Kees Cook <keescook@chromium.org>
	[PATCH v7 05/11] arm64: reserve x18 from general allocation with SCS
	[PATCH v7 06/11] arm64: preserve x18 when CPU is suspended
	[PATCH v7 07/11] arm64: efi: restore x18 if it was corrupted
	[PATCH v7 08/11] arm64: vdso: disable Shadow Call Stack
	[PATCH v7 09/11] arm64: disable SCS for hypervisor code
	[PATCH v7 10/11] arm64: implement Shadow Call Stack
	[PATCH v7 11/11] arm64: scs: add shadow stacks for SDEI
  ---
  Link: https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Base-commit included, you can branch using:
	git checkout -b v7_add_support_for_clang_s_shadow_call_stack b0be0eff1a5ab77d588b76bd8b1c92d5d17b3f73
	git am /tmp/v7_add_support_for_clang_s_shadow_call_stack.mbx

You'll see that it's properly version-aware and will get you the latest 
series version (once it gets to it). Since the series properly includes 
the 'base-commit' information, it will also handily give you the git 
command to use to branch from that parent.

If you want an earlier version of that series, you can specify it with a 
-v flag:

  $ ./get-lore-mbox.py -o/tmp -a -v6 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  Analyzing 280 messages in the thread
	Ignoring v1: [PATCH 00/18] add support for Clang's Shadow Call Stack
	Ignoring v1: [PATCH 01/18] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
    ...
	Ignoring v5: Re: [PATCH v5 14/14] arm64: implement Shadow Call Stack
  Found new series version: v6
	Processing: [PATCH v6 00/15] add support for Clang's Shadow Call Stack
	Processing: [PATCH v6 01/15] arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
    ...
	Processing: Re: [PATCH v6 14/15] arm64: implement Shadow Call Stack
  Found new series version: v7
	Ignoring v7: [PATCH v7 00/11] add support for Clang's Shadow Call Stack
	Ignoring v7: [PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
    ...
	Ignoring v7: Re: [PATCH v7 04/11] scs: disable when function graph tracing is enabled
  ---
  Writing /tmp/v6_add_support_for_clang_s_shadow_call_stack.mbx
	[PATCH v6 00/15] add support for Clang's Shadow Call Stack
	[PATCH v6 01/15] arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
	[PATCH v6 02/15] arm64/lib: copy_page: avoid x18 register in assembler code
	[PATCH v6 03/15] arm64: kvm: stop treating register x18 as caller save
	[PATCH v6 04/15] arm64: kernel: avoid x18 in __cpu_soft_restart
	[PATCH v6 05/15] add support for Clang's Shadow Call Stack (SCS)
	[PATCH v6 06/15] scs: add accounting
	[PATCH v6 07/15] scs: add support for stack usage debugging
	[PATCH v6 08/15] arm64: disable function graph tracing with SCS
	[PATCH v6 09/15] arm64: reserve x18 from general allocation with SCS
	  Adding trailer: Acked-by: Will Deacon <will@kernel.org>
	[PATCH v6 10/15] arm64: preserve x18 when CPU is suspended
	  Adding trailer: Acked-by: Will Deacon <will@kernel.org>
	[PATCH v6 11/15] arm64: efi: restore x18 if it was corrupted
	[PATCH v6 12/15] arm64: vdso: disable Shadow Call Stack
	[PATCH v6 13/15] arm64: disable SCS for hypervisor code
	[PATCH v6 14/15] arm64: implement Shadow Call Stack
	[PATCH v6 15/15] arm64: scs: add shadow stacks for SDEI
  ---
  Link: https://lore.kernel.org/r/20191206221351.38241-1-samitolvanen@google.com
  Base-commit included, you can branch using:
	git checkout -b v6_add_support_for_clang_s_shadow_call_stack 3cf2890f29ab6fe491361761df558ef9191cb468
	git am /tmp/v6_add_support_for_clang_s_shadow_call_stack.mbx

You can see that the git command it gives you has a different branch and 
parent info, which should let you do an easy diff between two branches, 
one using v6, and the other using v7 versions.
 
Please give it a try and let me know your feedback! You can find it here:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-lore-mbox.py

It's a bit raw around the edges and I keep finding corner-cases, so it 
would be super interesting for me to get input from people actually 
trying to use it for managing patch series.

Best,
-K

Re: get-lore-mbox: quickly grab full threads from lore

[Kees Cook]

On Fri, Jan 31, 2020 at 10:01:05PM -0500, Konstantin Ryabitsev wrote:
> You can see that the git command it gives you has a different branch and 
> parent info, which should let you do an easy diff between two branches, 
> one using v6, and the other using v7 versions.

I love it -- I've been missing this between gerrit and patchwork, and
now I don't need a GUI at all. :)

> Please give it a try and let me know your feedback! You can find it here:
> https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-lore-mbox.py
> 
> It's a bit raw around the edges and I keep finding corner-cases, so it 
> would be super interesting for me to get input from people actually 
> trying to use it for managing patch series.

I'll give it a spin; thanks!

-- 
Kees Cook

Re: get-lore-mbox: quickly grab full threads from lore

[Dan Williams]

On Fri, Jan 31, 2020 at 7:01 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
[..]
> Please give it a try and let me know your feedback! You can find it here:
> https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-lore-mbox.py

Nice, this is probably going to replace my usage of getpatchwork [1]
because that requires a login + configuration to a single patchwork
project whereas this is any patchset from any lore list.

The ability to specify any patch in the series and pull down the full
set + cover is slick!

I assume the "get latest revision" requires that when superseding a
set that new version needs to be a reply to the first version? The
auto-supersede feature of patchwork-bot, also slick, did not require
this. Maybe patchwork-bot could reply to the old version when it sees
a supersede, in case someone forgets to thread subsequent revisions as
replies?

[1]: https://github.com/getpatchwork/patchwork

Re: get-lore-mbox: quickly grab full threads from lore

[Kevin Hilman]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> I'd like your opinion on this quick helper script I wrote that uses any 
> message-id to grab a full thread from lore.kernel.org and save it as a 
> mbox file.

This is very useful, thank you!

One question/request: Is there a way for it to only grab a subset of a
series?  e.g. Some series contain patches that might end up going
through a couple different trees (e.g. DT patches typically take a
separate path than drivers) so as a maintainer for one of the
subsystems, I might want to only get a subset of the series into an
mbox, not the whole thing.

IOW, Right now even if I pass a msgid from the middle of the series, it
finds the whole series (which is cool!), but what if I want to apply
just that single patch?  Or even better, I might want to only apply
patches 3-5 and 9 from a 10-patch series.

Is this something do-able?

Kevin

Re: get-lore-mbox: quickly grab full threads from lore

[Toke Høiland-Jørgensen]

Kevin Hilman <khilman@baylibre.com> writes:

> Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
>
>> I'd like your opinion on this quick helper script I wrote that uses any 
>> message-id to grab a full thread from lore.kernel.org and save it as a 
>> mbox file.
>
> This is very useful, thank you!
>
> One question/request: Is there a way for it to only grab a subset of a
> series?  e.g. Some series contain patches that might end up going
> through a couple different trees (e.g. DT patches typically take a
> separate path than drivers) so as a maintainer for one of the
> subsystems, I might want to only get a subset of the series into an
> mbox, not the whole thing.
>
> IOW, Right now even if I pass a msgid from the middle of the series, it
> finds the whole series (which is cool!), but what if I want to apply
> just that single patch?  Or even better, I might want to only apply
> patches 3-5 and 9 from a 10-patch series.
>
> Is this something do-able?

git am --interactive ?

-Toke

Re: get-lore-mbox: quickly grab full threads from lore

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Fri, Feb 14, 2020 at 08:40:43PM +0100, Toke Høiland-Jørgensen wrote:
> Kevin Hilman <khilman@baylibre.com> writes:

> > IOW, Right now even if I pass a msgid from the middle of the series, it
> > finds the whole series (which is cool!), but what if I want to apply
> > just that single patch?  Or even better, I might want to only apply
> > patches 3-5 and 9 from a 10-patch series.

> > Is this something do-able?

> git am --interactive ?

That's not so great if your patch application is done as part of a
script (eg, with something like aiaiai) rather than interactively.  You
can always download the full mailbox, open it and delete unwanted
messages but it'd be handy to be able to do that as part of the download
stage.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: get-lore-mbox: quickly grab full threads from lore

[Konstantin Ryabitsev]

On Fri, Feb 14, 2020 at 11:30:42AM -0800, Kevin Hilman wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
> 
> > I'd like your opinion on this quick helper script I wrote that uses any 
> > message-id to grab a full thread from lore.kernel.org and save it as a 
> > mbox file.
> 
> This is very useful, thank you!
> 
> One question/request: Is there a way for it to only grab a subset of a
> series?  e.g. Some series contain patches that might end up going
> through a couple different trees (e.g. DT patches typically take a
> separate path than drivers) so as a maintainer for one of the
> subsystems, I might want to only get a subset of the series into an
> mbox, not the whole thing.
> 
> IOW, Right now even if I pass a msgid from the middle of the series, it
> finds the whole series (which is cool!), but what if I want to apply
> just that single patch?  Or even better, I might want to only apply
> patches 3-5 and 9 from a 10-patch series.
> 
> Is this something do-able?

I think for such cases it's easy enough to just edit the .mbx file to 
remove the patches you're not interested in. When you use the '-a' flag, 
the content is de-mimefied to remove various mta-transmission cruft like 
quoted-printable/base64 encodings, etc -- making it easy to identify and 
remove content that is not relevant.

-K

Re: get-lore-mbox: quickly grab full threads from lore

[Kevin Hilman]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> On Fri, Feb 14, 2020 at 11:30:42AM -0800, Kevin Hilman wrote:
>> Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
>> 
>> > I'd like your opinion on this quick helper script I wrote that uses any 
>> > message-id to grab a full thread from lore.kernel.org and save it as a 
>> > mbox file.
>> 
>> This is very useful, thank you!
>> 
>> One question/request: Is there a way for it to only grab a subset of a
>> series?  e.g. Some series contain patches that might end up going
>> through a couple different trees (e.g. DT patches typically take a
>> separate path than drivers) so as a maintainer for one of the
>> subsystems, I might want to only get a subset of the series into an
>> mbox, not the whole thing.
>> 
>> IOW, Right now even if I pass a msgid from the middle of the series, it
>> finds the whole series (which is cool!), but what if I want to apply
>> just that single patch?  Or even better, I might want to only apply
>> patches 3-5 and 9 from a 10-patch series.
>> 
>> Is this something do-able?
>
> I think for such cases it's easy enough to just edit the .mbx file to 
> remove the patches you're not interested in.

Yes, that was my first "solution", but it's not very easy to
automate. :)

If there needs to be a manual step, I prefer 'git am --interactive'.

Anyways, this tool is really great and it's already replacing some of my
homebrew scripts.

Thanks,

Kevin

Re: get-lore-mbox: quickly grab full threads from lore

[Geert Uytterhoeven]

Hi Konstantin,

On Fri, Feb 14, 2020 at 9:36 PM Kevin Hilman <khilman@baylibre.com> wrote:
> Anyways, this tool is really great and it's already replacing some of my
> homebrew scripts.

Indeed, thanks a lot!

BTW, I found another issue: 20200118124856.GA3421@nishad produces
a backtrace when using the "-a" option.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: get-lore-mbox: quickly grab full threads from lore

[Konstantin Ryabitsev]

On Mon, Feb 17, 2020 at 11:52:55AM +0100, Geert Uytterhoeven wrote:
> BTW, I found another issue: 20200118124856.GA3421@nishad produces
> a backtrace when using the "-a" option.

Geert:

I can't replicate that. Can you please make sure that you have the 
latest version from 
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-lore-mbox.py 
?

If you still see the backtrace, please run with -d and send me the 
entire output, along with your OS/python version.

-K

Re: get-lore-mbox: quickly grab full threads from lore

[Rob Herring]

On Fri, Feb 14, 2020 at 1:53 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Fri, Feb 14, 2020 at 11:30:42AM -0800, Kevin Hilman wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:
> >
> > > I'd like your opinion on this quick helper script I wrote that uses any
> > > message-id to grab a full thread from lore.kernel.org and save it as a
> > > mbox file.
> >
> > This is very useful, thank you!
> >
> > One question/request: Is there a way for it to only grab a subset of a
> > series?  e.g. Some series contain patches that might end up going
> > through a couple different trees (e.g. DT patches typically take a
> > separate path than drivers) so as a maintainer for one of the
> > subsystems, I might want to only get a subset of the series into an
> > mbox, not the whole thing.
> >
> > IOW, Right now even if I pass a msgid from the middle of the series, it
> > finds the whole series (which is cool!), but what if I want to apply
> > just that single patch?  Or even better, I might want to only apply
> > patches 3-5 and 9 from a 10-patch series.
> >
> > Is this something do-able?
>
> I think for such cases it's easy enough to just edit the .mbx file to
> remove the patches you're not interested in. When you use the '-a' flag,
> the content is de-mimefied to remove various mta-transmission cruft like
> quoted-printable/base64 encodings, etc -- making it easy to identify and
> remove content that is not relevant.

It may be easy enough unless you have something like this series:

https://lore.kernel.org/lkml/cover.1582361737.git.mchehab+huawei@kernel.org/

It backtraces if you give it patch 2 msgid:

83c5df4acbbe0fa55a1d58d4c4a435b51cd2a7ad.1582361737.git.mchehab+huawei@kernel.org

Presumably because patch 2 and and the cover letter aren't on the same
list? Arguably, that's bad practice, but it's quite common to only
have 1 or a few patches of the whole series. If you start with the
cover letter, then it can find everything.


Also, it didn't pick up 2 out of 3 Acked/Reviewed-by tags:

https://lore.kernel.org/linux-doc/CAHLCerP_UW-6CdaOziHTY01cD_6Ou4h0Jj6mOJKj60P4GL9H=w@mail.gmail.com/
https://lore.kernel.org/linux-doc/41551c09-5443-4980-9c6f-6bc7f48aa356@www.fastmail.com/

Rob

Re: get-lore-mbox: quickly grab full threads from lore

[Konstantin Ryabitsev]

On Mon, Feb 24, 2020 at 11:39:02AM -0600, Rob Herring wrote:
> It may be easy enough unless you have something like this series:
> 
> https://lore.kernel.org/lkml/cover.1582361737.git.mchehab+huawei@kernel.org/
> 
> It backtraces if you give it patch 2 msgid:
> 
> 83c5df4acbbe0fa55a1d58d4c4a435b51cd2a7ad.1582361737.git.mchehab+huawei@kernel.org
> 
> Presumably because patch 2 and and the cover letter aren't on the same
> list? Arguably, that's bad practice, but it's quite common to only
> have 1 or a few patches of the whole series. If you start with the
> cover letter, then it can find everything.

There is currently no way to reconstitute a single thread from multiple 
sources -- this is something that we would like to change in the near 
future. If a message is sent to multiple mailing lists, then there is no 
guarantee from which list you are going to get the thread, unless you 
specify -p projname. E.g.:

./get-lore-mbox.py -o/tmp -a -p linux-doc \
  83c5df4acbbe0fa55a1d58d4c4a435b51cd2a7ad.1582361737.git.mchehab+huawei@kernel.org

However, this will only pick up follow-ups from linux-doc, so if there 
are any trailers sent to another list, get-lore-mbox will know nothing 
about them:

> Also, it didn't pick up 2 out of 3 Acked/Reviewed-by tags:
> 
> https://lore.kernel.org/linux-doc/CAHLCerP_UW-6CdaOziHTY01cD_6Ou4h0Jj6mOJKj60P4GL9H=w@mail.gmail.com/
> https://lore.kernel.org/linux-doc/41551c09-5443-4980-9c6f-6bc7f48aa356@www.fastmail.com/

Changing this behaviour requires significant rewrites of the backend in 
order to be able to mux results from different sources. This is work 
that is planned and will be happening.

For the moment, I don't have any good solutions other than fixing the 
backtrace (which is now fixed).

-K

mbox-export-patch, notmuch-export-patch

[Sean Whitton]

Hello,

On Fri 31 Jan 2020 at 10:01PM -05, Konstantin Ryabitsev wrote:

> Hi, all:
>
> I'd like your opinion on this quick helper script I wrote that uses any
> message-id to grab a full thread from lore.kernel.org and save it as a
> mbox file.
>
> What's more useful, it can also prepare a mbox that you can pass
> straight to git-am:
>
> - it will tally up all "Acked-by", "Reviewed-by" etc trailers and add
>   them to the proper patch headers
> - it will properly sort patches in the series, so you don't have to do
>   it manually

Inspired by some of this functionality, I just uploaded
'mbox-extract-patch' to Debian unstable in the 'mailscripts' package, or
here:

    script: https://git.spwhitton.name/mailscripts/tree/mbox-extract-patch

    docs: https://git.spwhitton.name/mailscripts/tree/mbox-extract-patch.1.pod

It does what get-lore-mbox.py does except without the kernel-specific
functionality, so it might be interesting for people wanting to use a
git-send-email(1)-based workflow for projects whose mailing lists are
not archived on lore.kernel.org.

mailscripts also contains a wrapper, notmuch-extract-patch, for notmuch
users.

-- 
Sean Whitton