Workflows Archive on lore.kernel.org
 help / color / Atom feed
From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: workflows@vger.kernel.org
Subject: get-lore-mbox: quickly grab full threads from lore
Date: Fri, 31 Jan 2020 22:01:05 -0500
Message-ID: <20200201030105.k6akvbjpmlpcuiky@chatter.i7.local> (raw)

Hi, all:

I'd like your opinion on this quick helper script I wrote that uses any 
message-id to grab a full thread from lore.kernel.org and save it as a 
mbox file.

What's more useful, it can also prepare a mbox that you can pass 
straight to git-am:

- it will tally up all "Acked-by", "Reviewed-by" etc trailers and add 
  them to the proper patch headers
- it will properly sort patches in the series, so you don't have to do 
  it manually

Examples:

Just get the mbox file, in case you got cc'd somewhere in the middle and 
want to view the whole context:

  $ ./get-lore-mbox.py -o/tmp 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  $ mutt -f /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx

Now let's pass '--am-ready/-a' to get the same thing but in a format 
that's ready to be consumed by "git am":

  $ ./get-lore-mbox.py -o/tmp -a 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  Analyzing 280 messages in the thread
	Processing: [PATCH 00/18] add support for Clang's Shadow Call Stack
	Processing: [PATCH 01/18] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
  ...
  Found new series version: v2
    Processing: [PATCH v2 00/17] add support for Clang's Shadow Call Stack
    Processing: [PATCH v2 02/17] arm64/lib: copy_page: avoid x18 register in assembler code
  ...
  Found new series version: v7
	Processing: [PATCH v7 00/11] add support for Clang's Shadow Call Stack
	Processing: [PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
  ---
  Writing /tmp/v7_add_support_for_clang_s_shadow_call_stack.mbx
	[PATCH v7 00/11] add support for Clang's Shadow Call Stack
	[PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
	[PATCH v7 02/11] scs: add accounting
	[PATCH v7 03/11] scs: add support for stack usage debugging
	[PATCH v7 04/11] scs: disable when function graph tracing is enabled
	  Adding trailer: Reviewed-by: Kees Cook <keescook@chromium.org>
	[PATCH v7 05/11] arm64: reserve x18 from general allocation with SCS
	[PATCH v7 06/11] arm64: preserve x18 when CPU is suspended
	[PATCH v7 07/11] arm64: efi: restore x18 if it was corrupted
	[PATCH v7 08/11] arm64: vdso: disable Shadow Call Stack
	[PATCH v7 09/11] arm64: disable SCS for hypervisor code
	[PATCH v7 10/11] arm64: implement Shadow Call Stack
	[PATCH v7 11/11] arm64: scs: add shadow stacks for SDEI
  ---
  Link: https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Base-commit included, you can branch using:
	git checkout -b v7_add_support_for_clang_s_shadow_call_stack b0be0eff1a5ab77d588b76bd8b1c92d5d17b3f73
	git am /tmp/v7_add_support_for_clang_s_shadow_call_stack.mbx

You'll see that it's properly version-aware and will get you the latest 
series version (once it gets to it). Since the series properly includes 
the 'base-commit' information, it will also handily give you the git 
command to use to branch from that parent.

If you want an earlier version of that series, you can specify it with a 
-v flag:

  $ ./get-lore-mbox.py -o/tmp -a -v6 20200128184934.77625-1-samitolvanen@google.com
  Looking up https://lore.kernel.org/r/20200128184934.77625-1-samitolvanen@google.com
  Grabbing thread from https://lore.kernel.org/kernel-hardening/20200128184934.77625-1-samitolvanen@google.com/t.mbox.gz
  Saved thread into /tmp/20200128184934.77625-1-samitolvanen@google.com.t.mbx
  Analyzing 280 messages in the thread
	Ignoring v1: [PATCH 00/18] add support for Clang's Shadow Call Stack
	Ignoring v1: [PATCH 01/18] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
    ...
	Ignoring v5: Re: [PATCH v5 14/14] arm64: implement Shadow Call Stack
  Found new series version: v6
	Processing: [PATCH v6 00/15] add support for Clang's Shadow Call Stack
	Processing: [PATCH v6 01/15] arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
    ...
	Processing: Re: [PATCH v6 14/15] arm64: implement Shadow Call Stack
  Found new series version: v7
	Ignoring v7: [PATCH v7 00/11] add support for Clang's Shadow Call Stack
	Ignoring v7: [PATCH v7 01/11] add support for Clang's Shadow Call Stack (SCS)
    ...
	Ignoring v7: Re: [PATCH v7 04/11] scs: disable when function graph tracing is enabled
  ---
  Writing /tmp/v6_add_support_for_clang_s_shadow_call_stack.mbx
	[PATCH v6 00/15] add support for Clang's Shadow Call Stack
	[PATCH v6 01/15] arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
	[PATCH v6 02/15] arm64/lib: copy_page: avoid x18 register in assembler code
	[PATCH v6 03/15] arm64: kvm: stop treating register x18 as caller save
	[PATCH v6 04/15] arm64: kernel: avoid x18 in __cpu_soft_restart
	[PATCH v6 05/15] add support for Clang's Shadow Call Stack (SCS)
	[PATCH v6 06/15] scs: add accounting
	[PATCH v6 07/15] scs: add support for stack usage debugging
	[PATCH v6 08/15] arm64: disable function graph tracing with SCS
	[PATCH v6 09/15] arm64: reserve x18 from general allocation with SCS
	  Adding trailer: Acked-by: Will Deacon <will@kernel.org>
	[PATCH v6 10/15] arm64: preserve x18 when CPU is suspended
	  Adding trailer: Acked-by: Will Deacon <will@kernel.org>
	[PATCH v6 11/15] arm64: efi: restore x18 if it was corrupted
	[PATCH v6 12/15] arm64: vdso: disable Shadow Call Stack
	[PATCH v6 13/15] arm64: disable SCS for hypervisor code
	[PATCH v6 14/15] arm64: implement Shadow Call Stack
	[PATCH v6 15/15] arm64: scs: add shadow stacks for SDEI
  ---
  Link: https://lore.kernel.org/r/20191206221351.38241-1-samitolvanen@google.com
  Base-commit included, you can branch using:
	git checkout -b v6_add_support_for_clang_s_shadow_call_stack 3cf2890f29ab6fe491361761df558ef9191cb468
	git am /tmp/v6_add_support_for_clang_s_shadow_call_stack.mbx

You can see that the git command it gives you has a different branch and 
parent info, which should let you do an easy diff between two branches, 
one using v6, and the other using v7 versions.
 
Please give it a try and let me know your feedback! You can find it here:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/get-lore-mbox.py

It's a bit raw around the edges and I keep finding corner-cases, so it 
would be super interesting for me to get input from people actually 
trying to use it for managing patch series.

Best,
-K

             reply index

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-01  3:01 Konstantin Ryabitsev [this message]
2020-02-01 17:43 ` Kees Cook
2020-02-03 22:38 ` Dan Williams
2020-02-14 19:30 ` Kevin Hilman
2020-02-14 19:40   ` Toke Høiland-Jørgensen
2020-02-14 19:53     ` Mark Brown
2020-02-14 19:53   ` Konstantin Ryabitsev
2020-02-14 20:35     ` Kevin Hilman
2020-02-17 10:52       ` Geert Uytterhoeven
2020-02-17 14:30         ` Konstantin Ryabitsev
2020-02-15 22:23 ` mbox-export-patch, notmuch-export-patch Sean Whitton

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200201030105.k6akvbjpmlpcuiky@chatter.i7.local \
    --to=konstantin@linuxfoundation.org \
    --cc=workflows@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Workflows Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/workflows/0 workflows/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 workflows workflows/ https://lore.kernel.org/workflows \
		workflows@vger.kernel.org
	public-inbox-index workflows

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.workflows


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git