Linux maintainer tooling and workflows
 help / color / Atom feed
* b4: Ensure we read threadfile for message-id
@ 2021-04-21 20:29 Morten Linderud
  2021-04-21 23:33 ` Kyle Meyer
  0 siblings, 1 reply; 3+ messages in thread
From: Morten Linderud @ 2021-04-21 20:29 UTC (permalink / raw)
  To: tools; +Cc: Morten Linderud

This fixes a bug where reading from `get_msgid_from_stdin` couldn't grab
the message-id when we collect a thread from stdin.

This is mainly because there is no good way to override `sys.stdin`
(from what I can see) and it probably makes more sense to try fetch
message-ids from files instead. This allows us to replace the default
file "sys.stdin" with the thread file whenever we need.

No patch:

    $ curl -s "https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/raw" | b4 mbox
    Looking up https://lore.kernel.org/r/20210421130105.1226686-1-gregkh%40linuxfoundation.org
    Grabbing thread from lore.kernel.org/lkml
    272 messages in the thread
    Unable to find a valid message-id in stdin.

With patch:

    $ curl -s "https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/raw" | .4 mbox
    Looking up https://lore.kernel.org/r/20210421130105.1226686-1-gregkh%40linuxfoundation.org
    Grabbing thread from lore.kernel.org/lkml
    272 messages in the thread
    Saved ./20210421130105.1226686-1-gregkh@linuxfoundation.org.mbx

Signed-off-by: Morten Linderud <foxboron@archlinux.org>
---
 b4/__init__.py | 8 ++++----
 b4/mbox.py     | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/b4/__init__.py b/b4/__init__.py
index 32b5c02..e81d395 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -2179,18 +2179,18 @@ def get_requests_session():
     return REQSESSION
 
 
-def get_msgid_from_stdin():
+def get_msgid_from_file(file):
     if not sys.stdin.isatty():
-        message = email.message_from_string(sys.stdin.read())
+        message = email.message_from_string(file.read())
         return message.get('Message-ID', None)
     logger.error('Error: pipe a message or pass msgid as parameter')
     sys.exit(1)
 
 
-def get_msgid(cmdargs):
+def get_msgid(cmdargs, file=sys.stdin):
     if not cmdargs.msgid:
         logger.debug('Getting Message-ID from stdin')
-        msgid = get_msgid_from_stdin()
+        msgid = get_msgid_from_file(file)
         if msgid is None:
             logger.error('Unable to find a valid message-id in stdin.')
             sys.exit(1)
diff --git a/b4/mbox.py b/b4/mbox.py
index d3bde25..791f545 100644
--- a/b4/mbox.py
+++ b/b4/mbox.py
@@ -566,7 +566,7 @@ def main(cmdargs):
     if cmdargs.wantname:
         savefile = os.path.join(cmdargs.outdir, cmdargs.wantname)
     else:
-        msgid = b4.get_msgid(cmdargs)
+        msgid = b4.get_msgid(cmdargs, file=open(threadfile))
         savefile = os.path.join(cmdargs.outdir, '%s.mbx' % msgid)
 
     mbx.close()
-- 
2.31.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: b4: Ensure we read threadfile for message-id
  2021-04-21 20:29 b4: Ensure we read threadfile for message-id Morten Linderud
@ 2021-04-21 23:33 ` Kyle Meyer
  2021-05-14 20:55   ` Konstantin Ryabitsev
  0 siblings, 1 reply; 3+ messages in thread
From: Kyle Meyer @ 2021-04-21 23:33 UTC (permalink / raw)
  To: Morten Linderud; +Cc: tools

Morten Linderud writes:

> This fixes a bug where reading from `get_msgid_from_stdin` couldn't grab
> the message-id when we collect a thread from stdin.
>
> This is mainly because there is no good way to override `sys.stdin`
> (from what I can see) and it probably makes more sense to try fetch
> message-ids from files instead. This allows us to replace the default
> file "sys.stdin" with the thread file whenever we need.
>
> No patch:
>
>     $ curl -s "https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/raw" | b4 mbox
>     Looking up https://lore.kernel.org/r/20210421130105.1226686-1-gregkh%40linuxfoundation.org

So the message ID is successfully read from stdin here...

>     Grabbing thread from lore.kernel.org/lkml
>     272 messages in the thread
>     Unable to find a valid message-id in stdin.

... but then a subsequent call tries to read it from stdin again.

> With patch:
>
>     $ curl -s "https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/raw" | .4 mbox

typo: ".4"

>     Looking up https://lore.kernel.org/r/20210421130105.1226686-1-gregkh%40linuxfoundation.org
>     Grabbing thread from lore.kernel.org/lkml
>     272 messages in the thread
>     Saved ./20210421130105.1226686-1-gregkh@linuxfoundation.org.mbx
>
> Signed-off-by: Morten Linderud <foxboron@archlinux.org>

A similar error can still be triggered in the 'am --cherry-pick' code
path:

  curl -s "https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/raw" | b4 am -P _
  Looking up https://lore.kernel.org/r/20210421130105.1226686-1-gregkh%40linuxfoundation.org
  Grabbing thread from lore.kernel.org/lkml
  Analyzing 276 messages in the thread
  ---
  Unable to find a valid message-id in stdin.

I haven't tried, but it looks like you could update the get_msgid() call
in mbox_to_am() to use mboxfile.

Another approach would be to avoid collecting the msgid more than once
(something like below).


diff --git a/b4/mbox.py b/b4/mbox.py
index d3bde25..3783a56 100644
--- a/b4/mbox.py
+++ b/b4/mbox.py
@@ -27,7 +27,7 @@
 logger = b4.logger
 
 
-def mbox_to_am(mboxfile, cmdargs):
+def mbox_to_am(mboxfile, cmdargs, msgid):
     config = b4.get_main_config()
     outdir = cmdargs.outdir
     if outdir == '-':
@@ -81,7 +81,6 @@ def mbox_to_am(mboxfile, cmdargs):
     if cmdargs.cherrypick:
         cherrypick = list()
         if cmdargs.cherrypick == '_':
-            msgid = b4.get_msgid(cmdargs)
             # Only grab the exact msgid provided
             at = 0
             for lmsg in lser.patches[1:]:
@@ -500,16 +499,14 @@ def main(cmdargs):
 
     savefile = mkstemp('b4-mbox')[1]
 
+    msgid = b4.get_msgid(cmdargs)
     if not cmdargs.localmbox:
-        msgid = b4.get_msgid(cmdargs)
-
         threadfile = b4.get_pi_thread_by_msgid(msgid, savefile, useproject=cmdargs.useproject, nocache=cmdargs.nocache)
         if threadfile is None:
             os.unlink(savefile)
             return
     else:
         if os.path.exists(cmdargs.localmbox):
-            msgid = b4.get_msgid(cmdargs)
             if os.path.isdir(cmdargs.localmbox):
                 in_mbx = mailbox.Maildir(cmdargs.localmbox)
             else:
@@ -530,7 +527,7 @@ def main(cmdargs):
         get_extra_series(threadfile, direction=1)
 
     if cmdargs.subcmd == 'am':
-        mbox_to_am(threadfile, cmdargs)
+        mbox_to_am(threadfile, cmdargs, msgid)
         os.unlink(threadfile)
         return
 
@@ -566,7 +563,6 @@ def main(cmdargs):
     if cmdargs.wantname:
         savefile = os.path.join(cmdargs.outdir, cmdargs.wantname)
     else:
-        msgid = b4.get_msgid(cmdargs)
         savefile = os.path.join(cmdargs.outdir, '%s.mbx' % msgid)
 
     mbx.close()

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: b4: Ensure we read threadfile for message-id
  2021-04-21 23:33 ` Kyle Meyer
@ 2021-05-14 20:55   ` Konstantin Ryabitsev
  0 siblings, 0 replies; 3+ messages in thread
From: Konstantin Ryabitsev @ 2021-05-14 20:55 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: Morten Linderud, tools

On Wed, Apr 21, 2021 at 07:33:55PM -0400, Kyle Meyer wrote:
> I haven't tried, but it looks like you could update the get_msgid() call
> in mbox_to_am() to use mboxfile.
> 
> Another approach would be to avoid collecting the msgid more than once
> (something like below).

This is indeed a better approach, so I used it in the latest dev commit.
Thanks to both of you.

-K

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-21 20:29 b4: Ensure we read threadfile for message-id Morten Linderud
2021-04-21 23:33 ` Kyle Meyer
2021-05-14 20:55   ` Konstantin Ryabitsev

Linux maintainer tooling and workflows

Archives are clonable:
	git clone --mirror https://lore.kernel.org/tools/0 tools/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tools tools/ https://lore.kernel.org/tools \
		tools@linux.kernel.org
	public-inbox-index tools

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.linux.tools


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git