* [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend?
@ 2022-11-12 13:11 Oliver Smith
2022-11-12 13:11 ` [PATCH b4 1/1] RFC: Support SourceHut as additional backend Oliver Smith
2022-11-14 14:49 ` [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Konstantin Ryabitsev
0 siblings, 2 replies; 3+ messages in thread
From: Oliver Smith @ 2022-11-12 13:11 UTC (permalink / raw)
To: tools; +Cc: Oliver Smith
Hi,
I found that with relatively few changes it's possible to add SourceHut
as alternative backend to b4. Is this desirable to have in b4?
Best,
Oliver
Oliver Smith (1):
RFC: Support SourceHut as additional backend
b4/__init__.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 75 insertions(+), 4 deletions(-)
--
2.34.3
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH b4 1/1] RFC: Support SourceHut as additional backend
2022-11-12 13:11 [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Oliver Smith
@ 2022-11-12 13:11 ` Oliver Smith
2022-11-14 14:49 ` [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Konstantin Ryabitsev
1 sibling, 0 replies; 3+ messages in thread
From: Oliver Smith @ 2022-11-12 13:11 UTC (permalink / raw)
To: tools; +Cc: Oliver Smith
Add a SourceHut backend to b4, so it can be used with lists.sr.ht and
other instances. We've been using this successfully to merge a couple of
patches from https://lists.sr.ht/~postmarketos/pmbootstrap-devel/.
How to use:
1. Add a .b4-config to your git repo like this, adjust the paths:
[b4]
midmask = https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%s
linkmask = https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%%3C%s%%3E
send-series-to = ~postmarketos/pmbootstrap-devel@lists.sr.ht
send-endpoint-web = NONE
backend = sourcehut
2. Run 'b4 am' with either the short review ID integer, or the full
message ID as used in the archives:
$ b4 am 36506
$ b4 am 20221031111614.1377-1-ollieparanoid@postmarketos.org
Signed-off-by: Oliver Smith <ollieparanoid@postmarketos.org>
---
b4/__init__.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 75 insertions(+), 4 deletions(-)
diff --git a/b4/__init__.py b/b4/__init__.py
index 39cfeaf..787f628 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -17,6 +17,7 @@ import pathlib
import argparse
import smtplib
import shlex
+import re
import urllib.parse
import datetime
@@ -128,6 +129,8 @@ DEFAULT_CONFIG = {
'gpgbin': None,
# When sending mail, use this sendemail identity configuration
'sendemail-identity': None,
+ # What backend the mailing list runs, either "lore" or "sourcehut"
+ 'backend': 'lore',
}
# This is where we store actual config
@@ -2288,7 +2291,7 @@ def get_main_config() -> dict:
# some options can be provided via the toplevel .b4-config file,
# so load them up and use as defaults
topdir = git_get_toplevel()
- wtglobs = ['send-*', '*mask', '*template*', 'trailer*', 'pw-*']
+ wtglobs = ['send-*', '*mask', '*template*', 'trailer*', 'pw-*', 'backend']
if topdir:
wtcfg = os.path.join(topdir, '.b4-config')
if os.access(wtcfg, os.R_OK):
@@ -2625,6 +2628,15 @@ def split_and_dedupe_pi_results(t_mbox: bytes, cachedir: Optional[str] = None) -
return msgs
+def get_pi_thread_mbox_content(resp: requests.Session) -> bytes:
+ backend = get_main_config()['backend']
+ if backend == "lore":
+ return gzip.decompress(resp.content)
+ elif backend == "sourcehut":
+ return resp.content
+ raise KeyError
+
+
def get_pi_thread_by_url(t_mbx_url: str, nocache: bool = False):
msgs = list()
cachedir = get_cache_file(t_mbx_url, 'pi.msgs')
@@ -2644,7 +2656,7 @@ def get_pi_thread_by_url(t_mbx_url: str, nocache: bool = False):
if resp.status_code != 200:
logger.critical('Server returned an error: %s', resp.status_code)
return None
- t_mbox = gzip.decompress(resp.content)
+ t_mbox = get_pi_thread_mbox_content(resp)
resp.close()
if not len(t_mbox):
logger.critical('No messages found for that query')
@@ -2653,8 +2665,8 @@ def get_pi_thread_by_url(t_mbx_url: str, nocache: bool = False):
return split_and_dedupe_pi_results(t_mbox, cachedir=cachedir)
-def get_pi_thread_by_msgid(msgid: str, nocache: bool = False,
- onlymsgids: Optional[set] = None) -> Optional[list]:
+def get_pi_thread_by_msgid_lore(msgid: str, nocache: bool = False,
+ onlymsgids: Optional[set] = None) -> Optional[list]:
qmsgid = urllib.parse.quote_plus(msgid)
config = get_main_config()
loc = urllib.parse.urlparse(config['midmask'])
@@ -2699,6 +2711,65 @@ def get_pi_thread_by_msgid(msgid: str, nocache: bool = False,
return strict
+def get_pi_thread_by_msgid_sourcehut(msgid: str, nocache: bool = False,
+ onlymsgids: Optional[set] = None) -> Optional[list]:
+ review_id = urllib.parse.quote_plus(msgid)
+ config = get_main_config()
+ loc = urllib.parse.urlparse(config['midmask'])
+
+ if msgid.isdigit():
+ # This is the review ID, but we need the archives ID.
+ # TODO: use a proper api from sourcehut for this
+ # TODO: cache this, it won't change
+ review_url = config['midmask'] % f'patches/{msgid}'
+ logger.info('Looking up %s', review_url)
+ session = get_requests_session()
+ resp = session.get(review_url)
+ re_archives = re.compile('<a .*href="(.*)">View this thread in the archives')
+ match = re_archives.search(resp.text)
+ if not match:
+ raise ValueError('Failed to find the archives url')
+ archives_url = '%s://%s%s' % (loc.scheme, loc.netloc, match.group(1))
+
+ archives_id = urllib.parse.unquote(archives_url.split("/")[-1])
+ assert archives_id.startswith("<") and archives_id.endswith(">")
+ archives_id = archives_id[1:-1]
+ logger.info('Resolved to %s', archives_id)
+ else:
+ archives_url = config['midmask'] % f'<{msgid}>'
+ archives_id = msgid
+
+ t_mbx_url = '%s/mbox' % archives_url
+ logger.debug('t_mbx_url=%s', t_mbx_url)
+
+ msgs = get_pi_thread_by_url(t_mbx_url, nocache=nocache)
+ if not msgs:
+ return None
+
+ if onlymsgids:
+ strict = list()
+ for msg in msgs:
+ if LoreMessage.get_clean_msgid(msg) in onlymsgids:
+ strict.append(msg)
+ # also grab any messages where this msgid is in the references header
+ for onlymsgid in onlymsgids:
+ if msg.get('references', '').find(onlymsgid) >= 0:
+ strict.append(msg)
+ else:
+ strict = get_strict_thread(msgs, archives_id)
+
+ return strict
+
+
+def get_pi_thread_by_msgid(*args, **kwargs) -> Optional[list]:
+ backend = get_main_config()['backend']
+ if backend == "lore":
+ return get_pi_thread_by_msgid_lore(*args, **kwargs)
+ elif backend == "sourcehut":
+ return get_pi_thread_by_msgid_sourcehut(*args, **kwargs)
+ raise KeyError
+
+
def git_range_to_patches(gitdir: Optional[str], start: str, end: str,
covermsg: Optional[email.message.EmailMessage] = None,
prefixes: Optional[List[str]] = None,
--
2.34.3
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend?
2022-11-12 13:11 [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Oliver Smith
2022-11-12 13:11 ` [PATCH b4 1/1] RFC: Support SourceHut as additional backend Oliver Smith
@ 2022-11-14 14:49 ` Konstantin Ryabitsev
1 sibling, 0 replies; 3+ messages in thread
From: Konstantin Ryabitsev @ 2022-11-14 14:49 UTC (permalink / raw)
To: Oliver Smith; +Cc: tools
On Sat, Nov 12, 2022 at 02:11:13PM +0100, Oliver Smith wrote:
> Hi,
>
> I found that with relatively few changes it's possible to add SourceHut
> as alternative backend to b4. Is this desirable to have in b4?
Absolutely, I am all for making b4 work with sourcehut. I haven't yet had a
chance to review your patch, but I hope to be able to work on it today or
tomorrow. Some features will require rethinking for sure (e.g. we assume we're
always talking to a public-inbox instance, so when it's not the case we'll
have to adjust features like querying the remote for updates based on
change-id).
Cheers,
-K
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-11-14 14:49 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-12 13:11 [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Oliver Smith
2022-11-12 13:11 ` [PATCH b4 1/1] RFC: Support SourceHut as additional backend Oliver Smith
2022-11-14 14:49 ` [PATCH b4 0/1] How about adding SourceHut lists as 2nd backend? Konstantin Ryabitsev
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).