[PATCH] Remove URL redirect project lookup

[PATCH] Remove URL redirect project lookup

[Rob Herring]

Now that lore indexes all messages, there's no need to lookup the project
for the message-id. If the project is not specified, then 'all' is used.

The primary benefit of this change is that cached accesses can now work
offline instead of splatting with a network error.

Signed-off-by: Rob Herring <robh@kernel.org>
---

My usecase is twofold. First I want to speed up opening a thread by 
having it fetched in the background and cached. Second, I want to be 
able to work offline by fetching a list of threads (my PW queue) in 
advance and using the offline copy. With a sufficiently long cache 
timeout, the cache works perfectly for this use. Though maybe a 'use the 
cache if there's a network failure' mode is needed instead of always 
timing out the cache.

I also have this working using b4 to fetch my queue to an mbox and 
then using the 'use local mbox' option. This mostly works except for 
the handling of 'From ' in message bodies which is problematic for mbox 
format. The cache manages to avoid this problem.

Rob

 b4/__init__.py | 23 +++++++----------------
 1 file changed, 7 insertions(+), 16 deletions(-)

diff --git a/b4/__init__.py b/b4/__init__.py
index 0d506bbaa649..ec1a6da44144 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -2235,6 +2235,9 @@ def get_pi_thread_by_url(t_mbx_url, nocache=False):
         logger.critical('Grabbing thread from %s', t_mbx_url.split('://')[1])
         session = get_requests_session()
         resp = session.get(t_mbx_url)
+        if resp.status_code == 404:
+            logger.critical('That message-id is not known.')
+            return None
         if resp.status_code != 200:
             logger.critical('Server returned an error: %s', resp.status_code)
             return None
@@ -2263,22 +2266,10 @@ def get_pi_thread_by_url(t_mbx_url, nocache=False):
 def get_pi_thread_by_msgid(msgid, useproject=None, nocache=False, onlymsgids: Optional[set] = None):
     qmsgid = urllib.parse.quote_plus(msgid)
     config = get_main_config()
-    # Grab the head from lore, to see where we are redirected
-    midmask = config['midmask'] % qmsgid
-    loc = urllib.parse.urlparse(midmask)
-    if useproject:
-        projurl = '%s://%s/%s' % (loc.scheme, loc.netloc, useproject)
-    else:
-        logger.info('Looking up %s', midmask)
-        session = get_requests_session()
-        resp = session.head(midmask)
-        if resp.status_code < 300 or resp.status_code > 400:
-            logger.critical('That message-id is not known.')
-            return None
-        # Pop msgid from the end of the redirect
-        chunks = resp.headers['Location'].rstrip('/').split('/')
-        projurl = '/'.join(chunks[:-1])
-        resp.close()
+    loc = urllib.parse.urlparse(config['midmask'])
+    if not useproject:
+        useproject = 'all'
+    projurl = '%s://%s/%s' % (loc.scheme, loc.netloc, useproject)
     t_mbx_url = '%s/%s/t.mbox.gz' % (projurl, qmsgid)
     logger.debug('t_mbx_url=%s', t_mbx_url)
 
-- 
2.32.0

Re: [PATCH] Remove URL redirect project lookup

[Rob Herring]

On Thu, Feb 24, 2022 at 9:11 PM Rob Herring <robh@kernel.org> wrote:
>
> Now that lore indexes all messages, there's no need to lookup the project
> for the message-id. If the project is not specified, then 'all' is used.
>
> The primary benefit of this change is that cached accesses can now work
> offline instead of splatting with a network error.
>
> Signed-off-by: Rob Herring <robh@kernel.org>
> ---
>
> My usecase is twofold. First I want to speed up opening a thread by
> having it fetched in the background and cached. Second, I want to be
> able to work offline by fetching a list of threads (my PW queue) in
> advance and using the offline copy. With a sufficiently long cache
> timeout, the cache works perfectly for this use. Though maybe a 'use the
> cache if there's a network failure' mode is needed instead of always
> timing out the cache.
>
> I also have this working using b4 to fetch my queue to an mbox and
> then using the 'use local mbox' option. This mostly works except for
> the handling of 'From ' in message bodies which is problematic for mbox
> format. The cache manages to avoid this problem.
>
> Rob
>
>  b4/__init__.py | 23 +++++++----------------
>  1 file changed, 7 insertions(+), 16 deletions(-)

Ping!

>
> diff --git a/b4/__init__.py b/b4/__init__.py
> index 0d506bbaa649..ec1a6da44144 100644
> --- a/b4/__init__.py
> +++ b/b4/__init__.py
> @@ -2235,6 +2235,9 @@ def get_pi_thread_by_url(t_mbx_url, nocache=False):
>          logger.critical('Grabbing thread from %s', t_mbx_url.split('://')[1])
>          session = get_requests_session()
>          resp = session.get(t_mbx_url)
> +        if resp.status_code == 404:
> +            logger.critical('That message-id is not known.')
> +            return None
>          if resp.status_code != 200:
>              logger.critical('Server returned an error: %s', resp.status_code)
>              return None
> @@ -2263,22 +2266,10 @@ def get_pi_thread_by_url(t_mbx_url, nocache=False):
>  def get_pi_thread_by_msgid(msgid, useproject=None, nocache=False, onlymsgids: Optional[set] = None):
>      qmsgid = urllib.parse.quote_plus(msgid)
>      config = get_main_config()
> -    # Grab the head from lore, to see where we are redirected
> -    midmask = config['midmask'] % qmsgid
> -    loc = urllib.parse.urlparse(midmask)
> -    if useproject:
> -        projurl = '%s://%s/%s' % (loc.scheme, loc.netloc, useproject)
> -    else:
> -        logger.info('Looking up %s', midmask)
> -        session = get_requests_session()
> -        resp = session.head(midmask)
> -        if resp.status_code < 300 or resp.status_code > 400:
> -            logger.critical('That message-id is not known.')
> -            return None
> -        # Pop msgid from the end of the redirect
> -        chunks = resp.headers['Location'].rstrip('/').split('/')
> -        projurl = '/'.join(chunks[:-1])
> -        resp.close()
> +    loc = urllib.parse.urlparse(config['midmask'])
> +    if not useproject:
> +        useproject = 'all'
> +    projurl = '%s://%s/%s' % (loc.scheme, loc.netloc, useproject)
>      t_mbx_url = '%s/%s/t.mbox.gz' % (projurl, qmsgid)
>      logger.debug('t_mbx_url=%s', t_mbx_url)
>
> --
> 2.32.0
>

Re: [PATCH] Remove URL redirect project lookup

[Konstantin Ryabitsev]

On Mon, Mar 21, 2022 at 11:41:20AM -0500, Rob Herring wrote:
> > My usecase is twofold. First I want to speed up opening a thread by
> > having it fetched in the background and cached. Second, I want to be
> > able to work offline by fetching a list of threads (my PW queue) in
> > advance and using the offline copy. With a sufficiently long cache
> > timeout, the cache works perfectly for this use. Though maybe a 'use the
> > cache if there's a network failure' mode is needed instead of always
> > timing out the cache.
> >
> > I also have this working using b4 to fetch my queue to an mbox and
> > then using the 'use local mbox' option. This mostly works except for
> > the handling of 'From ' in message bodies which is problematic for mbox
> > format. The cache manages to avoid this problem.
> >
> > Rob
> >
> >  b4/__init__.py | 23 +++++++----------------
> >  1 file changed, 7 insertions(+), 16 deletions(-)
> 
> Ping!

Sorry, I've been a bit verklempt about things over the past few weeks. I'll
try to get to outstanding patches in very short order.

Best regards,
-K

Re: [PATCH] Remove URL redirect project lookup

[Konstantin Ryabitsev]

On Thu, Feb 24, 2022 at 09:11:35PM -0600, Rob Herring wrote:
> Now that lore indexes all messages, there's no need to lookup the project
> for the message-id. If the project is not specified, then 'all' is used.

Rob:

Sorry for the long delay, but I did finally get around to it. I didn't quite
use your patch directly, because the goal is to also support non-lore
public-inbox installations, and they may not provide the unified index in
/all/, so we needed to keep the old lookup option available.

The change is in master as commit bfe5df6694c8115fa8402943b125c6e47c8eec08.

Thanks,
-K