tip: New "Link:" tag to replace "LKML-Reference:"

tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

Hi all,

At Linus' request, I have modified the tip-bot to recognize a new form
of email archive references, which should work as clickable links.
Instead of:

LKML-Reference: <message-id>

use, instead:

Link: http://lkml.kernel.org/r/message-id

The entire bit "Link: http://lkml.kernel.org/r/" is a literal; the rest
is the message-id from the original message, *without* angle brackets.
This points to a redirection service on kernel.org, which currently
bounces to marc.info, but which we can redirect in the future.

I changed the name from LKML-Reference: to Link: to make the width of
the string tolerable, and since the link itself contains an explicit
reference to the specific archive used.

Please note that message-ids sometimes contain characters which have to
be escaped to fit in valid URI syntax!  The characters @ and $ which are
very common in message-ids do not, however, need to be escaped in this
particular context (even though they are not part of the URI safe
character set, and so most library functions will escape them.)  The
characters # ; & % as well as whitespace are definitely unsafe either
for use in an URI or for programs to recognize them as part of one, and
should definitely be escaped.

The tip-bot will recognize the *exact* form above (including URI
escaping) and should do the message threading correctly.  To keep people
from injecting malicious links, however, there is sanity filtering on
the unescaped content, so in theory a legal-but-weird message-id might
be rejected.

	-hpa

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Peter Zijlstra]

On Wed, 2011-04-06 at 15:09 -0700, H. Peter Anvin wrote:
> Hi all,
> 
> At Linus' request, I have modified the tip-bot to recognize a new form
> of email archive references, which should work as clickable links.
> Instead of:
> 
> LKML-Reference: <message-id>
> 
> use, instead:
> 
> Link: http://lkml.kernel.org/r/message-id
> 
> The entire bit "Link: http://lkml.kernel.org/r/" is a literal; the rest
> is the message-id from the original message, *without* angle brackets.
> This points to a redirection service on kernel.org, which currently
> bounces to marc.info, but which we can redirect in the future.
> 
> I changed the name from LKML-Reference: to Link: to make the width of
> the string tolerable, and since the link itself contains an explicit
> reference to the specific archive used.
> 
> Please note that message-ids sometimes contain characters which have to
> be escaped to fit in valid URI syntax!  The characters @ and $ which are
> very common in message-ids do not, however, need to be escaped in this
> particular context (even though they are not part of the URI safe
> character set, and so most library functions will escape them.)  The
> characters # ; & % as well as whitespace are definitely unsafe either
> for use in an URI or for programs to recognize them as part of one, and
> should definitely be escaped.
> 
> The tip-bot will recognize the *exact* form above (including URI
> escaping) and should do the message threading correctly.  To keep people
> from injecting malicious links, however, there is sanity filtering on
> the unescaped content, so in theory a legal-but-weird message-id might
> be rejected.

OK, so I fixed my scripts to match this and it all seems to work, except
for the:

 LKML-Reference: <new-submission>

case, where the tip-bot would mail it out for me. While I try not to use
it too much its convenient for quick fixes etc. Will the absence of a
Link tag suffice or is there more to it?

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/07/2011 07:08 AM, Peter Zijlstra wrote:
> 
> OK, so I fixed my scripts to match this and it all seems to work, except
> for the:
> 
>  LKML-Reference: <new-submission>
> 
> case, where the tip-bot would mail it out for me. While I try not to use
> it too much its convenient for quick fixes etc. Will the absence of a
> Link tag suffice or is there more to it?
> 

OK, I have asked for a long time that we do not use <new-submission>.

The recommended pattern has been:

LKML-Reference: <tip-*@git.kernel.org>

... which at least indicates to the user how the message-ID is to be
generated (* being a placeholder for the commit SHA1).

The only think I can think of for how to make Link: work would be to
recognize a pattern that the tip-bot would use as its own Message-ID
*instead* of the commit SHA1 pattern that it would normally use.  There
are disadvantages to every approach, of course.

We have a couple of alternatives:

a) Continue to use "LKML-Reference: <tip-*@git.kernel.org>"
   + At least gives a hint how to find the newly-formed LKML thread
   - Not a clickable link
   - Somewhat noisy
b) Just Cc: everything to LKML regardless of tag
   + Easy
   - No way to *not* post everything to LKML
   - No way to locate the thread without knowing the magic
c) A recognizable pattern with a unique pattern *generated by the
   committer*, something like:
   Link: http://lkml.kernel.org/r/tip-`uuidgen -r`@git.kernel.org
   + Clickable link
   - Long (the above is 86 characters wide)
   - Message-IDs of a nonstandard form
   - Bad things will happen if someone re-uses an identifier
   - String has to be generated by machine at commit time

[Using ranpwd instead of uuidgen allows for a more compact random string
by using a wider character set.  For example:

Link: http://lkml.kernel.org/r/tip-`ranpwd -l 24`@git.kernel.org

... has the same entropy but is 74 characters wide.]

Thoughts?

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Peter Zijlstra]

On Thu, 2011-04-07 at 07:43 -0700, H. Peter Anvin wrote:
> On 04/07/2011 07:08 AM, Peter Zijlstra wrote:
> > 
> > OK, so I fixed my scripts to match this and it all seems to work, except
> > for the:
> > 
> >  LKML-Reference: <new-submission>
> > 
> > case, where the tip-bot would mail it out for me. While I try not to use
> > it too much its convenient for quick fixes etc. Will the absence of a
> > Link tag suffice or is there more to it?
> > 
> 
> OK, I have asked for a long time that we do not use <new-submission>.
> 
> The recommended pattern has been:
> 
> LKML-Reference: <tip-*@git.kernel.org>
> 
> ... which at least indicates to the user how the message-ID is to be
> generated (* being a placeholder for the commit SHA1).

Oh, I never heard of that, it would've been easy to make my scripts do
that if that were the preferred form, a well.

> The only think I can think of for how to make Link: work would be to
> recognize a pattern that the tip-bot would use as its own Message-ID
> *instead* of the commit SHA1 pattern that it would normally use.  There
> are disadvantages to every approach, of course.
> 
> We have a couple of alternatives:
> 
> a) Continue to use "LKML-Reference: <tip-*@git.kernel.org>"
>    + At least gives a hint how to find the newly-formed LKML thread
>    - Not a clickable link
>    - Somewhat noisy
> b) Just Cc: everything to LKML regardless of tag
>    + Easy
>    - No way to *not* post everything to LKML
>    - No way to locate the thread without knowing the magic
> c) A recognizable pattern with a unique pattern *generated by the
>    committer*, something like:
>    Link: http://lkml.kernel.org/r/tip-`uuidgen -r`@git.kernel.org
>    + Clickable link
>    - Long (the above is 86 characters wide)
>    - Message-IDs of a nonstandard form
>    - Bad things will happen if someone re-uses an identifier
>    - String has to be generated by machine at commit time
> 
> [Using ranpwd instead of uuidgen allows for a more compact random string
> by using a wider character set.  For example:
> 
> Link: http://lkml.kernel.org/r/tip-`ranpwd -l 24`@git.kernel.org
> 
> ... has the same entropy but is 74 characters wide.]

Can't we use a prepare-commit-msg or commit-msg hook to re-write the
commit message to include a Link tag when one is missing, using
something like:

Link: http://lkml.kernel.org/r/tip-$SHA@git.kernel.org

And have the post-commit hook that sends the messages recognize this
form and send the message using the correct msgid.

That way the commit includes a correct and clickable link, doesn't
require magic knowledge and doesn't need the committer to do extra work.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/07/2011 07:54 AM, Peter Zijlstra wrote:
> 
> Can't we use a prepare-commit-msg or commit-msg hook to re-write the
> commit message to include a Link tag when one is missing, using
> something like:
> 
> Link: http://lkml.kernel.org/r/tip-$SHA@git.kernel.org
> 
> And have the post-commit hook that sends the messages recognize this
> form and send the message using the correct msgid.
> 
> That way the commit includes a correct and clickable link, doesn't
> require magic knowledge and doesn't need the committer to do extra work.
> 

Think about it... hint: the SHA1 is a cryptographically secure signature
of the commit object.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Peter Zijlstra]

On Thu, 2011-04-07 at 07:56 -0700, H. Peter Anvin wrote:
> On 04/07/2011 07:54 AM, Peter Zijlstra wrote:
> > 
> > Can't we use a prepare-commit-msg or commit-msg hook to re-write the
> > commit message to include a Link tag when one is missing, using
> > something like:
> > 
> > Link: http://lkml.kernel.org/r/tip-$SHA@git.kernel.org
> > 
> > And have the post-commit hook that sends the messages recognize this
> > form and send the message using the correct msgid.
> > 
> > That way the commit includes a correct and clickable link, doesn't
> > require magic knowledge and doesn't need the committer to do extra work.
> > 
> 
> Think about it... hint: the SHA1 is a cryptographically secure signature
> of the commit object.

D'0h ok.. but the scheme would work with any generated msgid, all we
need is to generate one for commits that lack a Link tag and insert it
using the prepare-commit-msg/commit-msg hooks and for the post-commit
hook to recognize it and use that actual msgid.

The only thing to avoid is msgid collision, but given that MUAs already
have this problem all we need to do is borrow some of their logic.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/07/2011 07:54 AM, Peter Zijlstra wrote:
> 
> Can't we use a prepare-commit-msg or commit-msg hook to re-write the
> commit message to include a Link tag when one is missing, using
> something like:
> 
> Link: http://lkml.kernel.org/r/tip-$SHA@git.kernel.org
> 
> And have the post-commit hook that sends the messages recognize this
> form and send the message using the correct msgid.
> 
> That way the commit includes a correct and clickable link, doesn't
> require magic knowledge and doesn't need the committer to do extra work.
> 

To summarize the problem I hinted at earlier:

1. The link that ends up in the commit message needs to be unique, or
   it is not going to be a working future-clickable link.
2. The link that ends up in the commit message CANNOT use the SHA1 of
   the commit itself, since the commit SHA1 is derived from the entire
   commit object including the message;
3. As a result, the only option is for the robot to change the
   Message-ID to something other than the SHA1 of the commit.  It is a
   bit unfortunate, but probably not disastrous.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/07/2011 08:03 AM, Peter Zijlstra wrote:
> 
> D'0h ok.. but the scheme would work with any generated msgid, all we
> need is to generate one for commits that lack a Link tag and insert it
> using the prepare-commit-msg/commit-msg hooks and for the post-commit
> hook to recognize it and use that actual msgid.
> 
> The only thing to avoid is msgid collision, but given that MUAs already
> have this problem all we need to do is borrow some of their logic.
> 

OK, this doesn't really need any hooks at all (the tip-bot is not a
hook).  The committer is, and *has to be* responsible for getting the
unique string into the Link: tag, through whatever means they feel is
appropriate.

Based on the previous message, I'm going to modify the bot to recognize
the pattern:

Link: http://lkml.kernel.org/r/tip-`ranpwd -l 24`@git.kernel.org

... to override the Message-ID.  It seems like the least ugly of a lot
of ugly options.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Peter Zijlstra]

On Thu, 2011-04-07 at 08:20 -0700, H. Peter Anvin wrote:
> 
> OK, this doesn't really need any hooks at all (the tip-bot is not a
> hook).  The committer is, and *has to be* responsible for getting the
> unique string into the Link: tag, through whatever means they feel is
> appropriate.
> 
> Based on the previous message, I'm going to modify the bot to
> recognize
> the pattern:
> 
> Link: http://lkml.kernel.org/r/tip-`ranpwd -l 24`@git.kernel.org
> 
> ... to override the Message-ID.  It seems like the least ugly of a lot
> of ugly options. 

OK, but to qualify, it needn't be the committer, that is, Ingo usually
commits the stuff for me, but I usually provide him with complete
patches including LKML-Reference, now Link, tag.

I'll change my script to:

Use an existing Link tag provided it doesn't match the above pattern, if
there's no such tag, use Message-id, lacking that, generate one using
the above pattern.

Thanks!

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Sam Ravnborg]

On Wed, Apr 06, 2011 at 03:09:50PM -0700, H. Peter Anvin wrote:
> Hi all,
> 
> At Linus' request, I have modified the tip-bot to recognize a new form
> of email archive references, which should work as clickable links.
> Instead of:
> 
> LKML-Reference: <message-id>
> 
> use, instead:
> 
> Link: http://lkml.kernel.org/r/message-id

If use of this grows outside plain linux-kernel development a
more generic name could make sense.

Suggestion:

    http://mid.kernel.org/r/message-id

mid for message-id.

	Sam

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/07/2011 08:28 AM, Sam Ravnborg wrote:
> 
> If use of this grows outside plain linux-kernel development a
> more generic name could make sense.
> 
> Suggestion:
> 
>     http://mid.kernel.org/r/message-id
> 
> mid for message-id.
> 

I explicitly did not want to do that, since I believe it is important
information "which mailing list does this come from".  Otherwise we may
need a database on kernel.org knowing exactly which mailing list any
particular message-id belongs to.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[Bjorn Helgaas]

On Wed, Apr 6, 2011 at 4:09 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> Hi all,
>
> At Linus' request, I have modified the tip-bot to recognize a new form
> of email archive references, which should work as clickable links.
> Instead of:
>
> LKML-Reference: <message-id>
>
> use, instead:
>
> Link: http://lkml.kernel.org/r/message-id

Really cool idea.  I assume you'll mention this in
Documentation/SubmittingPatches or somewhere similar.

Bjorn

Re: tip: New "Link:" tag to replace "LKML-Reference:"

[H. Peter Anvin]

On 04/08/2011 07:36 AM, Bjorn Helgaas wrote:
> On Wed, Apr 6, 2011 at 4:09 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>> Hi all,
>>
>> At Linus' request, I have modified the tip-bot to recognize a new form
>> of email archive references, which should work as clickable links.
>> Instead of:
>>
>> LKML-Reference: <message-id>
>>
>> use, instead:
>>
>> Link: http://lkml.kernel.org/r/message-id
> 
> Really cool idea.  I assume you'll mention this in
> Documentation/SubmittingPatches or somewhere similar.
> 

Well, it is better done by the committers rather than the patch
submitters, so it's mostly an issue for people using git as opposed to
submitting patches by email.

	-hpa

Link: tags for new submission -- UPDATE

[H. Peter Anvin]

On 04/07/2011 08:20 AM, H. Peter Anvin wrote:
> On 04/07/2011 08:03 AM, Peter Zijlstra wrote:
>>
>> D'0h ok.. but the scheme would work with any generated msgid, all we
>> need is to generate one for commits that lack a Link tag and insert it
>> using the prepare-commit-msg/commit-msg hooks and for the post-commit
>> hook to recognize it and use that actual msgid.
>>
>> The only thing to avoid is msgid collision, but given that MUAs already
>> have this problem all we need to do is borrow some of their logic.
>>
> 
> OK, this doesn't really need any hooks at all (the tip-bot is not a
> hook).  The committer is, and *has to be* responsible for getting the
> unique string into the Link: tag, through whatever means they feel is
> appropriate.
> 
> Based on the previous message, I'm going to modify the bot to recognize
> the pattern:
> 
> Link: http://lkml.kernel.org/r/tip-`ranpwd -l 24`@git.kernel.org
> 
> ... to override the Message-ID.  It seems like the least ugly of a lot
> of ugly options.
> 

<<<< IMPORTANT CHANGE >>>>

Please use the format:

Link: http://lkml.kernel.org/n/tip-`ranpwd -l 24`@git.kernel.org
                             ^

... with an "n" for "new" instead of "r" for "reference" instead.

This allows the bot to distinguish a "new submission" link from a
*reference* to a message that was created as a result of a new submission!

	-hpa