Stable list vs versioning

Stable list vs versioning

[Thomas Hellstrom]

Hi, Stable!

As you might be aware of, some companies that maintain linux kernel
drivers have the habit of assigning each driver change a new version
number. Typically a new subpatch number. This is in the interest of
their Quality Assurance department knowing exactly what they test. "Is
the bugfix for problem X part of the kernel I'm testing".

Now this doesn't play well with subdrivers. And it doesn't play well
with stable kernels and other distro-maintained kernels.

The argument "look at the source" doesn't always work. Particularly not
for some distro-maintained kernels where the Quality Assurance
department may not have access to the source or the skill to obtain it.

So basically I was wondering whether there might be a reasonable way
that this problem could be resolved. I was thinking along the lines of a
tool where the mainline git commit-id of a patch applied to stable and
touching a module would be added to a table of patch-ids for the modules
that so requested.

Any comments or ideas greatly appreciated.

Thanks,

Thomas Hellström

Re: Stable list vs versioning

[Greg KH]

On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
> Hi, Stable!
> 
> As you might be aware of, some companies that maintain linux kernel
> drivers have the habit of assigning each driver change a new version
> number.

And, as you have found out, that's a horrible thing to do for Linux and
doesn't work at all :)

Just because it works for other slower-moving operating systems, I
wouldn't recommend doing it for Linux.

Is there any specific driver out there that we don't have merged into
the kernel tree that could be helped from doing this type of thing?

thanks,

greg k-h

Re: Stable list vs versioning

[Thomas Hellstrom]

Hi!

On 10/06/2016 08:52 PM, Greg KH wrote:
> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>> Hi, Stable!
>>
>> As you might be aware of, some companies that maintain linux kernel
>> drivers have the habit of assigning each driver change a new version
>> number.
> And, as you have found out, that's a horrible thing to do for Linux and
> doesn't work at all :)
>
> Just because it works for other slower-moving operating systems, I
> wouldn't recommend doing it for Linux.

Yes, I'm fully aware of the difficulties, though I was hoping that I,
with the help some bright ideas from the list could come up with a
clever way to make everybody happy.

>
> Is there any specific driver out there that we don't have merged into
> the kernel tree that could be helped from doing this type of thing?
>
> thanks,
>
> greg k-h

 Nope, no out-of-tree driver that I know of. This is all in the interest
of sending crucial bugfixes through the proper channels without making
life very hard for Quality Assurance.

Thanks,

Thomas

Re: Stable list vs versioning

[Greg KH]

On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
> Hi!
> 
> On 10/06/2016 08:52 PM, Greg KH wrote:
> > On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
> >> Hi, Stable!
> >>
> >> As you might be aware of, some companies that maintain linux kernel
> >> drivers have the habit of assigning each driver change a new version
> >> number.
> > And, as you have found out, that's a horrible thing to do for Linux and
> > doesn't work at all :)
> >
> > Just because it works for other slower-moving operating systems, I
> > wouldn't recommend doing it for Linux.
> 
> Yes, I'm fully aware of the difficulties, though I was hoping that I,
> with the help some bright ideas from the list could come up with a
> clever way to make everybody happy.

But who has the problem here really?  Not the kernel community or
developers, but rather an odd set of unskilled QA people (your word, not
mine.)

Why can't they get more "skill"?  :)

thanks,

greg k-h

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/06/2016 09:22 PM, Greg KH wrote:
> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
>> Hi!
>>
>> On 10/06/2016 08:52 PM, Greg KH wrote:
>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>>>> Hi, Stable!
>>>>
>>>> As you might be aware of, some companies that maintain linux kernel
>>>> drivers have the habit of assigning each driver change a new version
>>>> number.
>>> And, as you have found out, that's a horrible thing to do for Linux and
>>> doesn't work at all :)
>>>
>>> Just because it works for other slower-moving operating systems, I
>>> wouldn't recommend doing it for Linux.
>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
>> with the help some bright ideas from the list could come up with a
>> clever way to make everybody happy.
> But who has the problem here really?  Not the kernel community or
> developers, but rather an odd set of unskilled QA people (your word, not
> mine.)
>
> Why can't they get more "skill"?  :)
>
> thanks,
>
> greg k-h

Well, I would in no way call our QA people unskilled just because they
in general don't have the skill to know how to locate a particular,
sometimes well-hidden git repo and find out if a certain bug is fixed or
not. Not even Einstein knew how to do that ;)

But I won't try to argue here. I do think, though, that as long as
people believe the easier solution is to version each change they will
keep on doing that and unfortunately as a result important patches won't
get CC'd stable because that would mess up the versioning.

From your answer I take it there is no interest from the stable
maintainers in helping solving this using some kind of mainline hash
registering tool. I guess perhaps another option is to locally automate
stable / distro git tree scanning.

Thanks,

Thomas

Re: Stable list vs versioning

[Greg KH]

On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
> On 10/06/2016 09:22 PM, Greg KH wrote:
> > On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
> >> Hi!
> >>
> >> On 10/06/2016 08:52 PM, Greg KH wrote:
> >>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
> >>>> Hi, Stable!
> >>>>
> >>>> As you might be aware of, some companies that maintain linux kernel
> >>>> drivers have the habit of assigning each driver change a new version
> >>>> number.
> >>> And, as you have found out, that's a horrible thing to do for Linux and
> >>> doesn't work at all :)
> >>>
> >>> Just because it works for other slower-moving operating systems, I
> >>> wouldn't recommend doing it for Linux.
> >> Yes, I'm fully aware of the difficulties, though I was hoping that I,
> >> with the help some bright ideas from the list could come up with a
> >> clever way to make everybody happy.
> > But who has the problem here really?  Not the kernel community or
> > developers, but rather an odd set of unskilled QA people (your word, not
> > mine.)
> >
> > Why can't they get more "skill"?  :)
> >
> > thanks,
> >
> > greg k-h
> 
> Well, I would in no way call our QA people unskilled just because they
> in general don't have the skill to know how to locate a particular,
> sometimes well-hidden git repo and find out if a certain bug is fixed or
> not. Not even Einstein knew how to do that ;)

Huh?  All of the kernel trees we "release" are in one single repo, and
it is very well known (linked to off of the kernel.org site front page):
	https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git

How is that difficult to find?

> But I won't try to argue here. I do think, though, that as long as
> people believe the easier solution is to version each change they will
> keep on doing that and unfortunately as a result important patches won't
> get CC'd stable because that would mess up the versioning.
> 
> From your answer I take it there is no interest from the stable
> maintainers in helping solving this using some kind of mainline hash
> registering tool. I guess perhaps another option is to locally automate
> stable / distro git tree scanning.

Maybe I really don't understand the "issue" you are trying to address
here, can you try to rephrase it by showing a real example of what you
are trying to solve?

But again, there's nothing we can do about out-of-tree code, remember,
they know where we are (and I'll take anything!), but we don't know
where they are...

thanks,

greg k-h

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/07/2016 05:48 AM, Greg KH wrote:
> On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
>> On 10/06/2016 09:22 PM, Greg KH wrote:
>>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
>>>> Hi!
>>>>
>>>> On 10/06/2016 08:52 PM, Greg KH wrote:
>>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>>>>>> Hi, Stable!
>>>>>>
>>>>>> As you might be aware of, some companies that maintain linux kernel
>>>>>> drivers have the habit of assigning each driver change a new version
>>>>>> number.
>>>>> And, as you have found out, that's a horrible thing to do for Linux and
>>>>> doesn't work at all :)
>>>>>
>>>>> Just because it works for other slower-moving operating systems, I
>>>>> wouldn't recommend doing it for Linux.
>>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
>>>> with the help some bright ideas from the list could come up with a
>>>> clever way to make everybody happy.
>>> But who has the problem here really?  Not the kernel community or
>>> developers, but rather an odd set of unskilled QA people (your word, not
>>> mine.)
>>>
>>> Why can't they get more "skill"?  :)
>>>
>>> thanks,
>>>
>>> greg k-h
>> Well, I would in no way call our QA people unskilled just because they
>> in general don't have the skill to know how to locate a particular,
>> sometimes well-hidden git repo and find out if a certain bug is fixed or
>> not. Not even Einstein knew how to do that ;)
> Huh?  All of the kernel trees we "release" are in one single repo, and
> it is very well known (linked to off of the kernel.org site front page):
> 	https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e= 
>
> How is that difficult to find?

The "vanilla" stable ones are easy. The distro ones may not be, save
Ubuntu that sometimes "take over" a stable tree. Typically the kernels
we test are a distro-modified version of a stable tree.

>
>> But I won't try to argue here. I do think, though, that as long as
>> people believe the easier solution is to version each change they will
>> keep on doing that and unfortunately as a result important patches won't
>> get CC'd stable because that would mess up the versioning.
>>
>> From your answer I take it there is no interest from the stable
>> maintainers in helping solving this using some kind of mainline hash
>> registering tool. I guess perhaps another option is to locally automate
>> stable / distro git tree scanning.
> Maybe I really don't understand the "issue" you are trying to address
> here, can you try to rephrase it by showing a real example of what you
> are trying to solve?
>
> But again, there's nothing we can do about out-of-tree code, remember,
> they know where we are (and I'll take anything!), but we don't know
> where they are...
>
> thanks,
>
> greg k-h

Yes. The problem would be

Given a *binary* version of distro kernel X, based on stable kernel Y.
What _upstreamed_ bugfix patches has touched our module since the stable
branch was created? Let's assume the distro git tree is hard to find.

a) Now if stable maintainers and distro kernel maintainers could use a
flag "record commit id" to the git am command, the mainline commit id
would be added to a binary visible table in the module, problem solved.

b) If we assume that the distro git tree is accessible to us, we can
solve this locally by building such a table by matching the distro
kernel version to a commit in that git tree and then scan the git tree
for the known bugfix patches we've sent out. Problem also solved.

So the reason for sending out the email was basically to enquire whether
there were others facing the same problem and if anybody was interested
in a common solution along a). I mean even if we code a) up but nobody
wanted to use it because they disliked the whole idea, that wouldn't
make much sense.

And if nobody else is interested, we'd probably be better off with b)
provided we can gain access to the git trees of the important distro
kernels.

Thanks,

Thomas

Re: Stable list vs versioning

[Greg KH]

On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
> On 10/07/2016 05:48 AM, Greg KH wrote:
> > On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
> >> On 10/06/2016 09:22 PM, Greg KH wrote:
> >>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
> >>>> Hi!
> >>>>
> >>>> On 10/06/2016 08:52 PM, Greg KH wrote:
> >>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
> >>>>>> Hi, Stable!
> >>>>>>
> >>>>>> As you might be aware of, some companies that maintain linux kernel
> >>>>>> drivers have the habit of assigning each driver change a new version
> >>>>>> number.
> >>>>> And, as you have found out, that's a horrible thing to do for Linux and
> >>>>> doesn't work at all :)
> >>>>>
> >>>>> Just because it works for other slower-moving operating systems, I
> >>>>> wouldn't recommend doing it for Linux.
> >>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
> >>>> with the help some bright ideas from the list could come up with a
> >>>> clever way to make everybody happy.
> >>> But who has the problem here really?  Not the kernel community or
> >>> developers, but rather an odd set of unskilled QA people (your word, not
> >>> mine.)
> >>>
> >>> Why can't they get more "skill"?  :)
> >>>
> >>> thanks,
> >>>
> >>> greg k-h
> >> Well, I would in no way call our QA people unskilled just because they
> >> in general don't have the skill to know how to locate a particular,
> >> sometimes well-hidden git repo and find out if a certain bug is fixed or
> >> not. Not even Einstein knew how to do that ;)
> > Huh?  All of the kernel trees we "release" are in one single repo, and
> > it is very well known (linked to off of the kernel.org site front page):
> > 	https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e= 
> >
> > How is that difficult to find?
> 
> The "vanilla" stable ones are easy. The distro ones may not be, save
> Ubuntu that sometimes "take over" a stable tree. Typically the kernels
> we test are a distro-modified version of a stable tree.

Then go complain to the distros!  And even then, all of them keep their
kernels in pretty well-known, and documented, locations.  If not, go bug
them, there is nothing we can do about it.

Also, shouldn't your QA scripts just suck in the correct distro
kernel/tree automatically?  No QA person should have to ever hunt for a
kernel tree, that means you have not automated it, which seems very
wrong to me.

> >> But I won't try to argue here. I do think, though, that as long as
> >> people believe the easier solution is to version each change they will
> >> keep on doing that and unfortunately as a result important patches won't
> >> get CC'd stable because that would mess up the versioning.
> >>
> >> From your answer I take it there is no interest from the stable
> >> maintainers in helping solving this using some kind of mainline hash
> >> registering tool. I guess perhaps another option is to locally automate
> >> stable / distro git tree scanning.
> > Maybe I really don't understand the "issue" you are trying to address
> > here, can you try to rephrase it by showing a real example of what you
> > are trying to solve?
> >
> > But again, there's nothing we can do about out-of-tree code, remember,
> > they know where we are (and I'll take anything!), but we don't know
> > where they are...
> >
> > thanks,
> >
> > greg k-h
> 
> Yes. The problem would be
> 
> Given a *binary* version of distro kernel X, based on stable kernel Y.
> What _upstreamed_ bugfix patches has touched our module since the stable
> branch was created? Let's assume the distro git tree is hard to find.
> 
> a) Now if stable maintainers and distro kernel maintainers could use a
> flag "record commit id" to the git am command, the mainline commit id
> would be added to a binary visible table in the module, problem solved.

But the stable mantainers DO all do that already today!  That info is
all there, and has been there, for over a decade!  Just look at every
commit in the stable kernel branches, it has that information for you,
in a semi-easy format to parse.

If you have distro issues, go complain to them, nothing this list can do
about that, sorry.

> And if nobody else is interested, we'd probably be better off with b)
> provided we can gain access to the git trees of the important distro
> kernels.

I find it hard to believe you don't have access to them already.  But
again, if not, there's nothing we can do here, right?

thanks,

greg k-h

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/07/2016 07:18 AM, Greg KH wrote:
> On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
>> On 10/07/2016 05:48 AM, Greg KH wrote:
>>> On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
>>>> On 10/06/2016 09:22 PM, Greg KH wrote:
>>>>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
>>>>>> Hi!
>>>>>>
>>>>>> On 10/06/2016 08:52 PM, Greg KH wrote:
>>>>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>>>>>>>> Hi, Stable!
>>>>>>>>
>>>>>>>> As you might be aware of, some companies that maintain linux kernel
>>>>>>>> drivers have the habit of assigning each driver change a new version
>>>>>>>> number.
>>>>>>> And, as you have found out, that's a horrible thing to do for Linux and
>>>>>>> doesn't work at all :)
>>>>>>>
>>>>>>> Just because it works for other slower-moving operating systems, I
>>>>>>> wouldn't recommend doing it for Linux.
>>>>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
>>>>>> with the help some bright ideas from the list could come up with a
>>>>>> clever way to make everybody happy.
>>>>> But who has the problem here really?  Not the kernel community or
>>>>> developers, but rather an odd set of unskilled QA people (your word, not
>>>>> mine.)
>>>>>
>>>>> Why can't they get more "skill"?  :)
>>>>>
>>>>> thanks,
>>>>>
>>>>> greg k-h
>>>> Well, I would in no way call our QA people unskilled just because they
>>>> in general don't have the skill to know how to locate a particular,
>>>> sometimes well-hidden git repo and find out if a certain bug is fixed or
>>>> not. Not even Einstein knew how to do that ;)
>>> Huh?  All of the kernel trees we "release" are in one single repo, and
>>> it is very well known (linked to off of the kernel.org site front page):
>>> 	https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e= 
>>>
>>> How is that difficult to find?
>> The "vanilla" stable ones are easy. The distro ones may not be, save
>> Ubuntu that sometimes "take over" a stable tree. Typically the kernels
>> we test are a distro-modified version of a stable tree.
> Then go complain to the distros!  And even then, all of them keep their
> kernels in pretty well-known, and documented, locations.  If not, go bug
> them, there is nothing we can do about it.
>
> Also, shouldn't your QA scripts just suck in the correct distro
> kernel/tree automatically?  No QA person should have to ever hunt for a
> kernel tree, that means you have not automated it, which seems very
> wrong to me.
>
>>>> But I won't try to argue here. I do think, though, that as long as
>>>> people believe the easier solution is to version each change they will
>>>> keep on doing that and unfortunately as a result important patches won't
>>>> get CC'd stable because that would mess up the versioning.
>>>>
>>>> From your answer I take it there is no interest from the stable
>>>> maintainers in helping solving this using some kind of mainline hash
>>>> registering tool. I guess perhaps another option is to locally automate
>>>> stable / distro git tree scanning.
>>> Maybe I really don't understand the "issue" you are trying to address
>>> here, can you try to rephrase it by showing a real example of what you
>>> are trying to solve?
>>>
>>> But again, there's nothing we can do about out-of-tree code, remember,
>>> they know where we are (and I'll take anything!), but we don't know
>>> where they are...
>>>
>>> thanks,
>>>
>>> greg k-h
>> Yes. The problem would be
>>
>> Given a *binary* version of distro kernel X, based on stable kernel Y.
>> What _upstreamed_ bugfix patches has touched our module since the stable
>> branch was created? Let's assume the distro git tree is hard to find.
>>
>> a) Now if stable maintainers and distro kernel maintainers could use a
>> flag "record commit id" to the git am command, the mainline commit id
>> would be added to a binary visible table in the module, problem solved.
> But the stable mantainers DO all do that already today!  That info is
> all there, and has been there, for over a decade!  Just look at every
> commit in the stable kernel branches, it has that information for you,
> in a semi-easy format to parse.

Indeed they do, but the idea here was to have that information
extractable from a binary, but that would have required cooperation both
from the stable maintainers and the distro maintainers (who typically
are on this list). That's why I posted.

>
> If you have distro issues, go complain to them, nothing this list can do
> about that, sorry.
>
>> And if nobody else is interested, we'd probably be better off with b)
>> provided we can gain access to the git trees of the important distro
>> kernels.
> I find it hard to believe you don't have access to them already.  But
> again, if not, there's nothing we can do here, right?

Yes, that's right. Item b) would be a local thing.

>
> thanks,
>
> greg k-h

Thanks for your input Greg!

Thomas

Re: Stable list vs versioning

[Willy Tarreau]

On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
> Given a *binary* version of distro kernel X, based on stable kernel Y.
> What _upstreamed_ bugfix patches has touched our module since the stable
> branch was created? Let's assume the distro git tree is hard to find.
> 
> a) Now if stable maintainers and distro kernel maintainers could use a
> flag "record commit id" to the git am command, the mainline commit id
> would be added to a binary visible table in the module, problem solved.
                      ^^^^^^^^^^^^^^
Are you kidding ? For this you'd have :
  1) to patch each stable kernel to include such a list of commit IDs
  2) to include the whole list of upstream commit IDs into each module
     since you never know what commit affects what. For example, good
     luck for guessing what module is affected by this patch merged in
     4.4.24 :

     commit 70cd763eb1574cac07138be91f474a661e02d694
     Author: Subash Abhinov Kasiviswanathan <subashab@codeaurora.org>
     Date:   Thu Aug 25 15:16:51 2016 -0700

     sysctl: handle error writing UINT_MAX to u32 fields
    
     commit e7d316a02f683864a12389f8808570e37fb90aa3 upstream.
    
     We have scripts which write to certain fields on 3.18 kernels but this
     seems to be failing on 4.4 kernels.  An entry which we write to here is
     xfrm_aevent_rseqth which is u32.
    
       echo 4294967295  > /proc/sys/net/core/xfrm_aevent_rseqth
     (...)

  3) to needless inflate the kernels and initrd people are using
  4) to add more burden on the shoulders of the people already doing the
     job the most transparent manner possible, and who are already
     asking several times a week for upstream IDs when someone asks for
     a patch to be backported.

All this to *hope* to get these IDs from your theorically opensourced
kernels which you hope to know the base version. So if a kernel is called
3.10.73.46-g12345 it is supposed to contain all of 3.10.73 plus some changes
local to your vendor. It means the vendor will backport such changes
themselves, and you can be certain that they won't spend their time going
through the extra pain you're asking for if they already don't take the
time to *at least* rebase on the latest stable release of the same branch.

In all cases you're supposed to have their sources. Even if you only have
a huge tarball, you can diff it against the mainline kernels (I do that
from time to time when I want to rebase a shitty vendor BSP on top of
latest stable), so you can actually always figure what was merged and
what not.

If you don't get enough transparency from your vendor and this causes you
such a burden to track changes, you should simply let your vendor know.

Willy

Re: Stable list vs versioning

[Greg KH]

On Fri, Oct 07, 2016 at 08:05:59AM -0700, Thomas Hellstrom wrote:
> On 10/07/2016 07:18 AM, Greg KH wrote:
> > On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
> >> On 10/07/2016 05:48 AM, Greg KH wrote:
> >>> On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
> >>>> On 10/06/2016 09:22 PM, Greg KH wrote:
> >>>>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
> >>>>>> Hi!
> >>>>>>
> >>>>>> On 10/06/2016 08:52 PM, Greg KH wrote:
> >>>>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
> >>>>>>>> Hi, Stable!
> >>>>>>>>
> >>>>>>>> As you might be aware of, some companies that maintain linux kernel
> >>>>>>>> drivers have the habit of assigning each driver change a new version
> >>>>>>>> number.
> >>>>>>> And, as you have found out, that's a horrible thing to do for Linux and
> >>>>>>> doesn't work at all :)
> >>>>>>>
> >>>>>>> Just because it works for other slower-moving operating systems, I
> >>>>>>> wouldn't recommend doing it for Linux.
> >>>>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
> >>>>>> with the help some bright ideas from the list could come up with a
> >>>>>> clever way to make everybody happy.
> >>>>> But who has the problem here really?  Not the kernel community or
> >>>>> developers, but rather an odd set of unskilled QA people (your word, not
> >>>>> mine.)
> >>>>>
> >>>>> Why can't they get more "skill"?  :)
> >>>>>
> >>>>> thanks,
> >>>>>
> >>>>> greg k-h
> >>>> Well, I would in no way call our QA people unskilled just because they
> >>>> in general don't have the skill to know how to locate a particular,
> >>>> sometimes well-hidden git repo and find out if a certain bug is fixed or
> >>>> not. Not even Einstein knew how to do that ;)
> >>> Huh?  All of the kernel trees we "release" are in one single repo, and
> >>> it is very well known (linked to off of the kernel.org site front page):
> >>> 	https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e= 
> >>>
> >>> How is that difficult to find?
> >> The "vanilla" stable ones are easy. The distro ones may not be, save
> >> Ubuntu that sometimes "take over" a stable tree. Typically the kernels
> >> we test are a distro-modified version of a stable tree.
> > Then go complain to the distros!  And even then, all of them keep their
> > kernels in pretty well-known, and documented, locations.  If not, go bug
> > them, there is nothing we can do about it.
> >
> > Also, shouldn't your QA scripts just suck in the correct distro
> > kernel/tree automatically?  No QA person should have to ever hunt for a
> > kernel tree, that means you have not automated it, which seems very
> > wrong to me.
> >
> >>>> But I won't try to argue here. I do think, though, that as long as
> >>>> people believe the easier solution is to version each change they will
> >>>> keep on doing that and unfortunately as a result important patches won't
> >>>> get CC'd stable because that would mess up the versioning.
> >>>>
> >>>> From your answer I take it there is no interest from the stable
> >>>> maintainers in helping solving this using some kind of mainline hash
> >>>> registering tool. I guess perhaps another option is to locally automate
> >>>> stable / distro git tree scanning.
> >>> Maybe I really don't understand the "issue" you are trying to address
> >>> here, can you try to rephrase it by showing a real example of what you
> >>> are trying to solve?
> >>>
> >>> But again, there's nothing we can do about out-of-tree code, remember,
> >>> they know where we are (and I'll take anything!), but we don't know
> >>> where they are...
> >>>
> >>> thanks,
> >>>
> >>> greg k-h
> >> Yes. The problem would be
> >>
> >> Given a *binary* version of distro kernel X, based on stable kernel Y.
> >> What _upstreamed_ bugfix patches has touched our module since the stable
> >> branch was created? Let's assume the distro git tree is hard to find.
> >>
> >> a) Now if stable maintainers and distro kernel maintainers could use a
> >> flag "record commit id" to the git am command, the mainline commit id
> >> would be added to a binary visible table in the module, problem solved.
> > But the stable mantainers DO all do that already today!  That info is
> > all there, and has been there, for over a decade!  Just look at every
> > commit in the stable kernel branches, it has that information for you,
> > in a semi-easy format to parse.
> 
> Indeed they do, but the idea here was to have that information
> extractable from a binary, but that would have required cooperation both
> from the stable maintainers and the distro maintainers (who typically
> are on this list). That's why I posted.

You can't extract each individual patch information from a binary, how
would you encode 10k patches in every release?

Oh wait, look, we already do that with the git commit id as part of the
version number, you always know exactly what is contained in that binary
based on that.

So again, the community has already done this for you, I don't know why
you are ignoring it :(

And again, if you have problems with distro source trees, go complain to
them.  Yes, there are some distro developers on this list, but it's not
a distro-specific place to complain to them about things, you know
better than that...

thanks,

greg k-h

Re: Stable list vs versioning

[Josh Boyer]

On Fri, Oct 7, 2016 at 11:26 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Fri, Oct 07, 2016 at 08:05:59AM -0700, Thomas Hellstrom wrote:
>> On 10/07/2016 07:18 AM, Greg KH wrote:
>> > On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
>> >> On 10/07/2016 05:48 AM, Greg KH wrote:
>> >>> On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
>> >>>> On 10/06/2016 09:22 PM, Greg KH wrote:
>> >>>>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
>> >>>>>> Hi!
>> >>>>>>
>> >>>>>> On 10/06/2016 08:52 PM, Greg KH wrote:
>> >>>>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>> >>>>>>>> Hi, Stable!
>> >>>>>>>>
>> >>>>>>>> As you might be aware of, some companies that maintain linux kernel
>> >>>>>>>> drivers have the habit of assigning each driver change a new version
>> >>>>>>>> number.
>> >>>>>>> And, as you have found out, that's a horrible thing to do for Linux and
>> >>>>>>> doesn't work at all :)
>> >>>>>>>
>> >>>>>>> Just because it works for other slower-moving operating systems, I
>> >>>>>>> wouldn't recommend doing it for Linux.
>> >>>>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
>> >>>>>> with the help some bright ideas from the list could come up with a
>> >>>>>> clever way to make everybody happy.
>> >>>>> But who has the problem here really?  Not the kernel community or
>> >>>>> developers, but rather an odd set of unskilled QA people (your word, not
>> >>>>> mine.)
>> >>>>>
>> >>>>> Why can't they get more "skill"?  :)
>> >>>>>
>> >>>>> thanks,
>> >>>>>
>> >>>>> greg k-h
>> >>>> Well, I would in no way call our QA people unskilled just because they
>> >>>> in general don't have the skill to know how to locate a particular,
>> >>>> sometimes well-hidden git repo and find out if a certain bug is fixed or
>> >>>> not. Not even Einstein knew how to do that ;)
>> >>> Huh?  All of the kernel trees we "release" are in one single repo, and
>> >>> it is very well known (linked to off of the kernel.org site front page):
>> >>>   https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e=
>> >>>
>> >>> How is that difficult to find?
>> >> The "vanilla" stable ones are easy. The distro ones may not be, save
>> >> Ubuntu that sometimes "take over" a stable tree. Typically the kernels
>> >> we test are a distro-modified version of a stable tree.
>> > Then go complain to the distros!  And even then, all of them keep their
>> > kernels in pretty well-known, and documented, locations.  If not, go bug
>> > them, there is nothing we can do about it.
>> >
>> > Also, shouldn't your QA scripts just suck in the correct distro
>> > kernel/tree automatically?  No QA person should have to ever hunt for a
>> > kernel tree, that means you have not automated it, which seems very
>> > wrong to me.
>> >
>> >>>> But I won't try to argue here. I do think, though, that as long as
>> >>>> people believe the easier solution is to version each change they will
>> >>>> keep on doing that and unfortunately as a result important patches won't
>> >>>> get CC'd stable because that would mess up the versioning.
>> >>>>
>> >>>> From your answer I take it there is no interest from the stable
>> >>>> maintainers in helping solving this using some kind of mainline hash
>> >>>> registering tool. I guess perhaps another option is to locally automate
>> >>>> stable / distro git tree scanning.
>> >>> Maybe I really don't understand the "issue" you are trying to address
>> >>> here, can you try to rephrase it by showing a real example of what you
>> >>> are trying to solve?
>> >>>
>> >>> But again, there's nothing we can do about out-of-tree code, remember,
>> >>> they know where we are (and I'll take anything!), but we don't know
>> >>> where they are...
>> >>>
>> >>> thanks,
>> >>>
>> >>> greg k-h
>> >> Yes. The problem would be
>> >>
>> >> Given a *binary* version of distro kernel X, based on stable kernel Y.
>> >> What _upstreamed_ bugfix patches has touched our module since the stable
>> >> branch was created? Let's assume the distro git tree is hard to find.
>> >>
>> >> a) Now if stable maintainers and distro kernel maintainers could use a
>> >> flag "record commit id" to the git am command, the mainline commit id
>> >> would be added to a binary visible table in the module, problem solved.
>> > But the stable mantainers DO all do that already today!  That info is
>> > all there, and has been there, for over a decade!  Just look at every
>> > commit in the stable kernel branches, it has that information for you,
>> > in a semi-easy format to parse.
>>
>> Indeed they do, but the idea here was to have that information
>> extractable from a binary, but that would have required cooperation both
>> from the stable maintainers and the distro maintainers (who typically
>> are on this list). That's why I posted.
>
> You can't extract each individual patch information from a binary, how
> would you encode 10k patches in every release?
>
> Oh wait, look, we already do that with the git commit id as part of the
> version number, you always know exactly what is contained in that binary
> based on that.
>
> So again, the community has already done this for you, I don't know why
> you are ignoring it :(
>
> And again, if you have problems with distro source trees, go complain to
> them.  Yes, there are some distro developers on this list, but it's not
> a distro-specific place to complain to them about things, you know
> better than that...

I am curious which distros they're having trouble finding the kernel
tree for though.

josh

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/07/2016 08:26 AM, Greg KH wrote:
> On Fri, Oct 07, 2016 at 08:05:59AM -0700, Thomas Hellstrom wrote:
>> On 10/07/2016 07:18 AM, Greg KH wrote:
>>> On Fri, Oct 07, 2016 at 06:47:47AM -0700, Thomas Hellstrom wrote:
>>>> On 10/07/2016 05:48 AM, Greg KH wrote:
>>>>> On Thu, Oct 06, 2016 at 09:51:08PM -0700, Thomas Hellstrom wrote:
>>>>>> On 10/06/2016 09:22 PM, Greg KH wrote:
>>>>>>> On Thu, Oct 06, 2016 at 09:19:50PM -0700, Thomas Hellstrom wrote:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> On 10/06/2016 08:52 PM, Greg KH wrote:
>>>>>>>>> On Thu, Oct 06, 2016 at 06:54:43PM -0700, Thomas Hellstrom wrote:
>>>>>>>>>> Hi, Stable!
>>>>>>>>>>
>>>>>>>>>> As you might be aware of, some companies that maintain linux kernel
>>>>>>>>>> drivers have the habit of assigning each driver change a new version
>>>>>>>>>> number.
>>>>>>>>> And, as you have found out, that's a horrible thing to do for Linux and
>>>>>>>>> doesn't work at all :)
>>>>>>>>>
>>>>>>>>> Just because it works for other slower-moving operating systems, I
>>>>>>>>> wouldn't recommend doing it for Linux.
>>>>>>>> Yes, I'm fully aware of the difficulties, though I was hoping that I,
>>>>>>>> with the help some bright ideas from the list could come up with a
>>>>>>>> clever way to make everybody happy.
>>>>>>> But who has the problem here really?  Not the kernel community or
>>>>>>> developers, but rather an odd set of unskilled QA people (your word, not
>>>>>>> mine.)
>>>>>>>
>>>>>>> Why can't they get more "skill"?  :)
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> greg k-h
>>>>>> Well, I would in no way call our QA people unskilled just because they
>>>>>> in general don't have the skill to know how to locate a particular,
>>>>>> sometimes well-hidden git repo and find out if a certain bug is fixed or
>>>>>> not. Not even Einstein knew how to do that ;)
>>>>> Huh?  All of the kernel trees we "release" are in one single repo, and
>>>>> it is very well known (linked to off of the kernel.org site front page):
>>>>> 	https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_cgit_linux_kernel_git_stable_linux-2Dstable.git&d=CwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=vpukPkBtpoNQp2IUKuFviOmPNYWVKmen3Jeeu55zmEA&m=2nFSKLtpsbVgl3FEz2G3Io4y14rAxcjmJACORglPiwI&s=E02w2V0waHQkqaQ4KAcPYM3o2nWfYavhd12uJDJ24dI&e= 
>>>>>
>>>>> How is that difficult to find?
>>>> The "vanilla" stable ones are easy. The distro ones may not be, save
>>>> Ubuntu that sometimes "take over" a stable tree. Typically the kernels
>>>> we test are a distro-modified version of a stable tree.
>>> Then go complain to the distros!  And even then, all of them keep their
>>> kernels in pretty well-known, and documented, locations.  If not, go bug
>>> them, there is nothing we can do about it.
>>>
>>> Also, shouldn't your QA scripts just suck in the correct distro
>>> kernel/tree automatically?  No QA person should have to ever hunt for a
>>> kernel tree, that means you have not automated it, which seems very
>>> wrong to me.
>>>
>>>>>> But I won't try to argue here. I do think, though, that as long as
>>>>>> people believe the easier solution is to version each change they will
>>>>>> keep on doing that and unfortunately as a result important patches won't
>>>>>> get CC'd stable because that would mess up the versioning.
>>>>>>
>>>>>> From your answer I take it there is no interest from the stable
>>>>>> maintainers in helping solving this using some kind of mainline hash
>>>>>> registering tool. I guess perhaps another option is to locally automate
>>>>>> stable / distro git tree scanning.
>>>>> Maybe I really don't understand the "issue" you are trying to address
>>>>> here, can you try to rephrase it by showing a real example of what you
>>>>> are trying to solve?
>>>>>
>>>>> But again, there's nothing we can do about out-of-tree code, remember,
>>>>> they know where we are (and I'll take anything!), but we don't know
>>>>> where they are...
>>>>>
>>>>> thanks,
>>>>>
>>>>> greg k-h
>>>> Yes. The problem would be
>>>>
>>>> Given a *binary* version of distro kernel X, based on stable kernel Y.
>>>> What _upstreamed_ bugfix patches has touched our module since the stable
>>>> branch was created? Let's assume the distro git tree is hard to find.
>>>>
>>>> a) Now if stable maintainers and distro kernel maintainers could use a
>>>> flag "record commit id" to the git am command, the mainline commit id
>>>> would be added to a binary visible table in the module, problem solved.
>>> But the stable mantainers DO all do that already today!  That info is
>>> all there, and has been there, for over a decade!  Just look at every
>>> commit in the stable kernel branches, it has that information for you,
>>> in a semi-easy format to parse.
>> Indeed they do, but the idea here was to have that information
>> extractable from a binary, but that would have required cooperation both
>> from the stable maintainers and the distro maintainers (who typically
>> are on this list). That's why I posted.
> You can't extract each individual patch information from a binary, how
> would you encode 10k patches in every release?

Well, that wasn't the idea. The idea was to have the id's of the
*stable* backports
encoded automatically *only* for those modules that requested it.
However, I realize that
such a thing could easily grow..

>
> Oh wait, look, we already do that with the git commit id as part of the
> version number, you always know exactly what is contained in that binary
> based on that.
>
> So again, the community has already done this for you, I don't know why
> you are ignoring it :(
>
> And again, if you have problems with distro source trees, go complain to
> them.  Yes, there are some distro developers on this list, but it's not
> a distro-specific place to complain to them about things, you know
> better than that...
>
> thanks,
>
> greg k-h

So the intention was never to complain but to look for input and comments,
and I think I got it.

Thanks,

Thomas

Re: Stable list vs versioning

[Greg KH]

On Fri, Oct 07, 2016 at 08:45:28AM -0700, Thomas Hellstrom wrote:
> >> Indeed they do, but the idea here was to have that information
> >> extractable from a binary, but that would have required cooperation both
> >> from the stable maintainers and the distro maintainers (who typically
> >> are on this list). That's why I posted.
> > You can't extract each individual patch information from a binary, how
> > would you encode 10k patches in every release?
> 
> Well, that wasn't the idea. The idea was to have the id's of the
> *stable* backports
> encoded automatically *only* for those modules that requested it.
> However, I realize that such a thing could easily grow..

So, you want to see the 8-10 patches we add every single day to the tree
somehow?  For the 4.4.y kernel right now that would be 2753 patches.  Do
you want to waste that much memory for something that the source tree
provides for you today already, automatically in a provable way?

I really don't think you want this.

What vendor kernels are you having problems with, and why isn't your QA
scripts already set up to automatically pull from them with every push
they do?  I know a number of good vendors also provide build systems,
for free, that do this work for you, whenever the base repo changes.
Why aren't you using them today already for your kernel modules?

And again, the question you don't seem to ever answer, what kernel
modules are you needing this work for?  Why aren't they upstream?  What
is preventing that from happening?  Is it a failure somehow on our
development process?

thanks,

greg k-h

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/07/2016 10:13 AM, Greg KH wrote:
> On Fri, Oct 07, 2016 at 08:45:28AM -0700, Thomas Hellstrom wrote:
>>>> Indeed they do, but the idea here was to have that information
>>>> extractable from a binary, but that would have required cooperation both
>>>> from the stable maintainers and the distro maintainers (who typically
>>>> are on this list). That's why I posted.
>>> You can't extract each individual patch information from a binary, how
>>> would you encode 10k patches in every release?
>> Well, that wasn't the idea. The idea was to have the id's of the
>> *stable* backports
>> encoded automatically *only* for those modules that requested it.
>> However, I realize that such a thing could easily grow..
> So, you want to see the 8-10 patches we add every single day to the tree
> somehow?  For the 4.4.y kernel right now that would be 2753 patches.  Do
> you want to waste that much memory for something that the source tree
> provides for you today already, automatically in a provable way?
>
> I really don't think you want this.

Understood. Point taken.

>
> What vendor kernels are you having problems with, and why isn't your QA
> scripts already set up to automatically pull from them with every push
> they do?  I know a number of good vendors also provide build systems,
> for free, that do this work for you, whenever the base repo changes.
> Why aren't you using them today already for your kernel modules?
>
> And again, the question you don't seem to ever answer, what kernel
> modules are you needing this work for?  Why aren't they upstream?  What
> is preventing that from happening?  Is it a failure somehow on our
> development process?
>
> thanks,
>
> greg k-h

So my end goal here is to find a way for the vmware paravirtual in-tree
kernel modules to stop bumping version numbers on each commit / commit
series and start pushing important bugfixes to stable, while making
kernel maintainers happy and company people happy. There are no
out-of-tree modules whatsoever involved in this discussion.

It's totally clear to me at this point that binary commit information is
not the way to go, and that we have to evaluate other ways, perhaps some
of which you are suggesting above.

Thanks

Thomas

Re: Stable list vs versioning

[Greg KH]

On Fri, Oct 07, 2016 at 10:35:19AM -0700, Thomas Hellstrom wrote:
> So my end goal here is to find a way for the vmware paravirtual in-tree
> kernel modules to stop bumping version numbers on each commit / commit
> series and start pushing important bugfixes to stable, while making
> kernel maintainers happy and company people happy. There are no
> out-of-tree modules whatsoever involved in this discussion.

kernel maintainers don't care about version numbers for drivers at all,
just drop them, they cause more problems than they are worth.

thanks,

greg k-h

Re: Stable list vs versioning

[Willy Tarreau]

On Fri, Oct 07, 2016 at 10:35:19AM -0700, Thomas Hellstrom wrote:
> So my end goal here is to find a way for the vmware paravirtual in-tree
> kernel modules to stop bumping version numbers on each commit / commit
> series and start pushing important bugfixes to stable, while making
> kernel maintainers happy and company people happy.

Here it's better to simply drop module version at all, because not all
patches will be backported to stable, and the version becomes confusing.
>From time to time when we backport fixes to older kernels, we find a
driver fix refusing to apply because it also updates the version to
something much higher than what we have. It shows how inappropriate
these versions are. Enumerating patches between branches is not hard,
is much more accurate than relying on a version which a single patch
could increase while still missing other important patches.

Willy

Re: Stable list vs versioning

[Thomas Hellstrom]

On 10/08/2016 07:57 AM, Willy Tarreau wrote:
> On Fri, Oct 07, 2016 at 10:35:19AM -0700, Thomas Hellstrom wrote:
>> So my end goal here is to find a way for the vmware paravirtual in-tree
>> kernel modules to stop bumping version numbers on each commit / commit
>> series and start pushing important bugfixes to stable, while making
>> kernel maintainers happy and company people happy.
> Here it's better to simply drop module version at all, because not all
> patches will be backported to stable, and the version becomes confusing.
> From time to time when we backport fixes to older kernels, we find a
> driver fix refusing to apply because it also updates the version to
> something much higher than what we have. It shows how inappropriate
> these versions are. Enumerating patches between branches is not hard,
> is much more accurate than relying on a version which a single patch
> could increase while still missing other important patches.
>
> Willy

Thanks for your input Willy.

Thomas