Patches potentially missing from stable releases

Patches potentially missing from stable releases

[Henri Rosten]

We did some work on analyzing patches potentially missing from stable 
releases based on the Fixes: and Revert references in the commit 
messages. Our script is based on similar idea as described by Guenter 
Roeck in this earlier mail: 
https://lore.kernel.org/stable/20190827171621.GA30360@roeck-us.net/.

Although the list is not comprehensive, we figured it makes sense to 
publish it in case the results are of interest to someone else also.

The below list of potentially missing commits is based on 4.19, but some 
of the commits might also apply to 5.4 and 5.6.

For each potentially missing commit flagged by the script, we read the 
commit message and had a short look at the change. We then added 
comments on our own judgement if it might be stable material or not. No 
comments simply means the potentially missing patch appears stable 
material. "Based on commit" is the mainline patch that has been 
backported to 4.19 and is referenced by the missing commit. We did not 
check if the patch applies without changes, nor did we build or execute 
any tests.


6011002c1584    uio: make symbol 'uio_class_registered' static
    Based on commit: ae61cf5b9913

968339fad422    powerpc/boot: Delete unneeded .globl _zimage_start
    Based on commit: ee9d21b3b358
    Comment: not stable material?

ed4d81c4b3f2    net: aquantia: when cleaning hw cache it should be toggled
    Based on commit: 7a1bb49461b1
    Comment: this was backported to 5.3, but for some reason not to 
    older stable releases

b27507bb59ed    net/ibmvnic: unlock rtnl_lock in reset so linkwatch_event can ru
    Based on commit: a5681e20b541

61c347ba5511    afs: Clear AFS_VNODE_CB_PROMISED if we detect callback expiry
    Based on commit: ae3b7361dc0e
    Comment: likely requires manual backport

1a49b2fd8f58    kbuild: strip whitespace in cmd_record_mcount findstring
    Based on commit: 6977f95e63b9
    Comment: not stable material?

669e859b5ea7    btrfs: drop the lock on error in btrfs_dev_replace_cancel
    Based on commit: d189dd70e255
    Comment: earlier backport failed, this would likely require manual 
    backport: https://lore.kernel.org/stable/15531096685894@kroah.com/

dddaf89e2fbc    netfilter: ipt_CLUSTERIP: make symbol 'cip_netdev_notifier' stat
    Based on commit: 5a86d68bcf02

01091c496f92    acpi/nfit: improve bounds checking for 'func'
    Based on commit: 11189c1089da
    Comment: the missing commit was picked by AUTOSEL to 5.4, 5.5, and 
    5.6, but for some reason, it was not backported 4.19: 
    https://lore.kernel.org/stable/?q=01091c496f92*

2b74c878b0ea    IB/hfi1: Unreserve a flushed OPFN request
    Based on commit: ca95f802ef51
    Comment: earlier backport failed, this would likely require manual 
    backport: https://lore.kernel.org/stable/15649835016938@kroah.com/

0b815023a1d4    bnxt_en: Fix ring checking logic on 57500 chips.
    Based on commit: 36d65be9a880
    Comment: likely requires manual backport

6ae865615fc4    x86/uaccess: Dont leak the AC flag into __put_user() argument ev
    Based on commit: 2a418cf3f5f1
    Comment: commit 9b8bd476e78e which mentions complementing 
    6ae865615fc4, has already been backported to 4.19

70db5b04cbe1    f2fs: give some messages for inline_xattr_size
    Based on commit: 500e0b28ecd3
    Comment: not stable material?

2a06b8982f8f    net: reorder 'struct net' fields to avoid false sharing
    Based on commit: 355b98553789
    Comment: this was backported to 5.3, but for some reason not to 
    older stable releases

0fbf21c3b36a    ALSA: hda/realtek - Enable micmute LED for Huawei laptops
    Based on commit: 8ac51bbc4cfe
    Comment: not stable material?

Thanks,
-- Henri

Re: Patches potentially missing from stable releases

[Greg KH]

On Fri, May 29, 2020 at 03:24:47PM +0300, Henri Rosten wrote:
> We did some work on analyzing patches potentially missing from stable 
> releases based on the Fixes: and Revert references in the commit 
> messages. Our script is based on similar idea as described by Guenter 
> Roeck in this earlier mail: 
> https://lore.kernel.org/stable/20190827171621.GA30360@roeck-us.net/.
> 
> Although the list is not comprehensive, we figured it makes sense to 
> publish it in case the results are of interest to someone else also.
> 
> The below list of potentially missing commits is based on 4.19, but some 
> of the commits might also apply to 5.4 and 5.6.
> 
> For each potentially missing commit flagged by the script, we read the 
> commit message and had a short look at the change. We then added 
> comments on our own judgement if it might be stable material or not. No 
> comments simply means the potentially missing patch appears stable 
> material. "Based on commit" is the mainline patch that has been 
> backported to 4.19 and is referenced by the missing commit. We did not 
> check if the patch applies without changes, nor did we build or execute 
> any tests.

That last sentence should have been a huge red flag when writing it and
sending out this email...

> 6011002c1584    uio: make symbol 'uio_class_registered' static
>     Based on commit: ae61cf5b9913

You looked at this patch and thought it was relevant for backporting?

Why?  It "obviously" does not fix anything, and is just a cleanup patch.

Why did it pass your criteria?

> 968339fad422    powerpc/boot: Delete unneeded .globl _zimage_start
>     Based on commit: ee9d21b3b358
>     Comment: not stable material?

Ok, I'm going to stop here.

I appreciate sending lists of commits that you have determined should be
applied, and will be glad to review them, but you don't have to give me
a list of all potential patches and your comments on them, as that
doesn't help much.

I have scripts that can churn out the false-positive lists like these,
that's relatively easy to create.  It's the curated lists that you have
looked at and reviewed and determined, in your opinion, should be
applied that are much more valuable and better to work with.

> 61c347ba5511    afs: Clear AFS_VNODE_CB_PROMISED if we detect callback expiry
>     Based on commit: ae3b7361dc0e
>     Comment: likely requires manual backport

At this point, I will need others to do backporting for older kernels
like this, unless there is a good reason why you can't do it yourself.
That shows you actually want the patch to be backported, as you have
done the effort to do so and check that it builds properly.

Again, random lists aren't all that helpful, but curated ones are.

> 2b74c878b0ea    IB/hfi1: Unreserve a flushed OPFN request
>     Based on commit: ca95f802ef51
>     Comment: earlier backport failed, this would likely require manual 
>     backport: https://lore.kernel.org/stable/15649835016938@kroah.com/

This was known not to work.  I asked for help, and just asking for help
again isn't probably going to do much :(

> 0fbf21c3b36a    ALSA: hda/realtek - Enable micmute LED for Huawei laptops
>     Based on commit: 8ac51bbc4cfe
>     Comment: not stable material?

It doesn't apply and needs manual backporting.  Why didn't you test
that?

Again, curated ids, and backported patches are the best thing you can
do to help out here.  See Guenter's email for an example of the first,
and Ben's emails of backported patches for examples of the second.

thanks,

greg k-h

Re: Patches potentially missing from stable releases

[Henri Rosten]

On Fri, May 29, 2020 at 02:46:55PM +0200, Greg KH wrote:
> On Fri, May 29, 2020 at 03:24:47PM +0300, Henri Rosten wrote:
> > We did some work on analyzing patches potentially missing from stable 
> > releases based on the Fixes: and Revert references in the commit 
> > messages. Our script is based on similar idea as described by Guenter 
> > Roeck in this earlier mail: 
> > https://lore.kernel.org/stable/20190827171621.GA30360@roeck-us.net/.
> > 
> > Although the list is not comprehensive, we figured it makes sense to 
> > publish it in case the results are of interest to someone else also.
> > 
> > The below list of potentially missing commits is based on 4.19, but some 
> > of the commits might also apply to 5.4 and 5.6.
> > 
> > For each potentially missing commit flagged by the script, we read the 
> > commit message and had a short look at the change. We then added 
> > comments on our own judgement if it might be stable material or not. No 
> > comments simply means the potentially missing patch appears stable 
> > material. "Based on commit" is the mainline patch that has been 
> > backported to 4.19 and is referenced by the missing commit. We did not 
> > check if the patch applies without changes, nor did we build or execute 
> > any tests.
> 
> That last sentence should have been a huge red flag when writing it and
> sending out this email...

Thank you for your comments.

This list was a byproduct of other analysis we did, I did not intent to 
ask these patches for inclusion into stable. We simply wanted to report 
this list in case it includes some patches that might have dropped off 
your radar. We are aware that such information without further work, 
e.g. backporting and testing does not add much yet.

Thanks,
-- Henri

Re: Patches potentially missing from stable releases

[Guenter Roeck]

On 8/29/19 4:00 AM, Sasha Levin wrote:
> On Wed, Aug 28, 2019 at 08:22:40AM -0400, Sasha Levin wrote:
>> On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
>>> make sense to start with looking at Fixes: ? After all, additional
>>> references (wich higher chance for false positives) can always be
>>> searched for later.
>>
>> Yes, let me send a branch out for review later today and we could
>> compare our results.
> 
> The AUTOSEL set I've just sent
> (https://lore.kernel.org/stable/20190829105009.2265-1-sashal@kernel.org/)
> is really a batch of these fixes for v4.19 and older.
> 

Excellent!

I am working on improving my script further, letting me tag fixes as nuisance.
This already turns out to be quite useful for my own use ...

Thanks,
Guenter

Re: Patches potentially missing from stable releases

[Sasha Levin]

On Wed, Aug 28, 2019 at 08:22:40AM -0400, Sasha Levin wrote:
>On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
>>make sense to start with looking at Fixes: ? After all, additional
>>references (wich higher chance for false positives) can always be
>>searched for later.
>
>Yes, let me send a branch out for review later today and we could
>compare our results.

The AUTOSEL set I've just sent
(https://lore.kernel.org/stable/20190829105009.2265-1-sashal@kernel.org/)
is really a batch of these fixes for v4.19 and older.

--
Thanks,
Sasha

Re: Patches potentially missing from stable releases

[Guenter Roeck]

On Wed, Aug 28, 2019 at 04:54:43PM +0200, Greg Kroah-Hartman wrote:
> 
> I'm not saying it would be 30 minutes "from scratch", as I already have
> most of this all working already as I am doing this today for the syzbot
> stuff:
> 	https://github.com/gregkh/gregkh-linux/blob/master/scripts/syzbot_search
> 
> A "real" script would be wonderful to have, thanks!
> 

My script is a bit more fancy. It uses sqlite databases to keep track of
commits.

	https://github.com/groeck/findmissing.git

The initial setup takes a while, but subsequent runs are decently fast.
Example use: Run "./setup.sh" followed by "./missing.py 5.2" to get:

Checking branch linux-5.2.y
SHA 5319277968c1 [025bf37725f1] ('gpio: Fix return value mismatch of function gpiod_get_from_of_node()')
  fixed by upstream commit 2a6fc3cb5cb6
  Fix may be missing from linux-5.2.y; trying to apply it results in conflicts/errors
SHA b942dcdab8d1 [bd293d071ffe] ('dm bufio: fix deadlock with loop device')
  fixed by upstream commit cf3591ef8329
  Fix is missing from linux-5.2.y and applies cleanly
SHA 7c0044c1eec3 [d35661fcf95d] ('selftests/bpf: add wrapper scripts for test_xdp_vlan.sh')
  fixed by upstream commit 3035bb72ee47
  Fix is missing from linux-5.2.y and applies cleanly
SHA 60956b018bfe [883a2a80f79c] ('Input: elantech - enable SMBus on new (2018+) systems')
  fixed by upstream commit f3b5720cabaf
  Fix is missing from linux-5.2.y and applies cleanly

which seems about right.

Guenter

Re: Patches potentially missing from stable releases

[Greg Kroah-Hartman]

On Wed, Aug 28, 2019 at 06:43:43AM -0700, Guenter Roeck wrote:
> On 8/28/19 1:41 AM, Greg Kroah-Hartman wrote:
> > On Tue, Aug 27, 2019 at 01:48:41PM -0700, Guenter Roeck wrote:
> > > On Tue, Aug 27, 2019 at 10:29:01PM +0200, Greg Kroah-Hartman wrote:
> > > > On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
> > > > > On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
> > > > > > On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I recently wrote a script which identifies patches potentially missing
> > > > > > > in downstream kernel branches. The idea is to identify patches backported/
> > > > > > > applied to a downstream branch for which patches tagged with Fixes: are
> > > > > > > available in the upstream kernel, but those fixes are missing from the
> > > > > > > downstream branch. The script workflow is something like:
> > > > > > > 
> > > > > > > - Identify locally applied patches in downstream branch
> > > > > > > - For each patch, identify the matching upstream SHA
> > > > > > > - Search the upstream kernel for Fixes: tags with this SHA
> > > > > > > - If one or more patches with matching Fixes: tags are found, check
> > > > > > > if the patch was applied to the downstream branch.
> > > > > > > - If the patch was not applied to the downstream branch, report
> > > > > > > 
> > > > > > > Running this script on chromeos-4.19 identified, not surprisingly, a number
> > > > > > > of such patches. However, and more surprisingly, it also identified several
> > > > > > > patches applied to v4.19.y for which fixes are available in the upstream
> > > > > > > kernel, but those fixes have not been applied to v4.19.y. Some of those
> > > > > > > are on the cosmetic side, but several seem to be relevant. I didn't
> > > > > > > cross-check all of them, but the ones I tried did apply to linux-4.19.y.
> > > > > > > The complete list is attached below.
> > > > > > > 
> > > > > > > Question: Do Sasha's automated scripts identify such patches ? If not,
> > > > > > > would it make sense to do it ? Or is there some reason why the patches
> > > > > > > have not been applied to v4.19.y ?
> > > > > > 
> > > > > > Hey Guenter,
> > > > > > 
> > > > > > I have a very similar script with a slight difference: I don't try to
> > > > > > find just "Fixes:" tags, but rather just any reference from one patch to
> > > > > > another. This tends to catch cases where once patch states it's "a
> > > > > > similar fix to ..." and such.
> > > > > > 
> > > > > > The tricky part is that it's causing a whole bunch of false positives,
> > > > > > which takes a while to weed through - and that's where the issue is
> > > > > > right now.
> > > > > > 
> > > > > 
> > > > > I didn't see any false positives, at least not yet. Would it possibly
> > > > > make sense to start with looking at Fixes: ? After all, additional
> > > > > references (wich higher chance for false positives) can always be
> > > > > searched for later.
> > > > 
> > > > I used to do this "by hand" with a tiny bit of automation, but need to
> > > > step it up and do it "correctly" like you did.
> > > > 
> > > > If you have a pointer to your script, I'd be glad to run it here locally
> > > > to keep track of this, like I do so for patches tagged with syzbot
> > > > issues.
> > > > 
> > > 
> > > I'd have to rewrite it to work with stable branches. Its current
> > > primary goal is to assist the rebase of one chromeos branch to
> > > a later upstream kernel release. I just repurposed part of it and
> > > use the generated databases to identify patches tagged with Fixes:.
> > > 
> > > I'll be happy to do that and make it work on stable branches in
> > > general if you think it would add value.
> > 
> > No worries, I can add the functionality to my local scripts that I have.
> > If you just had happened to have it in stand-alone format, it would have
> > saved me 30 minutes or so :)
> > 
> 
> I'll do it anyway. Already almost done. You are apparently better than me,
> though. It takes me a bit more than 30 minutes, but then I am using the
> opportunity to improve the scripts a bit. I'll publish the whole thing
> at github once complete.

I'm not saying it would be 30 minutes "from scratch", as I already have
most of this all working already as I am doing this today for the syzbot
stuff:
	https://github.com/gregkh/gregkh-linux/blob/master/scripts/syzbot_search

A "real" script would be wonderful to have, thanks!

thanks,

greg k-h

Re: Patches potentially missing from stable releases

[Guenter Roeck]

On 8/28/19 1:41 AM, Greg Kroah-Hartman wrote:
> On Tue, Aug 27, 2019 at 01:48:41PM -0700, Guenter Roeck wrote:
>> On Tue, Aug 27, 2019 at 10:29:01PM +0200, Greg Kroah-Hartman wrote:
>>> On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
>>>> On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
>>>>> On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I recently wrote a script which identifies patches potentially missing
>>>>>> in downstream kernel branches. The idea is to identify patches backported/
>>>>>> applied to a downstream branch for which patches tagged with Fixes: are
>>>>>> available in the upstream kernel, but those fixes are missing from the
>>>>>> downstream branch. The script workflow is something like:
>>>>>>
>>>>>> - Identify locally applied patches in downstream branch
>>>>>> - For each patch, identify the matching upstream SHA
>>>>>> - Search the upstream kernel for Fixes: tags with this SHA
>>>>>> - If one or more patches with matching Fixes: tags are found, check
>>>>>> if the patch was applied to the downstream branch.
>>>>>> - If the patch was not applied to the downstream branch, report
>>>>>>
>>>>>> Running this script on chromeos-4.19 identified, not surprisingly, a number
>>>>>> of such patches. However, and more surprisingly, it also identified several
>>>>>> patches applied to v4.19.y for which fixes are available in the upstream
>>>>>> kernel, but those fixes have not been applied to v4.19.y. Some of those
>>>>>> are on the cosmetic side, but several seem to be relevant. I didn't
>>>>>> cross-check all of them, but the ones I tried did apply to linux-4.19.y.
>>>>>> The complete list is attached below.
>>>>>>
>>>>>> Question: Do Sasha's automated scripts identify such patches ? If not,
>>>>>> would it make sense to do it ? Or is there some reason why the patches
>>>>>> have not been applied to v4.19.y ?
>>>>>
>>>>> Hey Guenter,
>>>>>
>>>>> I have a very similar script with a slight difference: I don't try to
>>>>> find just "Fixes:" tags, but rather just any reference from one patch to
>>>>> another. This tends to catch cases where once patch states it's "a
>>>>> similar fix to ..." and such.
>>>>>
>>>>> The tricky part is that it's causing a whole bunch of false positives,
>>>>> which takes a while to weed through - and that's where the issue is
>>>>> right now.
>>>>>
>>>>
>>>> I didn't see any false positives, at least not yet. Would it possibly
>>>> make sense to start with looking at Fixes: ? After all, additional
>>>> references (wich higher chance for false positives) can always be
>>>> searched for later.
>>>
>>> I used to do this "by hand" with a tiny bit of automation, but need to
>>> step it up and do it "correctly" like you did.
>>>
>>> If you have a pointer to your script, I'd be glad to run it here locally
>>> to keep track of this, like I do so for patches tagged with syzbot
>>> issues.
>>>
>>
>> I'd have to rewrite it to work with stable branches. Its current
>> primary goal is to assist the rebase of one chromeos branch to
>> a later upstream kernel release. I just repurposed part of it and
>> use the generated databases to identify patches tagged with Fixes:.
>>
>> I'll be happy to do that and make it work on stable branches in
>> general if you think it would add value.
> 
> No worries, I can add the functionality to my local scripts that I have.
> If you just had happened to have it in stand-alone format, it would have
> saved me 30 minutes or so :)
> 

I'll do it anyway. Already almost done. You are apparently better than me,
though. It takes me a bit more than 30 minutes, but then I am using the
opportunity to improve the scripts a bit. I'll publish the whole thing
at github once complete.

Guenter

Re: Patches potentially missing from stable releases

[Sasha Levin]

On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
>On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
>> On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
>> >Hi,
>> >
>> >I recently wrote a script which identifies patches potentially missing
>> >in downstream kernel branches. The idea is to identify patches backported/
>> >applied to a downstream branch for which patches tagged with Fixes: are
>> >available in the upstream kernel, but those fixes are missing from the
>> >downstream branch. The script workflow is something like:
>> >
>> >- Identify locally applied patches in downstream branch
>> >- For each patch, identify the matching upstream SHA
>> >- Search the upstream kernel for Fixes: tags with this SHA
>> >- If one or more patches with matching Fixes: tags are found, check
>> > if the patch was applied to the downstream branch.
>> >- If the patch was not applied to the downstream branch, report
>> >
>> >Running this script on chromeos-4.19 identified, not surprisingly, a number
>> >of such patches. However, and more surprisingly, it also identified several
>> >patches applied to v4.19.y for which fixes are available in the upstream
>> >kernel, but those fixes have not been applied to v4.19.y. Some of those
>> >are on the cosmetic side, but several seem to be relevant. I didn't
>> >cross-check all of them, but the ones I tried did apply to linux-4.19.y.
>> >The complete list is attached below.
>> >
>> >Question: Do Sasha's automated scripts identify such patches ? If not,
>> >would it make sense to do it ? Or is there some reason why the patches
>> >have not been applied to v4.19.y ?
>>
>> Hey Guenter,
>>
>> I have a very similar script with a slight difference: I don't try to
>> find just "Fixes:" tags, but rather just any reference from one patch to
>> another. This tends to catch cases where once patch states it's "a
>> similar fix to ..." and such.
>>
>> The tricky part is that it's causing a whole bunch of false positives,
>> which takes a while to weed through - and that's where the issue is
>> right now.
>>
>
>I didn't see any false positives, at least not yet. Would it possibly

I was referring to things that say that they "fixes:", but the fix it
not stable material (typos, fallthrough, etc).

>make sense to start with looking at Fixes: ? After all, additional
>references (wich higher chance for false positives) can always be
>searched for later.

Yes, let me send a branch out for review later today and we could
compare our results.

--
Thanks,
Sasha

Re: Patches potentially missing from stable releases

[Greg Kroah-Hartman]

On Tue, Aug 27, 2019 at 01:48:41PM -0700, Guenter Roeck wrote:
> On Tue, Aug 27, 2019 at 10:29:01PM +0200, Greg Kroah-Hartman wrote:
> > On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
> > > On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
> > > > On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
> > > > >Hi,
> > > > >
> > > > >I recently wrote a script which identifies patches potentially missing
> > > > >in downstream kernel branches. The idea is to identify patches backported/
> > > > >applied to a downstream branch for which patches tagged with Fixes: are
> > > > >available in the upstream kernel, but those fixes are missing from the
> > > > >downstream branch. The script workflow is something like:
> > > > >
> > > > >- Identify locally applied patches in downstream branch
> > > > >- For each patch, identify the matching upstream SHA
> > > > >- Search the upstream kernel for Fixes: tags with this SHA
> > > > >- If one or more patches with matching Fixes: tags are found, check
> > > > > if the patch was applied to the downstream branch.
> > > > >- If the patch was not applied to the downstream branch, report
> > > > >
> > > > >Running this script on chromeos-4.19 identified, not surprisingly, a number
> > > > >of such patches. However, and more surprisingly, it also identified several
> > > > >patches applied to v4.19.y for which fixes are available in the upstream
> > > > >kernel, but those fixes have not been applied to v4.19.y. Some of those
> > > > >are on the cosmetic side, but several seem to be relevant. I didn't
> > > > >cross-check all of them, but the ones I tried did apply to linux-4.19.y.
> > > > >The complete list is attached below.
> > > > >
> > > > >Question: Do Sasha's automated scripts identify such patches ? If not,
> > > > >would it make sense to do it ? Or is there some reason why the patches
> > > > >have not been applied to v4.19.y ?
> > > > 
> > > > Hey Guenter,
> > > > 
> > > > I have a very similar script with a slight difference: I don't try to
> > > > find just "Fixes:" tags, but rather just any reference from one patch to
> > > > another. This tends to catch cases where once patch states it's "a
> > > > similar fix to ..." and such.
> > > > 
> > > > The tricky part is that it's causing a whole bunch of false positives,
> > > > which takes a while to weed through - and that's where the issue is
> > > > right now.
> > > > 
> > > 
> > > I didn't see any false positives, at least not yet. Would it possibly
> > > make sense to start with looking at Fixes: ? After all, additional
> > > references (wich higher chance for false positives) can always be
> > > searched for later.
> > 
> > I used to do this "by hand" with a tiny bit of automation, but need to
> > step it up and do it "correctly" like you did.
> > 
> > If you have a pointer to your script, I'd be glad to run it here locally
> > to keep track of this, like I do so for patches tagged with syzbot
> > issues.
> > 
> 
> I'd have to rewrite it to work with stable branches. Its current
> primary goal is to assist the rebase of one chromeos branch to
> a later upstream kernel release. I just repurposed part of it and
> use the generated databases to identify patches tagged with Fixes:.
> 
> I'll be happy to do that and make it work on stable branches in
> general if you think it would add value.

No worries, I can add the functionality to my local scripts that I have.
If you just had happened to have it in stand-alone format, it would have
saved me 30 minutes or so :)

thanks for the reminder that we need to be doing this more,

greg k-h

Re: Patches potentially missing from stable releases

[Guenter Roeck]

On Tue, Aug 27, 2019 at 10:29:01PM +0200, Greg Kroah-Hartman wrote:
> On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
> > On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
> > > On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
> > > >Hi,
> > > >
> > > >I recently wrote a script which identifies patches potentially missing
> > > >in downstream kernel branches. The idea is to identify patches backported/
> > > >applied to a downstream branch for which patches tagged with Fixes: are
> > > >available in the upstream kernel, but those fixes are missing from the
> > > >downstream branch. The script workflow is something like:
> > > >
> > > >- Identify locally applied patches in downstream branch
> > > >- For each patch, identify the matching upstream SHA
> > > >- Search the upstream kernel for Fixes: tags with this SHA
> > > >- If one or more patches with matching Fixes: tags are found, check
> > > > if the patch was applied to the downstream branch.
> > > >- If the patch was not applied to the downstream branch, report
> > > >
> > > >Running this script on chromeos-4.19 identified, not surprisingly, a number
> > > >of such patches. However, and more surprisingly, it also identified several
> > > >patches applied to v4.19.y for which fixes are available in the upstream
> > > >kernel, but those fixes have not been applied to v4.19.y. Some of those
> > > >are on the cosmetic side, but several seem to be relevant. I didn't
> > > >cross-check all of them, but the ones I tried did apply to linux-4.19.y.
> > > >The complete list is attached below.
> > > >
> > > >Question: Do Sasha's automated scripts identify such patches ? If not,
> > > >would it make sense to do it ? Or is there some reason why the patches
> > > >have not been applied to v4.19.y ?
> > > 
> > > Hey Guenter,
> > > 
> > > I have a very similar script with a slight difference: I don't try to
> > > find just "Fixes:" tags, but rather just any reference from one patch to
> > > another. This tends to catch cases where once patch states it's "a
> > > similar fix to ..." and such.
> > > 
> > > The tricky part is that it's causing a whole bunch of false positives,
> > > which takes a while to weed through - and that's where the issue is
> > > right now.
> > > 
> > 
> > I didn't see any false positives, at least not yet. Would it possibly
> > make sense to start with looking at Fixes: ? After all, additional
> > references (wich higher chance for false positives) can always be
> > searched for later.
> 
> I used to do this "by hand" with a tiny bit of automation, but need to
> step it up and do it "correctly" like you did.
> 
> If you have a pointer to your script, I'd be glad to run it here locally
> to keep track of this, like I do so for patches tagged with syzbot
> issues.
> 

I'd have to rewrite it to work with stable branches. Its current
primary goal is to assist the rebase of one chromeos branch to
a later upstream kernel release. I just repurposed part of it and
use the generated databases to identify patches tagged with Fixes:.

I'll be happy to do that and make it work on stable branches in
general if you think it would add value.

Thanks,
Guenter

Re: Patches potentially missing from stable releases

[Greg Kroah-Hartman]

On Tue, Aug 27, 2019 at 01:01:51PM -0700, Guenter Roeck wrote:
> On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
> > On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
> > >Hi,
> > >
> > >I recently wrote a script which identifies patches potentially missing
> > >in downstream kernel branches. The idea is to identify patches backported/
> > >applied to a downstream branch for which patches tagged with Fixes: are
> > >available in the upstream kernel, but those fixes are missing from the
> > >downstream branch. The script workflow is something like:
> > >
> > >- Identify locally applied patches in downstream branch
> > >- For each patch, identify the matching upstream SHA
> > >- Search the upstream kernel for Fixes: tags with this SHA
> > >- If one or more patches with matching Fixes: tags are found, check
> > > if the patch was applied to the downstream branch.
> > >- If the patch was not applied to the downstream branch, report
> > >
> > >Running this script on chromeos-4.19 identified, not surprisingly, a number
> > >of such patches. However, and more surprisingly, it also identified several
> > >patches applied to v4.19.y for which fixes are available in the upstream
> > >kernel, but those fixes have not been applied to v4.19.y. Some of those
> > >are on the cosmetic side, but several seem to be relevant. I didn't
> > >cross-check all of them, but the ones I tried did apply to linux-4.19.y.
> > >The complete list is attached below.
> > >
> > >Question: Do Sasha's automated scripts identify such patches ? If not,
> > >would it make sense to do it ? Or is there some reason why the patches
> > >have not been applied to v4.19.y ?
> > 
> > Hey Guenter,
> > 
> > I have a very similar script with a slight difference: I don't try to
> > find just "Fixes:" tags, but rather just any reference from one patch to
> > another. This tends to catch cases where once patch states it's "a
> > similar fix to ..." and such.
> > 
> > The tricky part is that it's causing a whole bunch of false positives,
> > which takes a while to weed through - and that's where the issue is
> > right now.
> > 
> 
> I didn't see any false positives, at least not yet. Would it possibly
> make sense to start with looking at Fixes: ? After all, additional
> references (wich higher chance for false positives) can always be
> searched for later.

I used to do this "by hand" with a tiny bit of automation, but need to
step it up and do it "correctly" like you did.

If you have a pointer to your script, I'd be glad to run it here locally
to keep track of this, like I do so for patches tagged with syzbot
issues.

thanks,

greg k-h

Re: Patches potentially missing from stable releases

[Guenter Roeck]

On Tue, Aug 27, 2019 at 02:10:03PM -0400, Sasha Levin wrote:
> On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
> >Hi,
> >
> >I recently wrote a script which identifies patches potentially missing
> >in downstream kernel branches. The idea is to identify patches backported/
> >applied to a downstream branch for which patches tagged with Fixes: are
> >available in the upstream kernel, but those fixes are missing from the
> >downstream branch. The script workflow is something like:
> >
> >- Identify locally applied patches in downstream branch
> >- For each patch, identify the matching upstream SHA
> >- Search the upstream kernel for Fixes: tags with this SHA
> >- If one or more patches with matching Fixes: tags are found, check
> > if the patch was applied to the downstream branch.
> >- If the patch was not applied to the downstream branch, report
> >
> >Running this script on chromeos-4.19 identified, not surprisingly, a number
> >of such patches. However, and more surprisingly, it also identified several
> >patches applied to v4.19.y for which fixes are available in the upstream
> >kernel, but those fixes have not been applied to v4.19.y. Some of those
> >are on the cosmetic side, but several seem to be relevant. I didn't
> >cross-check all of them, but the ones I tried did apply to linux-4.19.y.
> >The complete list is attached below.
> >
> >Question: Do Sasha's automated scripts identify such patches ? If not,
> >would it make sense to do it ? Or is there some reason why the patches
> >have not been applied to v4.19.y ?
> 
> Hey Guenter,
> 
> I have a very similar script with a slight difference: I don't try to
> find just "Fixes:" tags, but rather just any reference from one patch to
> another. This tends to catch cases where once patch states it's "a
> similar fix to ..." and such.
> 
> The tricky part is that it's causing a whole bunch of false positives,
> which takes a while to weed through - and that's where the issue is
> right now.
> 

I didn't see any false positives, at least not yet. Would it possibly
make sense to start with looking at Fixes: ? After all, additional
references (wich higher chance for false positives) can always be
searched for later.

Thanks,
Guenter

> I try to review a few each week and queue them up together with my
> autosel patches, but I guess I should step it up a bit.
> 
> Let me go over my list and try to catch up - I think I'll have time in
> the very near future.
> 
> --
> Thanks,
> Sasha

Re: Patches potentially missing from stable releases

[Sasha Levin]

On Tue, Aug 27, 2019 at 10:16:21AM -0700, Guenter Roeck wrote:
>Hi,
>
>I recently wrote a script which identifies patches potentially missing
>in downstream kernel branches. The idea is to identify patches backported/
>applied to a downstream branch for which patches tagged with Fixes: are
>available in the upstream kernel, but those fixes are missing from the
>downstream branch. The script workflow is something like:
>
>- Identify locally applied patches in downstream branch
>- For each patch, identify the matching upstream SHA
>- Search the upstream kernel for Fixes: tags with this SHA
>- If one or more patches with matching Fixes: tags are found, check
>  if the patch was applied to the downstream branch.
>- If the patch was not applied to the downstream branch, report
>
>Running this script on chromeos-4.19 identified, not surprisingly, a number
>of such patches. However, and more surprisingly, it also identified several
>patches applied to v4.19.y for which fixes are available in the upstream
>kernel, but those fixes have not been applied to v4.19.y. Some of those
>are on the cosmetic side, but several seem to be relevant. I didn't
>cross-check all of them, but the ones I tried did apply to linux-4.19.y.
>The complete list is attached below.
>
>Question: Do Sasha's automated scripts identify such patches ? If not,
>would it make sense to do it ? Or is there some reason why the patches
>have not been applied to v4.19.y ?

Hey Guenter,

I have a very similar script with a slight difference: I don't try to
find just "Fixes:" tags, but rather just any reference from one patch to
another. This tends to catch cases where once patch states it's "a
similar fix to ..." and such.

The tricky part is that it's causing a whole bunch of false positives,
which takes a while to weed through - and that's where the issue is
right now.

I try to review a few each week and queue them up together with my
autosel patches, but I guess I should step it up a bit.

Let me go over my list and try to catch up - I think I'll have time in
the very near future.

--
Thanks,
Sasha

Patches potentially missing from stable releases

[Guenter Roeck]

Hi,

I recently wrote a script which identifies patches potentially missing
in downstream kernel branches. The idea is to identify patches backported/
applied to a downstream branch for which patches tagged with Fixes: are
available in the upstream kernel, but those fixes are missing from the
downstream branch. The script workflow is something like:

- Identify locally applied patches in downstream branch
- For each patch, identify the matching upstream SHA
- Search the upstream kernel for Fixes: tags with this SHA
- If one or more patches with matching Fixes: tags are found, check
  if the patch was applied to the downstream branch. 
- If the patch was not applied to the downstream branch, report

Running this script on chromeos-4.19 identified, not surprisingly, a number
of such patches. However, and more surprisingly, it also identified several
patches applied to v4.19.y for which fixes are available in the upstream
kernel, but those fixes have not been applied to v4.19.y. Some of those
are on the cosmetic side, but several seem to be relevant. I didn't
cross-check all of them, but the ones I tried did apply to linux-4.19.y.
The complete list is attached below.

Question: Do Sasha's automated scripts identify such patches ? If not,
would it make sense to do it ? Or is there some reason why the patches
have not been applied to v4.19.y ?

Thanks,
Guenter

---
SHA ce081fc137c8 [ce49d8436cff] ('perf strbuf: Match va_{add,copy} with va_end')
  fixed by upstream commit 099be748865e
  Fix is missing from chromeos-4.19 and applies cleanly
SHA c21099be0233 [ae61cf5b9913] ('uio: ensure class is registered before devices')
  fixed by upstream commit 6011002c1584
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 3252b60cf810 [578bdaabd015] ('crypto: speck - remove Speck')
  fixed by upstream commit 733ac4f9935c
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 0858d74e8845 [bcc71cc3cde1] ('scsi: qla2xxx: Fix for double free of SRB structure')
  fixed by upstream commit ef801f07e7b3
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 3d8c2945fcbf [8985167ecf57] ('clk: s2mps11: Fix matching when built as module and DT node contains compatible')
  fixed by upstream commit 9c940bbe2bb4
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 67a19f87a02b [ac5b2c18911f] ('mm: thp: relax __GFP_THISNODE for MADV_HUGEPAGE mappings')
  fixed by upstream commit 2f0799a0ffc0
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA f55e301ec4d5 [a74cfffb03b7] ('x86/speculation: Rework SMT state change')
  fixed by upstream commit 34d66caf251d
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 84d2023c14ea [e949b6db51dc] ('riscv/function_graph: Simplify with function_graph_enter()')
  fixed by upstream commit 397182e0db56
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 0e79e30e6121 [4cff280a5fcc] ('nvme-fc: resolve io failures during connect')
  fixed by upstream commit 8730c1ddb69b
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 52da87f0e2e8 [ae3b7361dc0e] ('afs: Fix validation/callback interaction')
  fixed by upstream commit 61c347ba5511
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA cd50eeeb6646 [3e9efc3299dd] ('i2c: aspeed: fix build warning')
  fixed by upstream commit 2be6b47211e1
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 2658687568cd [7aa54be29765] ('locking/qspinlock, x86: Provide liveness guarantee')
  fixed by upstream commit b987ffc18fb3
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 6ffd9f25c0e9 [c3494801cd17] ('bpf: check pending signals while verifying programs')
  fixed by upstream commit 86edaed37963
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 53471f0d893d [04f05230c5c1] ('bnx2x: Remove configured vlans as part of unload sequence.')
  fixed by upstream commit 4a4d2d372fb9
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 341b8840435a [308c6cafde01] ('net: hns: All ports can not work when insmod hns ko after rmmod.')
  fixed by upstream commit c77804be5336
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA ba1fe90be68f [6977f95e63b9] ('powerpc: avoid -mno-sched-epilog on GCC 4.9 and newer')
  fixed by upstream commit 1a49b2fd8f58
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 7398668b3110 [4be9bd10e22d] ('drm/fb_helper: Allow leaking fbdev smem_start')
  fixed by upstream commit b31a3ca745a4
  Fix is missing from chromeos-4.19 and applies cleanly
SHA ad7013cd6d6a [8cc4ccf58379] ('netfilter: ipset: Allow matching on destination MAC address for mac and ipmac sets')
  fixed by upstream commit b89d15480d0c
  Fix is missing from chromeos-4.19 and applies cleanly
SHA ad7013cd6d6a [8cc4ccf58379] ('netfilter: ipset: Allow matching on destination MAC address for mac and ipmac sets')
  fixed by upstream commit 1b4a75108d5b
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 38b17eee7074 [d189dd70e255] ('btrfs: fix use-after-free due to race between replace start and cancel')
  fixed by upstream commit 669e859b5ea7
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 9d51378a6893 [5a86d68bcf02] ('netfilter: ipt_CLUSTERIP: fix deadlock in netns exit routine')
  fixed by upstream commit dddaf89e2fbc
  Fix is missing from chromeos-4.19 and applies cleanly
SHA b18931c5fe0d [11189c1089da] ('acpi/nfit: Fix command-supported detection')
  fixed by upstream commit 0171b6b78131
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 91e46947d02f [ca95f802ef51] ('IB/hfi1: Unreserve a reserved request when it is completed')
  fixed by upstream commit 2b74c878b0ea
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 4cd197bfa6e1 [36d65be9a880] ('bnxt_en: Disable MSIX before re-reserving NQs/CMPL rings.')
  fixed by upstream commit 0b815023a1d4
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA c709eeb02c04 [0d640732dbeb] ('arm64: KVM: Skip MMIO insn after emulation')
  fixed by upstream commit 2113c5f62b74
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 7371994d6cfa [2a418cf3f5f1] ('x86/uaccess: Don't leak the AC flag into __put_user() value evaluation')
  fixed by upstream commit 6ae865615fc4
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 8ce41db0dcfc [dd9ee3444014] ('vti4: Fix a ipip packet processing bug in 'IPCOMP' virtual tunnel')
  fixed by upstream commit 01ce31c57b3f
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 3355d641269f [2035f3ff8eaa] ('netfilter: ebtables: compat: un-break 32bit setsockopt when no rules are present')
  fixed by upstream commit 3b48300d5cc7
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 33e83ea302c0 [2c2ade81741c] ('mm: page_alloc: fix ref bias in page_frag_alloc() for 1-byte allocs')
  fixed by upstream commit 8644772637de
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 4ab78f4d75c6 [500e0b28ecd3] ('f2fs: fix to check inline_xattr_size boundary correctly')
  fixed by upstream commit 70db5b04cbe1
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 54fb5c9da6cd [272e5326c783] ('btrfs: prop: fix vanished compression property after failed set')
  fixed by upstream commit aa53e3bfac72
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA c963475972f6 [ef05bcb60c1a] ('arm64: dts: rockchip: fix vcc_host1_5v pin assign on rk3328-rock64')
  fixed by upstream commit 26e2d7b03ea7
  Fix is missing from chromeos-4.19 and applies cleanly
SHA e434fbf4f049 [8ac51bbc4cfe] ('ALSA: hda: fix front speakers on Huawei MBXP')
  fixed by upstream commit 0fbf21c3b36a
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA a1da981f6643 [131ac62253db] ('staging: most: core: use device description as name')
  fixed by upstream commit 3970d0d81816
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 6ff17bc5936e [04f5866e41fb] ('coredump: fix race condition between mmget_not_zero()/get_task_mm() and core dumping')
  fixed by upstream commit 46d0b24c5ee1
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA f7ab4818f74e [c01908a14bf7] ('HID: input: add mapping for "Toggle Display" key')
  fixed by upstream commit 1c703b53e5bf
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 7b115755fb9d [3c79107631db] ('netfilter: ctnetlink: don't use conntrack/expect object addresses as id')
  fixed by upstream commit 656c8e9cc1ba
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 8b13bb911f0c [c2d1b3aae336] ('btrfs: Honour FITRIM range constraints during free space trim')
  fixed by upstream commit 8103d10b7161
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 0388d45afc50 [03628cdbc64d] ('Btrfs: do not start a transaction during fiemap')
  fixed by upstream commit a6d155d2e363
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 2d2017675b1a [b5b5a27bee58] ('media: stm32-dcmi: return appropriate error codes during probe')
  fixed by upstream commit dbb9fcc8c2d8
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 8715ce033eb3 [f2c65fb3221a] ('x86/modules: Avoid breaking W^X while loading modules')
  fixed by upstream commit 2eef1399a866
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 8034a6b89990 [95f18c9d1310] ('bcache: avoid potential memleak of list of journal_replay(s) in the CACHE_SYNC branch of run_cache_set')
  fixed by upstream commit cdca22bcbc64
  Fix is missing from chromeos-4.19 and applies cleanly
SHA cd83c78897d5 [631207314d88] ('bcache: fix failure in journal relplay')
  fixed by upstream commit 2d5abb9a1e8e
  Fix is missing from chromeos-4.19 and applies cleanly
SHA fec8a09f79ec [56c46bba9bbf] ('powerpc/64: Fix booting large kernels with STRICT_KERNEL_RWX')
  fixed by upstream commit 9c4e4c90ec24
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 5d5652b51c87 [09f11b6c99fe] ('thunderbolt: Take domain lock in switch sysfs attribute callbacks')
  fixed by upstream commit 4f7c2e0d8765
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 9d57cfd4e9d8 [5cec2d2e5839] ('binder: fix race between munmap() and direct reclaim')
  fixed by upstream commit 60d488571083
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 6fa953c94882 [d2a68c4effd8] ('x86/ftrace: Do not call function graph from dynamic trampolines')
  fixed by upstream commit 745cfeaac09c
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 25511676362d [54c7a8916a88] ('initramfs: free initrd memory if opening /initrd.image fails')
  fixed by upstream commit 5d59aa8f9ce9
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 525b5265fd75 [fbc2a15e3433] ('blk-mq: move cancel of requeue_work into blk_mq_release')
  fixed by upstream commit e26cc08265dd
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 3e1d7417b4d6 [fc82d93e57e3] ('selftests: fib_rule_tests: fix local IPv4 address typo')
  fixed by upstream commit 34632975cafd
  Fix is missing from chromeos-4.19 and applies cleanly
SHA ca4c34037bb9 [e3ff9c3678b4] ('timekeeping: Repair ktime_get_coarse*() granularity')
  fixed by upstream commit 0354c1a3cdf3
  Fix is missing from chromeos-4.19 and applies cleanly
SHA ccf6a155844b [33d915d9e8ce] ('{nl,mac}80211: allow 4addr AP operation on crypto controlled devices')
  fixed by upstream commit e6f4051123fd
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 027e043f9c78 [652b8b086538] ('drm: panel-orientation-quirks: Add quirk for GPD MicroPC')
  fixed by upstream commit dae1ccee012e
  Fix is missing from chromeos-4.19 and applies cleanly
SHA c854d9b6ef8d [d5b844a2cf50] ('ftrace/x86: Remove possible deadlock between register_kprobe() and ftrace_run_update_code()')
  fixed by upstream commit 074376ac0e1d
  Fix is missing from chromeos-4.19 and applies cleanly
SHA c854d9b6ef8d [d5b844a2cf50] ('ftrace/x86: Remove possible deadlock between register_kprobe() and ftrace_run_update_code()')
  fixed by upstream commit f1c6ece23729
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 3ae98dc2db1e [a3fb01ba5af0] ('blk-iolatency: only account submitted bios')
  fixed by upstream commit c9b3007feca0
  Fix may be missing from chromeos-4.19; trying to apply it results in conflicts/errors
SHA 025eb12bb4b0 [bd293d071ffe] ('dm bufio: fix deadlock with loop device')
  fixed by upstream commit cf3591ef8329
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 9d3586bcdae3 [a9eeb998c28d] ('hv_sock: Add support for delayed close')
  fixed by upstream commit 685703b497ba
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 9e441c7844a6 [c7944ebb9ce9] ('NFSv4: Fix lookup revalidate of regular files')
  fixed by upstream commit 42f72cf368c5
  Fix is missing from chromeos-4.19 and applies cleanly
SHA 3d180fe5cd76 [883a2a80f79c] ('Input: elantech - enable SMBus on new (2018+) systems')
  fixed by upstream commit f3b5720cabaf
  Fix is missing from chromeos-4.19 and applies cleanly