Re: [PATCH 4.19 000/306] 4.19.87-stable review

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Naresh Kamboju]

On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> This is the start of the stable review cycle for the 4.19.87 release.
> There are 306 patches in this series, all will be posted as a response
> to this one.  If anyone has any issues with these being applied, please
> let me know.
>
> Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
> Anything received after that time might be too late.
>
> The whole patch series can be found in one patch at:
>         https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
> or in the git tree and branch at:
>         git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.19.y
> and the diffstat can be found below.
>
> thanks,
>
> greg k-h

Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.

The problematic patch is,

> Jouni Hogander <jouni.hogander@unikie.com>
>     net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject

And this kernel panic is been fixed by below patch,

commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
Author: Eric Dumazet <edumazet@google.com>
Date:   Wed Nov 20 19:19:07 2019 -0800

    net-sysfs: fix netdev_queue_add_kobject() breakage

    kobject_put() should only be called in error path.

    Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
rx|netdev_queue_add_kobject")
    Signed-off-by: Eric Dumazet <edumazet@google.com>
    Cc: Jouni Hogander <jouni.hogander@unikie.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

Summary
------------------------------------------------------------------------

kernel: 4.19.87-rc1
git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
git branch: linux-4.19.y
git commit: 57c5d287ed483d6100bdca528c57562b894487b5
git describe: v4.19.86-307-g57c5d287ed48
Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.19-oe-sanity/build/v4.19.86-307-g57c5d287ed48

Regressions (compared to build v4.19.86)

[    3.556598] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000090
[    3.569683] PGD 0 P4D 0
[    3.572221] Oops: 0000 [#1] SMP PTI
[    3.575705] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.19.87-rc1 #1
[    3.582049] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.0b 07/27/2017
[    3.589523] RIP: 0010:kernfs_find_ns+0x1f/0x130
[    3.594053] Code: fe ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
55 48 89 e5 41 57 41 56 41 55 41 54 49 89 ff 53 49 89 f6 49 89 d5 48
83 ec 08 <0f> b7 87 90 00 00 00 48 8b 5f 68 66 83 e0 20 66 89 45 d6 8b
05 68
[    3.612788] RSP: 0000:ffffaf514002fba8 EFLAGS: 00010292
[    3.618007] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff95d15b89
[    3.625130] RDX: 0000000000000000 RSI: ffffffff95ddefc7 RDI: 0000000000000000
[    3.632254] RBP: ffffaf514002fbd8 R08: ffffffff94b88f05 R09: 0000000000000001
[    3.639377] R10: ffffaf514002fbd8 R11: 0000000000000001 R12: ffffffff95ddefc7
[    3.646502] R13: 0000000000000000 R14: ffffffff95ddefc7 R15: 0000000000000000
[    3.653625] FS:  0000000000000000(0000) GS:ffff95c0dfb00000(0000)
knlGS:0000000000000000
[    3.661704] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.667442] CR2: 0000000000000090 CR3: 00000003bc01e001 CR4: 00000000003606e0
[    3.674565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    3.681689] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    3.688811] Call Trace:
[    3.691259]  kernfs_find_and_get_ns+0x33/0x60
[    3.695616]  sysfs_remove_group+0x2a/0x90
[    3.699622]  netdev_queue_update_kobjects+0xc6/0x150
[    3.704587]  netif_set_real_num_tx_queues+0x7e/0x230
[    3.709546]  ? igb_configure_msix+0xde/0x170
[    3.713816]  __igb_open+0x19e/0x5e0
[    3.717322]  igb_open+0x10/0x20
[    3.720506]  __dev_open+0xd7/0x170
[    3.723904]  ? _raw_spin_unlock_bh+0x35/0x40
[    3.728168]  __dev_change_flags+0x17e/0x1d0
[    3.732363]  dev_change_flags+0x29/0x60
[    3.736195]  ip_auto_config+0x28b/0xf04
[    3.740033]  ? tcp_set_default_congestion_control+0xac/0x150
[    3.745683]  ? root_nfs_parse_addr+0xa5/0xa5
[    3.749948]  ? set_debug_rodata+0x17/0x17
[    3.753951]  do_one_initcall+0x61/0x2b4
[    3.757783]  ? do_one_initcall+0x61/0x2b4
[    3.761793]  ? set_debug_rodata+0xa/0x17
[    3.765713]  ? rcu_read_lock_sched_held+0x81/0x90
[    3.770418]  kernel_init_freeable+0x1d8/0x270
[    3.774777]  ? rest_init+0x190/0x190
[    3.778354]  kernel_init+0xe/0x110
[    3.781753]  ret_from_fork+0x3a/0x50
[    3.785349] Modules linked in:
[    3.788427] CR2: 0000000000000090
[    3.791740] ---[ end trace 831b7578b86a527b ]---
[    3.796358] RIP: 0010:kernfs_find_ns+0x1f/0x130
[    3.800889] Code: fe ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
55 48 89 e5 41 57 41 56 41 55 41 54 49 89 ff 53 49 89 f6 49 89 d5 48
83 ec 08 <0f> b7 87 90 00 00 00 48 8b 5f 68 66 83 e0 20 66 89 45 d6 8b
05 68
[    3.819625] RSP: 0000:ffffaf514002fba8 EFLAGS: 00010292
[    3.824843] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff95d15b89
[    3.831968] RDX: 0000000000000000 RSI: ffffffff95ddefc7 RDI: 0000000000000000
[    3.839091] RBP: ffffaf514002fbd8 R08: ffffffff94b88f05 R09: 0000000000000001
[    3.846216] R10: ffffaf514002fbd8 R11: 0000000000000001 R12: ffffffff95ddefc7
[    3.853363] R13: 0000000000000000 R14: ffffffff95ddefc7 R15: 0000000000000000
[    3.860499] FS:  0000000000000000(0000) GS:ffff95c0dfb00000(0000)
knlGS:0000000000000000
[    3.868583] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.874323] CR2: 0000000000000090 CR3: 00000003bc01e001 CR4: 00000000003606e0
[    3.881454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    3.888576] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    3.895702] BUG: sleeping function called from invalid context at
/usr/src/kernel/include/linux/percpu-rwsem.h:34
[    3.905946] in_atomic(): 0, irqs_disabled(): 1, pid: 1, name: swapper/0
[    3.912550] INFO: lockdep is turned off.
[    3.916465] irq event stamp: 1027104
[    3.920038] hardirqs last  enabled at (1027103):
[<ffffffff9553abd6>] _raw_spin_unlock_irqrestore+0x36/0x50
[    3.929770] hardirqs last disabled at (1027104):
[<ffffffff94801c8b>] trace_hardirqs_off_thunk+0x1a/0x1c
[    3.939233] softirqs last  enabled at (1025718):
[<ffffffff9580031f>] __do_softirq+0x31f/0x426
[    3.947832] softirqs last disabled at (1025703):
[<ffffffff948eddb6>] irq_exit+0xd6/0xe0
[    3.955916] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G      D
  4.19.87-rc1 #1
[    3.963648] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.0b 07/27/2017
[    3.971126] Call Trace:
[    3.973572]  dump_stack+0x7a/0xa5
[    3.976890]  ___might_sleep+0x152/0x240
[    3.980720]  __might_sleep+0x4a/0x80
[    3.984309]  exit_signals+0x33/0x240
[    3.987896]  do_exit+0xbd/0xcf0
[    3.991035]  ? kernel_init_freeable+0x1d8/0x270
[    3.995567]  ? rest_init+0x190/0x190
[    3.999136]  rewind_stack_do_exit+0x17/0x20
[    4.003348] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x00000009
[    4.003348]
[    4.012537] Kernel Offset: 0x13800000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    4.023318] ---[ end Kernel panic - not syncing: Attempted to kill
init! exitcode=0x00000009
[    4.023318]  ]---


--
Linaro LKFT
https://lkft.linaro.org

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Greg Kroah-Hartman]

On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
> On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > This is the start of the stable review cycle for the 4.19.87 release.
> > There are 306 patches in this series, all will be posted as a response
> > to this one.  If anyone has any issues with these being applied, please
> > let me know.
> >
> > Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
> > Anything received after that time might be too late.
> >
> > The whole patch series can be found in one patch at:
> >         https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
> > or in the git tree and branch at:
> >         git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.19.y
> > and the diffstat can be found below.
> >
> > thanks,
> >
> > greg k-h
> 
> Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
> 
> The problematic patch is,
> 
> > Jouni Hogander <jouni.hogander@unikie.com>
> >     net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> 
> And this kernel panic is been fixed by below patch,
> 
> commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Wed Nov 20 19:19:07 2019 -0800
> 
>     net-sysfs: fix netdev_queue_add_kobject() breakage
> 
>     kobject_put() should only be called in error path.
> 
>     Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
> rx|netdev_queue_add_kobject")
>     Signed-off-by: Eric Dumazet <edumazet@google.com>
>     Cc: Jouni Hogander <jouni.hogander@unikie.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>

Now queued up, I'll push out -rc2 versions with this fix.

greg k-h

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[shuah]

On 11/28/19 12:36 AM, Greg Kroah-Hartman wrote:
> On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
>> On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>>>
>>> This is the start of the stable review cycle for the 4.19.87 release.
>>> There are 306 patches in this series, all will be posted as a response
>>> to this one.  If anyone has any issues with these being applied, please
>>> let me know.
>>>
>>> Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
>>> Anything received after that time might be too late.
>>>
>>> The whole patch series can be found in one patch at:
>>>          https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
>>> or in the git tree and branch at:
>>>          git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.19.y
>>> and the diffstat can be found below.
>>>
>>> thanks,
>>>
>>> greg k-h
>>
>> Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
>>
>> The problematic patch is,
>>
>>> Jouni Hogander <jouni.hogander@unikie.com>
>>>      net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
>>
>> And this kernel panic is been fixed by below patch,
>>
>> commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
>> Author: Eric Dumazet <edumazet@google.com>
>> Date:   Wed Nov 20 19:19:07 2019 -0800
>>
>>      net-sysfs: fix netdev_queue_add_kobject() breakage
>>
>>      kobject_put() should only be called in error path.
>>
>>      Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
>> rx|netdev_queue_add_kobject")
>>      Signed-off-by: Eric Dumazet <edumazet@google.com>
>>      Cc: Jouni Hogander <jouni.hogander@unikie.com>
>>      Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> Now queued up, I'll push out -rc2 versions with this fix.
> 
> greg k-h
> 

Ran into this on my test system. I will try rc2.

thanks,
-- Shuah

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[shuah]

On 11/28/19 8:56 AM, shuah wrote:
> On 11/28/19 12:36 AM, Greg Kroah-Hartman wrote:
>> On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
>>> On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
>>> <gregkh@linuxfoundation.org> wrote:
>>>>
>>>> This is the start of the stable review cycle for the 4.19.87 release.
>>>> There are 306 patches in this series, all will be posted as a response
>>>> to this one.  If anyone has any issues with these being applied, please
>>>> let me know.
>>>>
>>>> Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
>>>> Anything received after that time might be too late.
>>>>
>>>> The whole patch series can be found in one patch at:
>>>>          
>>>> https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz 
>>>>
>>>> or in the git tree and branch at:
>>>>          
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git 
>>>> linux-4.19.y
>>>> and the diffstat can be found below.
>>>>
>>>> thanks,
>>>>
>>>> greg k-h
>>>
>>> Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
>>>
>>> The problematic patch is,
>>>
>>>> Jouni Hogander <jouni.hogander@unikie.com>
>>>>      net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
>>>
>>> And this kernel panic is been fixed by below patch,
>>>
>>> commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
>>> Author: Eric Dumazet <edumazet@google.com>
>>> Date:   Wed Nov 20 19:19:07 2019 -0800
>>>
>>>      net-sysfs: fix netdev_queue_add_kobject() breakage
>>>
>>>      kobject_put() should only be called in error path.
>>>
>>>      Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
>>> rx|netdev_queue_add_kobject")
>>>      Signed-off-by: Eric Dumazet <edumazet@google.com>
>>>      Cc: Jouni Hogander <jouni.hogander@unikie.com>
>>>      Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>> Now queued up, I'll push out -rc2 versions with this fix.
>>
>> greg k-h
>>
> 
> Ran into this on my test system. I will try rc2.
> 

rc2 worked for me.

thanks,
-- Shuah

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Lukas Bulwahn]

On Thu, Nov 28, 2019 at 8:37 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
> > On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > This is the start of the stable review cycle for the 4.19.87 release.
> > > There are 306 patches in this series, all will be posted as a response
> > > to this one.  If anyone has any issues with these being applied, please
> > > let me know.
> > >
> > > Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
> > > Anything received after that time might be too late.
> > >
> > > The whole patch series can be found in one patch at:
> > >         https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
> > > or in the git tree and branch at:
> > >         git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.19.y
> > > and the diffstat can be found below.
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
> >
> > The problematic patch is,
> >
> > > Jouni Hogander <jouni.hogander@unikie.com>
> > >     net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> >
> > And this kernel panic is been fixed by below patch,
> >
> > commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
> > Author: Eric Dumazet <edumazet@google.com>
> > Date:   Wed Nov 20 19:19:07 2019 -0800
> >
> >     net-sysfs: fix netdev_queue_add_kobject() breakage
> >
> >     kobject_put() should only be called in error path.
> >
> >     Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
> > rx|netdev_queue_add_kobject")
> >     Signed-off-by: Eric Dumazet <edumazet@google.com>
> >     Cc: Jouni Hogander <jouni.hogander@unikie.com>
> >     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> Now queued up, I'll push out -rc2 versions with this fix.
>
> greg k-h

We have also been informed about another regression these two commits
are causing:

https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/

I suggest to drop these two patches from this queue, and give us a
week to shake out the regressions of the change, and once ready, we
can include the complete set of fixes to stable (probably in a week or
two).

Lukas

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Greg Kroah-Hartman]

On Thu, Nov 28, 2019 at 04:57:09PM -0700, shuah wrote:
> On 11/28/19 8:56 AM, shuah wrote:
> > On 11/28/19 12:36 AM, Greg Kroah-Hartman wrote:
> > > On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
> > > > On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
> > > > <gregkh@linuxfoundation.org> wrote:
> > > > > 
> > > > > This is the start of the stable review cycle for the 4.19.87 release.
> > > > > There are 306 patches in this series, all will be posted as a response
> > > > > to this one.  If anyone has any issues with these being applied, please
> > > > > let me know.
> > > > > 
> > > > > Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
> > > > > Anything received after that time might be too late.
> > > > > 
> > > > > The whole patch series can be found in one patch at:
> > > > >           https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
> > > > > 
> > > > > or in the git tree and branch at:
> > > > >           git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> > > > > linux-4.19.y
> > > > > and the diffstat can be found below.
> > > > > 
> > > > > thanks,
> > > > > 
> > > > > greg k-h
> > > > 
> > > > Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
> > > > 
> > > > The problematic patch is,
> > > > 
> > > > > Jouni Hogander <jouni.hogander@unikie.com>
> > > > >      net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> > > > 
> > > > And this kernel panic is been fixed by below patch,
> > > > 
> > > > commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
> > > > Author: Eric Dumazet <edumazet@google.com>
> > > > Date:   Wed Nov 20 19:19:07 2019 -0800
> > > > 
> > > >      net-sysfs: fix netdev_queue_add_kobject() breakage
> > > > 
> > > >      kobject_put() should only be called in error path.
> > > > 
> > > >      Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
> > > > rx|netdev_queue_add_kobject")
> > > >      Signed-off-by: Eric Dumazet <edumazet@google.com>
> > > >      Cc: Jouni Hogander <jouni.hogander@unikie.com>
> > > >      Signed-off-by: David S. Miller <davem@davemloft.net>
> > > 
> > > Now queued up, I'll push out -rc2 versions with this fix.
> > > 
> > > greg k-h
> > > 
> > 
> > Ran into this on my test system. I will try rc2.
> > 
> 
> rc2 worked for me.

Great, thanks for testing and confirming it.

greg k-h

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Naresh Kamboju]

On Thu, 28 Nov 2019 at 13:07, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:


> Now queued up, I'll push out -rc2 versions with this fix.

Results from Linaro’s test farm.
Regressions detected on i386.

i386 build failed on 4.19 and 4.14

In function 'setup_cpu_entry_area_ptes',
    inlined from 'setup_cpu_entry_areas' at arch/x86/mm/cpu_entry_area.c:209:2:
include/linux/compiler.h:348:38: error: call to
'__compiletime_assert_192' declared with attribute error: BUILD_BUG_ON
failed: (CPU_ENTRY_AREA_PAGES+1)*PAGE_SIZE != CPU_ENTRY_AREA_MAP_SIZE
  _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
                                      ^
include/linux/compiler.h:329:4: note: in definition of macro
'__compiletime_assert'
    prefix ## suffix();    \
    ^~~~~~
include/linux/compiler.h:348:2: note: in expansion of macro
'_compiletime_assert'
  _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
  ^~~~~~~~~~~~~~~~~~~
include/linux/build_bug.h:45:37: note: in expansion of macro
'compiletime_assert'
 #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                     ^~~~~~~~~~~~~~~~~~

Bisection points to "x86/cpu_entry_area: Add guard page for entry
stack on 32bit" (e50622b4a1, also present in 4.14.y as 880a98c339).


Summary
------------------------------------------------------------------------

kernel: 4.19.87-rc2
git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
git branch: linux-4.19.y
git commit: 63633b307be0161e7bd6f854a28d7d9fa05f69ef
git describe: v4.19.86-309-g63633b307be0
Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-4.19-oe/build/v4.19.86-309-g63633b307be0

Regressions (compared to build v4.19.86)
------------------------------------------------------------------------

i386:
  build:
    * build_process


No fixes (compared to build v4.19.86)


Ran 18913 total tests in the following environments and test suites.

Environments
--------------
- dragonboard-410c - arm64
- hi6220-hikey - arm64
- i386
- juno-r2 - arm64
- qemu_arm
- qemu_arm64
- qemu_x86_64
- x15 - arm
- x86_64

Test Suites
-----------
* build
* install-android-platform-tools-r2600
* kselftest
* libhugetlbfs
* linux-log-parser
* ltp-cap_bounds-tests
* ltp-commands-tests
* ltp-containers-tests
* ltp-cpuhotplug-tests
* ltp-cve-tests
* ltp-dio-tests
* ltp-fcntl-locktests-tests
* ltp-filecaps-tests
* ltp-fs_bind-tests
* ltp-fs_perms_simple-tests
* ltp-fsx-tests
* ltp-hugetlb-tests
* ltp-io-tests
* ltp-ipc-tests
* ltp-math-tests
* ltp-mm-tests
* ltp-nptl-tests
* ltp-pty-tests
* ltp-sched-tests
* ltp-securebits-tests
* ltp-syscalls-tests
* perf
* spectre-meltdown-checker-test
* v4l2-compliance
* ltp-fs-tests
* network-basic-tests
* kvm-unit-tests
* ltp-open-posix-tests
* ssuite
* kselftest-vsyscall-mode-native
* kselftest-vsyscall-mode-none

-- 
Linaro LKFT
https://lkft.linaro.org

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Greg Kroah-Hartman]

On Fri, Nov 29, 2019 at 06:46:23AM +0100, Lukas Bulwahn wrote:
> On Thu, Nov 28, 2019 at 8:37 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Nov 28, 2019 at 12:23:41PM +0530, Naresh Kamboju wrote:
> > > On Thu, 28 Nov 2019 at 02:25, Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > This is the start of the stable review cycle for the 4.19.87 release.
> > > > There are 306 patches in this series, all will be posted as a response
> > > > to this one.  If anyone has any issues with these being applied, please
> > > > let me know.
> > > >
> > > > Responses should be made by Fri, 29 Nov 2019 20:18:09 +0000.
> > > > Anything received after that time might be too late.
> > > >
> > > > The whole patch series can be found in one patch at:
> > > >         https://www.kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.19.87-rc1.gz
> > > > or in the git tree and branch at:
> > > >         git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-4.19.y
> > > > and the diffstat can be found below.
> > > >
> > > > thanks,
> > > >
> > > > greg k-h
> > >
> > > Kernel BUG noticed on x86_64 device while booting 4.19.87-rc1 kernel.
> > >
> > > The problematic patch is,
> > >
> > > > Jouni Hogander <jouni.hogander@unikie.com>
> > > >     net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> > >
> > > And this kernel panic is been fixed by below patch,
> > >
> > > commit 48a322b6f9965b2f1e4ce81af972f0e287b07ed0
> > > Author: Eric Dumazet <edumazet@google.com>
> > > Date:   Wed Nov 20 19:19:07 2019 -0800
> > >
> > >     net-sysfs: fix netdev_queue_add_kobject() breakage
> > >
> > >     kobject_put() should only be called in error path.
> > >
> > >     Fixes: b8eb718348b8 ("net-sysfs: Fix reference count leak in
> > > rx|netdev_queue_add_kobject")
> > >     Signed-off-by: Eric Dumazet <edumazet@google.com>
> > >     Cc: Jouni Hogander <jouni.hogander@unikie.com>
> > >     Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > Now queued up, I'll push out -rc2 versions with this fix.
> >
> > greg k-h
> 
> We have also been informed about another regression these two commits
> are causing:
> 
> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
> 
> I suggest to drop these two patches from this queue, and give us a
> week to shake out the regressions of the change, and once ready, we
> can include the complete set of fixes to stable (probably in a week or
> two).

Ok, thanks for the information, I've now dropped them from all of the
queues that had them in them.

greg k-h

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Jouni Högander]

Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>> > Now queued up, I'll push out -rc2 versions with this fix.
>> >
>> > greg k-h
>> 
>> We have also been informed about another regression these two commits
>> are causing:
>> 
>> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
>> 
>> I suggest to drop these two patches from this queue, and give us a
>> week to shake out the regressions of the change, and once ready, we
>> can include the complete set of fixes to stable (probably in a week or
>> two).
>
> Ok, thanks for the information, I've now dropped them from all of the
> queues that had them in them.
>
> greg k-h

I have now run more extensive Syzkaller testing on following patches:

cb626bf566eb net-sysfs: Fix reference count leak
ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject

These patches are fixing couple of memory leaks including this one found
by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2

I can reproduce these memory leaks in following stable branches: 4.14,
4.19, and 5.4.

These are all now merged into net/master tree and based on my testing
they are ready to be taken into stable branches as well.

Best Regards,

Jouni Högander

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Lukas Bulwahn]

[-- Attachment #1: Type: text/plain, Size: 3390 bytes --]


On Wed, 22 Jan 2020, Jouni Högander wrote:

> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> >> > Now queued up, I'll push out -rc2 versions with this fix.
> >> >
> >> > greg k-h
> >> 
> >> We have also been informed about another regression these two commits
> >> are causing:
> >> 
> >> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
> >> 
> >> I suggest to drop these two patches from this queue, and give us a
> >> week to shake out the regressions of the change, and once ready, we
> >> can include the complete set of fixes to stable (probably in a week or
> >> two).
> >
> > Ok, thanks for the information, I've now dropped them from all of the
> > queues that had them in them.
> >
> > greg k-h
> 
> I have now run more extensive Syzkaller testing on following patches:
> 
> cb626bf566eb net-sysfs: Fix reference count leak
> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> 
> These patches are fixing couple of memory leaks including this one found
> by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
> 
> I can reproduce these memory leaks in following stable branches: 4.14,
> 4.19, and 5.4.
> 
> These are all now merged into net/master tree and based on my testing
> they are ready to be taken into stable branches as well.
>

+ syzkaller list
Jouni et. al, please drop Linus in further responses; Linus, it was wrong 
to add you to this thread in the first place (reason is explained below)

Jouni, thanks for investigating.

It raises the following questions and comments:

- Does the memory leak NOT appear on 4.9 and earlier LTS branches (or did 
you not check that)? If it does not appear, can you bisect it with the 
reproducer to the commit between 4.14 and 4.9?

- Do the reproducers you found with your syzkaller testing show the same 
behaviour (same bisection) as the reproducers from syzbot?

- I fear syzbot's automatic bisection on is wrong, and Linus' commit 
0e034f5c4bc4 ("iwlwifi: fix mis-merge that breaks the driver") is not to 
blame here; that commit did not cause the memory leak, but fixed some 
unrelated issue that simply confuses syzbot's automatic bisection.

Just FYI: Dmitry Vyukov's evaluation of the syzbot bisection shows that 
about 50% are wrong, e.g., due to multiple bugs being triggered with one 
reproducer and the difficulty of automatically identifying them of being 
different due to different root causes (despite the smart heuristics of 
syzkaller & syzbot). So, to identify the actual commit on which the memory 
leak first appeared, you need to bisect manually with your own judgement 
if the reported bug stack trace fits to the issue you investigating. Or 
you use syzbot's automatic bisection but then with a reduced kernel config 
that cannot be confused by other issues. You might possibly also hit a 
"beginning of time" in your bisection, where KASAN was simply not 
supported, then the initially causing commit can simply not determined by 
bisection with the reproducer and needs some code inspection and 
archaeology with git. Can you go ahead try to identify the correct commit 
for this issue?


Lukas

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Jouni Högander]

Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:

> On Wed, 22 Jan 2020, Jouni Högander wrote:
>
>> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>> >> > Now queued up, I'll push out -rc2 versions with this fix.
>> >> >
>> >> > greg k-h
>> >> 
>> >> We have also been informed about another regression these two commits
>> >> are causing:
>> >> 
>> >> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
>> >> 
>> >> I suggest to drop these two patches from this queue, and give us a
>> >> week to shake out the regressions of the change, and once ready, we
>> >> can include the complete set of fixes to stable (probably in a week or
>> >> two).
>> >
>> > Ok, thanks for the information, I've now dropped them from all of the
>> > queues that had them in them.
>> >
>> > greg k-h
>> 
>> I have now run more extensive Syzkaller testing on following patches:
>> 
>> cb626bf566eb net-sysfs: Fix reference count leak
>> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
>> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
>> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
>> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
>> 
>> These patches are fixing couple of memory leaks including this one found
>> by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
>> 
>> I can reproduce these memory leaks in following stable branches: 4.14,
>> 4.19, and 5.4.
>> 
>> These are all now merged into net/master tree and based on my testing
>> they are ready to be taken into stable branches as well.
>>
>
> + syzkaller list
> Jouni et. al, please drop Linus in further responses; Linus, it was wrong 
> to add you to this thread in the first place (reason is explained below)
>
> Jouni, thanks for investigating.
>
> It raises the following questions and comments:
>
> - Does the memory leak NOT appear on 4.9 and earlier LTS branches (or did 
> you not check that)? If it does not appear, can you bisect it with the 
> reproducer to the commit between 4.14 and 4.9?

I tested and these memory leaks are not reproucible in 4.9 and earlier.

>
> - Do the reproducers you found with your syzkaller testing show the same 
> behaviour (same bisection) as the reproducers from syzbot?

Yes, they are same.

>
> - I fear syzbot's automatic bisection on is wrong, and Linus' commit 
> 0e034f5c4bc4 ("iwlwifi: fix mis-merge that breaks the driver") is not to 
> blame here; that commit did not cause the memory leak, but fixed some 
> unrelated issue that simply confuses syzbot's automatic bisection.
>
> Just FYI: Dmitry Vyukov's evaluation of the syzbot bisection shows that 
> about 50% are wrong, e.g., due to multiple bugs being triggered with one 
> reproducer and the difficulty of automatically identifying them of being 
> different due to different root causes (despite the smart heuristics of 
> syzkaller & syzbot). So, to identify the actual commit on which the memory 
> leak first appeared, you need to bisect manually with your own judgement 
> if the reported bug stack trace fits to the issue you investigating. Or 
> you use syzbot's automatic bisection but then with a reduced kernel config 
> that cannot be confused by other issues. You might possibly also hit a 
> "beginning of time" in your bisection, where KASAN was simply not 
> supported, then the initially causing commit can simply not determined by 
> bisection with the reproducer and needs some code inspection and 
> archaeology with git. Can you go ahead try to identify the correct commit 
> for this issue?

These two commits (that are not in 4.9 and earlier) are intorducing these leaks:

commit e331c9066901dfe40bea4647521b86e9fb9901bb
Author: YueHaibing <yuehaibing@huawei.com>
Date:   Tue Mar 19 10:16:53 2019 +0800

    net-sysfs: call dev_hold if kobject_init_and_add success
    
    [ Upstream commit a3e23f719f5c4a38ffb3d30c8d7632a4ed8ccd9e ]
    
    In netdev_queue_add_kobject and rx_queue_add_kobject,
    if sysfs_create_group failed, kobject_put will call
    netdev_queue_release to decrease dev refcont, however
    dev_hold has not be called. So we will see this while
    unregistering dev:
    
    unregister_netdevice: waiting for bcsh0 to become free. Usage count = -1
    
    Reported-by: Hulk Robot <hulkci@huawei.com>
    Fixes: d0d668371679 ("net: don't decrement kobj reference count on init fail
ure")
    Signed-off-by: YueHaibing <yuehaibing@huawei.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit d0d6683716791b2a2761a1bb025c613eb73da6c3
Author: stephen hemminger <stephen@networkplumber.org>
Date:   Fri Aug 18 13:46:19 2017 -0700

    net: don't decrement kobj reference count on init failure
    
    If kobject_init_and_add failed, then the failure path would
    decrement the reference count of the queue kobject whose reference
    count was already zero.
    
    Fixes: 114cf5802165 ("bql: Byte queue limits")
    Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

>
>
> Lukas

BR,

Jouni Högander

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Lukas Bulwahn]

[-- Attachment #1: Type: text/plain, Size: 7514 bytes --]



On Mon, 27 Jan 2020, Jouni Högander wrote:

> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
> 
> > On Wed, 22 Jan 2020, Jouni Högander wrote:
> >
> >> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> >> >> > Now queued up, I'll push out -rc2 versions with this fix.
> >> >> >
> >> >> > greg k-h
> >> >> 
> >> >> We have also been informed about another regression these two commits
> >> >> are causing:
> >> >> 
> >> >> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
> >> >> 
> >> >> I suggest to drop these two patches from this queue, and give us a
> >> >> week to shake out the regressions of the change, and once ready, we
> >> >> can include the complete set of fixes to stable (probably in a week or
> >> >> two).
> >> >
> >> > Ok, thanks for the information, I've now dropped them from all of the
> >> > queues that had them in them.
> >> >
> >> > greg k-h
> >> 
> >> I have now run more extensive Syzkaller testing on following patches:
> >> 
> >> cb626bf566eb net-sysfs: Fix reference count leak
> >> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
> >> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
> >> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
> >> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> >> 
> >> These patches are fixing couple of memory leaks including this one found
> >> by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
> >> 
> >> I can reproduce these memory leaks in following stable branches: 4.14,
> >> 4.19, and 5.4.
> >> 
> >> These are all now merged into net/master tree and based on my testing
> >> they are ready to be taken into stable branches as well.
> >>
> >
> > + syzkaller list
> > Jouni et. al, please drop Linus in further responses; Linus, it was wrong 
> > to add you to this thread in the first place (reason is explained below)
> >
> > Jouni, thanks for investigating.
> >
> > It raises the following questions and comments:
> >
> > - Does the memory leak NOT appear on 4.9 and earlier LTS branches (or did 
> > you not check that)? If it does not appear, can you bisect it with the 
> > reproducer to the commit between 4.14 and 4.9?
> 
> I tested and these memory leaks are not reproucible in 4.9 and earlier.
> 
> >
> > - Do the reproducers you found with your syzkaller testing show the same 
> > behaviour (same bisection) as the reproducers from syzbot?
> 
> Yes, they are same.
> 
> >
> > - I fear syzbot's automatic bisection on is wrong, and Linus' commit 
> > 0e034f5c4bc4 ("iwlwifi: fix mis-merge that breaks the driver") is not to 
> > blame here; that commit did not cause the memory leak, but fixed some 
> > unrelated issue that simply confuses syzbot's automatic bisection.
> >
> > Just FYI: Dmitry Vyukov's evaluation of the syzbot bisection shows that 
> > about 50% are wrong, e.g., due to multiple bugs being triggered with one 
> > reproducer and the difficulty of automatically identifying them of being 
> > different due to different root causes (despite the smart heuristics of 
> > syzkaller & syzbot). So, to identify the actual commit on which the memory 
> > leak first appeared, you need to bisect manually with your own judgement 
> > if the reported bug stack trace fits to the issue you investigating. Or 
> > you use syzbot's automatic bisection but then with a reduced kernel config 
> > that cannot be confused by other issues. You might possibly also hit a 
> > "beginning of time" in your bisection, where KASAN was simply not 
> > supported, then the initially causing commit can simply not determined by 
> > bisection with the reproducer and needs some code inspection and 
> > archaeology with git. Can you go ahead try to identify the correct commit 
> > for this issue?
> 
> These two commits (that are not in 4.9 and earlier) are intorducing these leaks:
> 
> commit e331c9066901dfe40bea4647521b86e9fb9901bb
> Author: YueHaibing <yuehaibing@huawei.com>
> Date:   Tue Mar 19 10:16:53 2019 +0800
> 
>     net-sysfs: call dev_hold if kobject_init_and_add success
>     
>     [ Upstream commit a3e23f719f5c4a38ffb3d30c8d7632a4ed8ccd9e ]
>     
>     In netdev_queue_add_kobject and rx_queue_add_kobject,
>     if sysfs_create_group failed, kobject_put will call
>     netdev_queue_release to decrease dev refcont, however
>     dev_hold has not be called. So we will see this while
>     unregistering dev:
>     
>     unregister_netdevice: waiting for bcsh0 to become free. Usage count = -1
>     
>     Reported-by: Hulk Robot <hulkci@huawei.com>
>     Fixes: d0d668371679 ("net: don't decrement kobj reference count on init fail
> ure")
>     Signed-off-by: YueHaibing <yuehaibing@huawei.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> commit d0d6683716791b2a2761a1bb025c613eb73da6c3
> Author: stephen hemminger <stephen@networkplumber.org>
> Date:   Fri Aug 18 13:46:19 2017 -0700
> 
>     net: don't decrement kobj reference count on init failure
>     
>     If kobject_init_and_add failed, then the failure path would
>     decrement the reference count of the queue kobject whose reference
>     count was already zero.
>     
>     Fixes: 114cf5802165 ("bql: Byte queue limits")
>     Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
> 

But, it seems that we now have just a long sequences of fix patches.

This commit from 2011 seems to be the initial buggy one:

commit 114cf5802165ee93e3ab461c9c505cd94a08b800
Author: Tom Herbert <therbert@google.com>
Date:   Mon Nov 28 16:33:09 2011 +0000

    bql: Byte queue limits

And then we just have fixes over fixes:

114cf5802165ee93e3ab461c9c505cd94a08b800
fixed by d0d6683716791b2a2761a1bb025c613eb73da6c3
fixed by a3e23f719f5c4a38ffb3d30c8d7632a4ed8ccd9e
fixed by the sequence of your five patches, mentioned above


If that is right, we should be able to find a reproducer with syzkaller on 
the versions before d0d668371679 ("net: don't decrement kobj reference 
count on init failure") with fault injection enabled or some manually 
injected fault by modifying the source code to always fail on init to 
really trigger the init failure, and see the reference count go below 
zero.

All further issues should also have reproducers found with syzkaller.
If we have a good feeling on the reproducers and this series of fixes 
really fixed the issue now here for all cases, we should suggest to 
backport all of the fixes to 4.4 and 4.9.

We should NOT just have Greg pick up a subset of the patches and backport 
them to 4.4 and 4.9, that will likely break more than it fixes.

Jouni, did you see Greg's bot inform you that he would pick up your latest 
patch for 4.4 and 4.9? Please respond to those emails to make sure a 
complete set of patches is picked up, which we tested with all those 
intermediate reproducers and an extensive syzkaller run hitting the 
net-sysfs interface (e.g., by configuring the corpus and check coverage).

If you cannot do this testing for 4.4 and 4.9 now quickly (you 
potentially have less than 24 hours), we should hold those new patches 
back for 4.4 and 4.9, as none of the fixes seem to be applied at all right 
now and the users have not complained yet on 4.4 and 4.9.
Once testing of the whole fix sequence is done, we request to backport all 
patches at once for 4.4 and 4.9.

Lukas

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Jouni Högander]

Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:

> On Mon, 27 Jan 2020, Jouni Högander wrote:
>
>> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
>> 
>> > On Wed, 22 Jan 2020, Jouni Högander wrote:
>> >
>> >> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>> >> >> > Now queued up, I'll push out -rc2 versions with this fix.
>> >> >> >
>> >> >> > greg k-h
>> >> >> 
>> >> >> We have also been informed about another regression these two commits
>> >> >> are causing:
>> >> >> 
>> >> >> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
>> >> >> 
>> >> >> I suggest to drop these two patches from this queue, and give us a
>> >> >> week to shake out the regressions of the change, and once ready, we
>> >> >> can include the complete set of fixes to stable (probably in a week or
>> >> >> two).
>> >> >
>> >> > Ok, thanks for the information, I've now dropped them from all of the
>> >> > queues that had them in them.
>> >> >
>> >> > greg k-h
>> >> 
>> >> I have now run more extensive Syzkaller testing on following patches:
>> >> 
>> >> cb626bf566eb net-sysfs: Fix reference count leak
>> >> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
>> >> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
>> >> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
>> >> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
>> >> 
>> >> These patches are fixing couple of memory leaks including this one found
>> >> by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
>> >> 
>> >> I can reproduce these memory leaks in following stable branches: 4.14,
>> >> 4.19, and 5.4.
>> >> 
>> >> These are all now merged into net/master tree and based on my testing
>> >> they are ready to be taken into stable branches as well.
>> >>
>> >
>> > + syzkaller list
>> > Jouni et. al, please drop Linus in further responses; Linus, it was wrong 
>> > to add you to this thread in the first place (reason is explained below)
>> >
>> > Jouni, thanks for investigating.
>> >
>> > It raises the following questions and comments:
>> >
>> > - Does the memory leak NOT appear on 4.9 and earlier LTS branches (or did 
>> > you not check that)? If it does not appear, can you bisect it with the 
>> > reproducer to the commit between 4.14 and 4.9?
>> 
>> I tested and these memory leaks are not reproucible in 4.9 and earlier.
>> 
>> >
>> > - Do the reproducers you found with your syzkaller testing show the same 
>> > behaviour (same bisection) as the reproducers from syzbot?
>> 
>> Yes, they are same.
>> 
>> >
>> > - I fear syzbot's automatic bisection on is wrong, and Linus' commit 
>> > 0e034f5c4bc4 ("iwlwifi: fix mis-merge that breaks the driver") is not to 
>> > blame here; that commit did not cause the memory leak, but fixed some 
>> > unrelated issue that simply confuses syzbot's automatic bisection.
>> >
>> > Just FYI: Dmitry Vyukov's evaluation of the syzbot bisection shows that 
>> > about 50% are wrong, e.g., due to multiple bugs being triggered with one 
>> > reproducer and the difficulty of automatically identifying them of being 
>> > different due to different root causes (despite the smart heuristics of 
>> > syzkaller & syzbot). So, to identify the actual commit on which the memory 
>> > leak first appeared, you need to bisect manually with your own judgement 
>> > if the reported bug stack trace fits to the issue you investigating. Or 
>> > you use syzbot's automatic bisection but then with a reduced kernel config 
>> > that cannot be confused by other issues. You might possibly also hit a 
>> > "beginning of time" in your bisection, where KASAN was simply not 
>> > supported, then the initially causing commit can simply not determined by 
>> > bisection with the reproducer and needs some code inspection and 
>> > archaeology with git. Can you go ahead try to identify the correct commit 
>> > for this issue?
>> 
>> These two commits (that are not in 4.9 and earlier) are intorducing these leaks:
>> 
>> commit e331c9066901dfe40bea4647521b86e9fb9901bb
>> Author: YueHaibing <yuehaibing@huawei.com>
>> Date:   Tue Mar 19 10:16:53 2019 +0800
>> 
>>     net-sysfs: call dev_hold if kobject_init_and_add success
>>     
>>     [ Upstream commit a3e23f719f5c4a38ffb3d30c8d7632a4ed8ccd9e ]
>>     
>>     In netdev_queue_add_kobject and rx_queue_add_kobject,
>>     if sysfs_create_group failed, kobject_put will call
>>     netdev_queue_release to decrease dev refcont, however
>>     dev_hold has not be called. So we will see this while
>>     unregistering dev:
>>     
>>     unregister_netdevice: waiting for bcsh0 to become free. Usage count = -1
>>     
>>     Reported-by: Hulk Robot <hulkci@huawei.com>
>>     Fixes: d0d668371679 ("net: don't decrement kobj reference count on init fail
>> ure")
>>     Signed-off-by: YueHaibing <yuehaibing@huawei.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>> 
>> commit d0d6683716791b2a2761a1bb025c613eb73da6c3
>> Author: stephen hemminger <stephen@networkplumber.org>
>> Date:   Fri Aug 18 13:46:19 2017 -0700
>> 
>>     net: don't decrement kobj reference count on init failure
>>     
>>     If kobject_init_and_add failed, then the failure path would
>>     decrement the reference count of the queue kobject whose reference
>>     count was already zero.
>>     
>>     Fixes: 114cf5802165 ("bql: Byte queue limits")
>>     Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>> 
>
> But, it seems that we now have just a long sequences of fix patches.
>
> This commit from 2011 seems to be the initial buggy one:
>
> commit 114cf5802165ee93e3ab461c9c505cd94a08b800
> Author: Tom Herbert <therbert@google.com>
> Date:   Mon Nov 28 16:33:09 2011 +0000
>
>     bql: Byte queue limits
>
> And then we just have fixes over fixes:
>
> 114cf5802165ee93e3ab461c9c505cd94a08b800
> fixed by d0d6683716791b2a2761a1bb025c613eb73da6c3
> fixed by a3e23f719f5c4a38ffb3d30c8d7632a4ed8ccd9e
> fixed by the sequence of your five patches, mentioned above
>
>
> If that is right, we should be able to find a reproducer with syzkaller on 
> the versions before d0d668371679 ("net: don't decrement kobj reference 
> count on init failure") with fault injection enabled or some manually 
> injected fault by modifying the source code to always fail on init to 
> really trigger the init failure, and see the reference count go below 
> zero.
>
> All further issues should also have reproducers found with syzkaller.
> If we have a good feeling on the reproducers and this series of fixes 
> really fixed the issue now here for all cases, we should suggest to 
> backport all of the fixes to 4.4 and 4.9.
>
> We should NOT just have Greg pick up a subset of the patches and backport 
> them to 4.4 and 4.9, that will likely break more than it fixes.

Yes, this is the case.

>
> Jouni, did you see Greg's bot inform you that he would pick up your latest 
> patch for 4.4 and 4.9? Please respond to those emails to make sure a 
> complete set of patches is picked up, which we tested with all those 
> intermediate reproducers and an extensive syzkaller run hitting the 
> net-sysfs interface (e.g., by configuring the corpus and check
> coverage).

I already responded to not pick these patches into 4.4 and 4.9. 

>
> If you cannot do this testing for 4.4 and 4.9 now quickly (you 
> potentially have less than 24 hours), we should hold those new patches 
> back for 4.4 and 4.9, as none of the fixes seem to be applied at all right 
> now and the users have not complained yet on 4.4 and 4.9.
> Once testing of the whole fix sequence is done, we request to backport all 
> patches at once for 4.4 and 4.9.

If we want to pick whole set including older patches I think I need more
time for identifying which older patches (apart from these two I
identified causing the memory leak) should be taken in and for testing.

>
> Lukas

BR,

Jouni Högander

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Jouni Högander]

Hello Greg,

jouni.hogander@unikie.com (Jouni Högander) writes:

> Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
>>> > Now queued up, I'll push out -rc2 versions with this fix.
>>> >
>>> > greg k-h
>>> 
>>> We have also been informed about another regression these two commits
>>> are causing:
>>> 
>>> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
>>> 
>>> I suggest to drop these two patches from this queue, and give us a
>>> week to shake out the regressions of the change, and once ready, we
>>> can include the complete set of fixes to stable (probably in a week or
>>> two).
>>
>> Ok, thanks for the information, I've now dropped them from all of the
>> queues that had them in them.
>>
>> greg k-h
>
> I have now run more extensive Syzkaller testing on following patches:
>
> cb626bf566eb net-sysfs: Fix reference count leak
> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
>
> These patches are fixing couple of memory leaks including this one found
> by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
>
> I can reproduce these memory leaks in following stable branches: 4.14,
> 4.19, and 5.4.
>
> These are all now merged into net/master tree and based on my testing
> they are ready to be taken into stable branches as well.
>
> Best Regards,
>
> Jouni Högander

These four patches are still missing from 4.14 and 4.19 branches:

ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject

Could you please consider taking them in or let me know if you want some
further activities from my side?

BR,

Jouni Högander

Re: [PATCH 4.19 000/306] 4.19.87-stable review

[Greg Kroah-Hartman]

On Tue, Jan 28, 2020 at 12:28:15PM +0200, Jouni Högander wrote:
> Hello Greg,
> 
> jouni.hogander@unikie.com (Jouni Högander) writes:
> 
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> writes:
> >>> > Now queued up, I'll push out -rc2 versions with this fix.
> >>> >
> >>> > greg k-h
> >>> 
> >>> We have also been informed about another regression these two commits
> >>> are causing:
> >>> 
> >>> https://lore.kernel.org/lkml/ace19af4-7cae-babd-bac5-cd3505dcd874@I-love.SAKURA.ne.jp/
> >>> 
> >>> I suggest to drop these two patches from this queue, and give us a
> >>> week to shake out the regressions of the change, and once ready, we
> >>> can include the complete set of fixes to stable (probably in a week or
> >>> two).
> >>
> >> Ok, thanks for the information, I've now dropped them from all of the
> >> queues that had them in them.
> >>
> >> greg k-h
> >
> > I have now run more extensive Syzkaller testing on following patches:
> >
> > cb626bf566eb net-sysfs: Fix reference count leak
> > ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
> > e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
> > 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
> > b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> >
> > These patches are fixing couple of memory leaks including this one found
> > by Syzbot: https://syzkaller.appspot.com/bug?extid=ad8ca40ecd77896d51e2
> >
> > I can reproduce these memory leaks in following stable branches: 4.14,
> > 4.19, and 5.4.
> >
> > These are all now merged into net/master tree and based on my testing
> > they are ready to be taken into stable branches as well.
> >
> > Best Regards,
> >
> > Jouni Högander
> 
> These four patches are still missing from 4.14 and 4.19 branches:
> 
> ddd9b5e3e765 net-sysfs: Call dev_hold always in rx_queue_add_kobject
> e0b60903b434 net-sysfs: Call dev_hold always in netdev_queue_add_kobje
> 48a322b6f996 net-sysfs: fix netdev_queue_add_kobject() breakage
> b8eb718348b8 net-sysfs: Fix reference count leak in rx|netdev_queue_add_kobject
> 
> Could you please consider taking them in or let me know if you want some
> further activities from my side?

Thanks for the list, I have now queued these all up.

greg k-h